enigmare/v2-crawler

1{"id":"doc-overview_spark_4_2_0_documentation-479dfe01","source":"documentation","title":"Overview - Spark 4.2.0 Documentation","url":"https://spark.apache.org/docs/latest/","text":"Apache Spark - A Unified engine for large-scale data analytics Apache Spark is a unified analytics engine for large-scale data processing. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs. It also supports a rich set of higher-level tools including Spark SQL for SQL and structured data processing, pandas API on Spark for pandas workloads, MLlib for machine learning, GraphX for graph processing, and Structured Streaming for incremental computation and stream processing.\n\nDownloading Get Spark from the downloads page of the project website. This documentation is for Spark version 4.2.0. Spark uses Hadoop’s client libraries for HDFS and YARN. Downloads are pre-packaged for a handful of popular Hadoop versions. Users can also download a “Hadoop free” binary and run Spark with any Hadoop version by augmenting Spark’s classpath. Scala and Java users can include Spark in their projects using its Maven coordinates and Python users can install Spark from PyPI. If you’d like to build Spark from source, visit Building Spark. Spark runs on both Windows and UNIX-like systems (e.g. Linux, Mac OS), and it should run on any platform that runs a supported version of Java. This should include JVMs on x86_64 and ARM64. It’s easy to run locally on one machine — all you need is to have java installed on your system PATH, or the JAVA_HOME environment variable pointing to a Java installation. Spark runs on Java 17/21/25, Scala 2.13, Python 3.10+, and R 4.0+ (Deprecated). Java 25 prior to version 25.0.3 support is deprecated as of Spark 4.2.0. When using the Scala API, it is necessary for applications to use the same version of Scala that Spark was compiled for. Since Spark 4.0.0, it’s Scala 2.13. Running the Examples and Shell Spark comes with several sample programs. Python, Scala, Java, and R examples are in the examples/src/main directory. To run Spark interactively in a Python interpreter, use bin/pyspark: ./bin/pyspark --master \"local[2]\" Sample applications are provided in Python. For /bin/spark-submit examples/src/main/python/pi.py 10 To run one of the Scala or Java sample programs, use bin/run-example <class> [params] in the top-level Spark directory. (Behind the scenes, this invokes the more general spark-submit script for launching applications). For example, ./bin/run-example SparkPi 10 You can also run Spark interactively through a modified version of the Scala shell. This is a great way to learn the framework. ./bin/spark-shell --master \"local[2]\" The --master option specifies the master URL for a distributed cluster, or local to run locally with one thread, or local[N] to run locally with N threads. You should start by using local for testing. For a full list of options, run the Spark shell with the --help option. Since version 1.4, Spark has provided an R API (only the DataFrame APIs are included). To run Spark interactively in an R interpreter, use bin/sparkR: ./bin/sparkR --master \"local[2]\" Example applications are also provided in R. For /bin/spark-submit examples/src/main/r/dataframe.R Running Spark Client Applications Anywhere with Spark Connect Spark Connect is a new client-server architecture introduced in Spark 3.4 that decouples Spark client applications and allows remote connectivity to Spark clusters. The separation between client and server allows Spark and its open ecosystem to be leveraged from anywhere, embedded in any application. In Spark 3.4, Spark Connect provides DataFrame API coverage for PySpark and DataFrame/Dataset API support in Scala. To learn more about Spark Connect and how to use it, see Spark Connect Overview. Launching on a Cluster The Spark cluster mode overview explains the key concepts in running on a cluster. Spark can run both by itself, or over several existing cluster managers. It currently provides several options for Deploy way to deploy Spark on a private cluster Hadoop YARN Kubernetes Where to Go from Here Programming quick introduction to the Spark API; start here! RDD Programming of Spark basics - RDDs (core but old API), accumulators, and broadcast variables Spark SQL, Datasets, and structured data with relational queries (newer API than RDDs) Structured structured data streams with relation queries (using Datasets and DataFrames, newer API than DStreams) Spark data streams using DStreams (old API) machine learning algorithms graphs SparkR (Deprecated): processing data with Spark in R data with Spark in Python Spark SQL data with SQL on the command line Declarative data pipelines that create and maintain multiple tables API Python API (Sphinx) Spark Scala API (Scaladoc) Spark Java API (Javadoc) Spark R API (Roxygen2) Spark SQL, Built-in Functions (MkDocs) Deployment of concepts and components when running on a cluster Submitting and deploying applications Deployment Deploy a standalone cluster quickly without a third-party cluster manager Spark on top of Hadoop NextGen (YARN) Spark apps on top of Kubernetes directly Amazon that let you launch a cluster on EC2 in about 5 minutes Spark Kubernetes : deploy Spark apps on top of Kubernetes via operator patterns Spark clusters on top of Kubernetes via operator patterns Other : customize Spark via its configuration system the behavior of your applications Web useful information about your applications Tuning practices to optimize performance and memory use Job resources across and within Spark applications security support Hardware for cluster hardware Integration with other storage Infrastructures OpenStack Swift Migration guides for Spark components Building Spark using the Maven system Contributing to Spark Third Party third party Spark projects External Homepage Spark Community resources, including local meetups StackOverflow tag apache-spark Mailing questions about Spark here AMP series of training camps at UC Berkeley that featured talks and exercises about Spark, Spark Streaming, Mesos, and more. Videos, are available online for free. Code are also available in the examples subfolder of Spark (Python, Scala, Java, R)\n\nExample:\n```text\n./bin/pyspark --master \"local[2]\"\n```\n\nExample:\n```text\n./bin/spark-submit examples/src/main/python/pi.py 10\n```\n\nExample:\n```text\n./bin/run-example SparkPi 10\n```\n\nExample:\n```text\n./bin/spark-shell --master \"local[2]\"\n```\n\nExample:\n```text\n./bin/sparkR --master \"local[2]\"\n```\n\nExample:\n```text\n./bin/spark-submit examples/src/main/r/dataframe.R\n```\n\n<|endofdoc|>","metadata":{"transformedAt":"2026-08-18T15:21:42.954Z","totalSectionsIncluded":2,"totalCodeBlocksIncluded":6,"totalLines":35,"estimatedTokens":1619}}2{"id":"doc-overview_spark_4_2_0_documentation-b8157c24","source":"documentation","title":"Overview - Spark 4.2.0 Documentation","url":"https://spark.apache.org/docs/latest/index.html","text":"Apache Spark - A Unified engine for large-scale data analytics Apache Spark is a unified analytics engine for large-scale data processing. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs. It also supports a rich set of higher-level tools including Spark SQL for SQL and structured data processing, pandas API on Spark for pandas workloads, MLlib for machine learning, GraphX for graph processing, and Structured Streaming for incremental computation and stream processing.\n\nDownloading Get Spark from the downloads page of the project website. This documentation is for Spark version 4.2.0. Spark uses Hadoop’s client libraries for HDFS and YARN. Downloads are pre-packaged for a handful of popular Hadoop versions. Users can also download a “Hadoop free” binary and run Spark with any Hadoop version by augmenting Spark’s classpath. Scala and Java users can include Spark in their projects using its Maven coordinates and Python users can install Spark from PyPI. If you’d like to build Spark from source, visit Building Spark. Spark runs on both Windows and UNIX-like systems (e.g. Linux, Mac OS), and it should run on any platform that runs a supported version of Java. This should include JVMs on x86_64 and ARM64. It’s easy to run locally on one machine — all you need is to have java installed on your system PATH, or the JAVA_HOME environment variable pointing to a Java installation. Spark runs on Java 17/21/25, Scala 2.13, Python 3.10+, and R 4.0+ (Deprecated). Java 25 prior to version 25.0.3 support is deprecated as of Spark 4.2.0. When using the Scala API, it is necessary for applications to use the same version of Scala that Spark was compiled for. Since Spark 4.0.0, it’s Scala 2.13. Running the Examples and Shell Spark comes with several sample programs. Python, Scala, Java, and R examples are in the examples/src/main directory. To run Spark interactively in a Python interpreter, use bin/pyspark: ./bin/pyspark --master \"local[2]\" Sample applications are provided in Python. For /bin/spark-submit examples/src/main/python/pi.py 10 To run one of the Scala or Java sample programs, use bin/run-example <class> [params] in the top-level Spark directory. (Behind the scenes, this invokes the more general spark-submit script for launching applications). For example, ./bin/run-example SparkPi 10 You can also run Spark interactively through a modified version of the Scala shell. This is a great way to learn the framework. ./bin/spark-shell --master \"local[2]\" The --master option specifies the master URL for a distributed cluster, or local to run locally with one thread, or local[N] to run locally with N threads. You should start by using local for testing. For a full list of options, run the Spark shell with the --help option. Since version 1.4, Spark has provided an R API (only the DataFrame APIs are included). To run Spark interactively in an R interpreter, use bin/sparkR: ./bin/sparkR --master \"local[2]\" Example applications are also provided in R. For /bin/spark-submit examples/src/main/r/dataframe.R Running Spark Client Applications Anywhere with Spark Connect Spark Connect is a new client-server architecture introduced in Spark 3.4 that decouples Spark client applications and allows remote connectivity to Spark clusters. The separation between client and server allows Spark and its open ecosystem to be leveraged from anywhere, embedded in any application. In Spark 3.4, Spark Connect provides DataFrame API coverage for PySpark and DataFrame/Dataset API support in Scala. To learn more about Spark Connect and how to use it, see Spark Connect Overview. Launching on a Cluster The Spark cluster mode overview explains the key concepts in running on a cluster. Spark can run both by itself, or over several existing cluster managers. It currently provides several options for Deploy way to deploy Spark on a private cluster Hadoop YARN Kubernetes Where to Go from Here Programming quick introduction to the Spark API; start here! RDD Programming of Spark basics - RDDs (core but old API), accumulators, and broadcast variables Spark SQL, Datasets, and structured data with relational queries (newer API than RDDs) Structured structured data streams with relation queries (using Datasets and DataFrames, newer API than DStreams) Spark data streams using DStreams (old API) machine learning algorithms graphs SparkR (Deprecated): processing data with Spark in R data with Spark in Python Spark SQL data with SQL on the command line Declarative data pipelines that create and maintain multiple tables API Python API (Sphinx) Spark Scala API (Scaladoc) Spark Java API (Javadoc) Spark R API (Roxygen2) Spark SQL, Built-in Functions (MkDocs) Deployment of concepts and components when running on a cluster Submitting and deploying applications Deployment Deploy a standalone cluster quickly without a third-party cluster manager Spark on top of Hadoop NextGen (YARN) Spark apps on top of Kubernetes directly Amazon that let you launch a cluster on EC2 in about 5 minutes Spark Kubernetes : deploy Spark apps on top of Kubernetes via operator patterns Spark clusters on top of Kubernetes via operator patterns Other : customize Spark via its configuration system the behavior of your applications Web useful information about your applications Tuning practices to optimize performance and memory use Job resources across and within Spark applications security support Hardware for cluster hardware Integration with other storage Infrastructures OpenStack Swift Migration guides for Spark components Building Spark using the Maven system Contributing to Spark Third Party third party Spark projects External Homepage Spark Community resources, including local meetups StackOverflow tag apache-spark Mailing questions about Spark here AMP series of training camps at UC Berkeley that featured talks and exercises about Spark, Spark Streaming, Mesos, and more. Videos, are available online for free. Code are also available in the examples subfolder of Spark (Python, Scala, Java, R)\n\nExample:\n```text\n./bin/pyspark --master \"local[2]\"\n```\n\nExample:\n```text\n./bin/spark-submit examples/src/main/python/pi.py 10\n```\n\nExample:\n```text\n./bin/run-example SparkPi 10\n```\n\nExample:\n```text\n./bin/spark-shell --master \"local[2]\"\n```\n\nExample:\n```text\n./bin/sparkR --master \"local[2]\"\n```\n\nExample:\n```text\n./bin/spark-submit examples/src/main/r/dataframe.R\n```\n\n<|endofdoc|>","metadata":{"transformedAt":"2026-08-18T15:21:42.955Z","totalSectionsIncluded":2,"totalCodeBlocksIncluded":6,"totalLines":35,"estimatedTokens":1619}}3{"id":"doc-spark_sql_and_dataframes_spark_4_2_0_documentati-ed80a874","source":"documentation","title":"Spark SQL and DataFrames - Spark 4.2.0 Documentation","url":"https://spark.apache.org/docs/latest/sql-programming-guide.html","text":"Spark SQL Guide Getting Started Data Sources Data Source V2 Performance Tuning Distributed SQL Engine PySpark Usage Guide for Pandas with Apache Arrow Migration Guide SQL Reference Error Conditions Spark SQL, DataFrames and Datasets Guide Spark SQL is a Spark module for structured data processing. Unlike the basic Spark RDD API, the interfaces provided by Spark SQL provide Spark with more information about the structure of both the data and the computation being performed. Internally, Spark SQL uses this extra information to perform extra optimizations. There are several ways to interact with Spark SQL including SQL and the Dataset API. When computing a result, the same execution engine is used, independent of which API/language you are using to express the computation. This unification means that developers can easily switch back and forth between different APIs based on which provides the most natural way to express a given transformation. All of the examples on this page use sample data included in the Spark distribution and can be run in the spark-shell, pyspark shell, or sparkR shell. SQL One use of Spark SQL is to execute SQL queries. Spark SQL can also be used to read data from an existing Hive installation. For more on how to configure this feature, please refer to the Hive Tables section. When running SQL from within another programming language the results will be returned as a Dataset/DataFrame. You can also interact with the SQL interface using the command-line or over JDBC/ODBC. Datasets and DataFrames A Dataset is a distributed collection of data. Dataset is a new interface added in Spark 1.6 that provides the benefits of RDDs (strong typing, ability to use powerful lambda functions) with the benefits of Spark SQL’s optimized execution engine. A Dataset can be constructed from JVM objects and then manipulated using functional transformations (map, flatMap, filter, etc.). The Dataset API is available in Scala and Java. Python does not have the support for the Dataset API. But due to Python’s dynamic nature, many of the benefits of the Dataset API are already available (i.e. you can access the field of a row by name naturally row.columnName). The case for R is similar. A DataFrame is a Dataset organized into named columns. It is conceptually equivalent to a table in a relational database or a data frame in R/Python, but with richer optimizations under the hood. DataFrames can be constructed from a wide array of sources such data files, tables in Hive, external databases, or existing RDDs. The DataFrame API is available in Python, Scala, Java and R. In Scala and Java, a DataFrame is represented by a Dataset of Rows. In the Scala API, DataFrame is simply a type alias of Dataset[Row]. While, in Java API, users need to use Dataset<Row> to represent a DataFrame. Throughout this document, we will often refer to Scala/Java Datasets of Rows as DataFrames.\n\n<|endofdoc|>","metadata":{"transformedAt":"2026-08-18T15:21:42.955Z","totalSectionsIncluded":1,"totalCodeBlocksIncluded":0,"totalLines":3,"estimatedTokens":732}}4{"id":"doc-getting_started_pyspark_4_2_0_documentation-81ce15c9","source":"documentation","title":"Getting Started — PySpark 4.2.0 documentation","url":"https://spark.apache.org/docs/latest/api/python/getting_started/index.html","text":"Site Navigation Overview Getting Started Tutorials User Guide API Reference Development More Migration Guides GitHub PyPI Section Navigation Installation Connect API on Spark Testing PySpark Getting Started Getting Started# This page summarizes the basic steps required to setup and get started with PySpark. There are more guides shared with other languages such as Quick Start in Programming Guides at the Spark documentation. There are live notebooks where you can try PySpark out without any other Live Connect Live API on Spark The list below is the contents of this quickstart Python Versions Supported Using PyPI Using Conda Using venv Manually Downloading Installing from Source Dependencies DataFrame Creation Viewing Data Selecting and Accessing Data Applying a Function Grouping Data Getting Data In/Out Working with SQL Connect Launch Spark server with Spark Connect Connect to Spark Connect server Create DataFrame API on Spark Object Creation Missing Data Operations Grouping Plotting Getting data in/out Testing PySpark Build a PySpark Application Testing your PySpark Application Putting It All Together! previous PySpark Overview next Installation Show Source\n\n<|endofdoc|>","metadata":{"transformedAt":"2026-08-18T15:21:42.956Z","totalSectionsIncluded":1,"totalCodeBlocksIncluded":0,"totalLines":3,"estimatedTokens":298}}5{"id":"doc-structured_streaming_programming_guide_spark_4_2-246ce084","source":"documentation","title":"Structured Streaming Programming Guide - Spark 4.2.0 Documentation","url":"https://spark.apache.org/docs/latest/streaming/index.html","text":"Structured Streaming Programming Guide Overview Getting Started APIs on DataFrames and Datasets Performance Tips Additional Information Structured Streaming Programming Guide Overview Structured Streaming is a scalable and fault-tolerant stream processing engine built on the Spark SQL engine. You can express your streaming computation the same way you would express a batch computation on static data. The Spark SQL engine will take care of running it incrementally and continuously and updating the final result as streaming data continues to arrive. You can use the Dataset/DataFrame API in Scala, Java, Python or R to express streaming aggregations, event-time windows, stream-to-batch joins, etc. The computation is executed on the same optimized Spark SQL engine. Finally, the system ensures end-to-end exactly-once fault-tolerance guarantees through checkpointing and Write-Ahead Logs. In short, Structured Streaming provides fast, scalable, fault-tolerant, end-to-end exactly-once stream processing without the user having to reason about streaming. Internally, by default, Structured Streaming queries are processed using a micro-batch processing engine, which processes data streams as a series of small batch jobs thereby achieving end-to-end latencies as low as 100 milliseconds and exactly-once fault-tolerance guarantees. However, since Spark 2.3, we have introduced a new low-latency processing mode called Continuous Processing, which can achieve end-to-end latencies as low as 1 millisecond with at-least-once guarantees. Without changing the Dataset/DataFrame operations in your queries, you will be able to choose the mode based on your application requirements. In this guide, we are going to walk you through the programming model and the APIs. We are going to explain the concepts mostly using the default micro-batch processing model, and then later discuss Continuous Processing model. First, let’s start with a simple example of a Structured Streaming query - a streaming word count.\n\n<|endofdoc|>","metadata":{"transformedAt":"2026-08-18T15:21:42.956Z","totalSectionsIncluded":1,"totalCodeBlocksIncluded":0,"totalLines":3,"estimatedTokens":506}}6{"id":"doc-mllib_main_guide_spark_4_2_0_documentation-f9dc228a","source":"documentation","title":"MLlib: Main Guide - Spark 4.2.0 Documentation","url":"https://spark.apache.org/docs/latest/ml-guide.html","text":"Guide Basic statistics Data sources Pipelines Extracting, transforming and selecting features Classification and Regression Clustering Collaborative filtering Frequent Pattern Mining Model selection and tuning Advanced topics API Guide Data types Basic statistics Classification and regression Collaborative filtering Clustering Dimensionality reduction Feature extraction and transformation Frequent pattern mining Evaluation metrics PMML model export Optimization (developer) ML Model Security Machine Learning Library (MLlib) Guide MLlib is Spark’s machine learning (ML) library. Its goal is to make practical machine learning scalable and easy. At a high level, it provides tools such learning algorithms such as classification, regression, clustering, and collaborative filtering extraction, transformation, dimensionality reduction, and selection for constructing, evaluating, and tuning ML Pipelines and load algorithms, models, and Pipelines algebra, statistics, data handling, etc. API is primary API The MLlib RDD-based API is now in maintenance mode. As of Spark 2.0, the RDD-based APIs in the spark.mllib package have entered maintenance mode. The primary Machine Learning API for Spark is now the DataFrame-based API in the spark.ml package. What are the implications? MLlib will still support the RDD-based API in spark.mllib with bug fixes. MLlib will not add new features to the RDD-based API. In the Spark 2.x releases, MLlib will add features to the DataFrames-based API to reach feature parity with the RDD-based API. Why is MLlib switching to the DataFrame-based API? DataFrames provide a more user-friendly API than RDDs. The many benefits of DataFrames include Spark Datasources, SQL/DataFrame queries, Tungsten and Catalyst optimizations, and uniform APIs across languages. The DataFrame-based API for MLlib provides a uniform API across ML algorithms and across multiple languages. DataFrames facilitate practical ML Pipelines, particularly feature transformations. See the Pipelines guide for details. What is “Spark ML”? “Spark ML” is not an official name but occasionally used to refer to the MLlib DataFrame-based API. This is majorly due to the org.apache.spark.ml Scala package name used by the DataFrame-based API, and the “Spark ML Pipelines” term we used initially to emphasize the pipeline concept. Is MLlib deprecated? No. MLlib includes both the RDD-based API and the DataFrame-based API. The RDD-based API is now in maintenance mode. But neither API is deprecated, nor MLlib as a whole. Dependencies MLlib uses linear algebra packages Breeze and dev.ludovic.netlib for optimised numerical processing1. Those packages may call native acceleration libraries such as Intel MKL or OpenBLAS if they are available as system libraries or in runtime library paths. However, native acceleration libraries can’t be distributed with Spark. See MLlib Linear Algebra Acceleration Guide for how to enable accelerated linear algebra processing. If accelerated native libraries are not enabled, you will see a warning message like below and a pure JVM implementation will be used : Failed to load implementation To use MLlib in Python, you will need NumPy version 1.21 or newer. Highlights in 3.0 The list below highlights some of the new features and enhancements added to MLlib in the 3.0 release of columns support was added to Binarizer (SPARK-23578), StringIndexer (SPARK-11215), StopWordsRemover (SPARK-29808) and PySpark QuantileDiscretizer (SPARK-22796). Tree-Based Feature Transformation was added (SPARK-13677). Two new evaluators MultilabelClassificationEvaluator (SPARK-16692) and RankingEvaluator (SPARK-28045) were added. Sample weights support was added in DecisionTreeClassifier/Regressor (SPARK-19591), RandomForestClassifier/Regressor (SPARK-9478), GBTClassifier/Regressor (SPARK-9612), MulticlassClassificationEvaluator (SPARK-24101), RegressionEvaluator (SPARK-24102), BinaryClassificationEvaluator (SPARK-24103), BisectingKMeans (SPARK-30351), KMeans (SPARK-29967) and GaussianMixture (SPARK-30102). R API for PowerIterationClustering was added (SPARK-19827). Added Spark ML listener for tracking ML pipeline status (SPARK-23674). Fit with validation set was added to Gradient Boosted Trees in Python (SPARK-24333). RobustScaler transformer was added (SPARK-28399). Factorization Machines classifier and regressor were added (SPARK-29224). Gaussian Naive Bayes Classifier (SPARK-16872) and Complement Naive Bayes Classifier (SPARK-29942) were added. ML function parity between Scala and Python (SPARK-28958). predictRaw is made public in all the Classification models. predictProbability is made public in all the Classification models except LinearSVCModel (SPARK-30358). Migration Guide The migration guide is now archived on this page. To learn more about the benefits and background of system optimised natives, you may wish to watch Sam Halliday’s ScalaX talk on High Performance Linear Algebra in Scala. ↩\n\nExample:\n```text\nWARNING: Failed to load implementation from:dev.ludovic.netlib.blas.JNIBLAS\n```\n\n<|endofdoc|>","metadata":{"transformedAt":"2026-08-18T15:21:42.956Z","totalSectionsIncluded":1,"totalCodeBlocksIncluded":1,"totalLines":8,"estimatedTokens":1267}}7{"id":"doc-quick_start_spark_4_2_0_documentation-032fd3f5","source":"documentation","title":"Quick Start - Spark 4.2.0 Documentation","url":"https://spark.apache.org/docs/latest/quick-start.html","text":"Quick Start Interactive Analysis with the Spark Shell Basics More on Dataset Operations Caching Self-Contained Applications Where to Go from Here This tutorial provides a quick introduction to using Spark. We will first introduce the API through Spark’s interactive shell (in Python or Scala), then show how to write applications in Java, Scala, and Python. To follow along with this guide, first, download a packaged release of Spark from the Spark website. Since we won’t be using HDFS, you can download a package for any version of Hadoop. Note that, before Spark 2.0, the main programming interface of Spark was the Resilient Distributed Dataset (RDD). After Spark 2.0, RDDs are replaced by Dataset, which is strongly-typed like an RDD, but with richer optimizations under the hood. The RDD interface is still supported, and you can get a more detailed reference at the RDD programming guide. However, we highly recommend you to switch to use Dataset, which has better performance than RDD. See the SQL programming guide to get more information about Dataset. Interactive Analysis with the Spark Shell Basics Spark’s shell provides a simple way to learn the API, as well as a powerful tool to analyze data interactively. It is available in either Scala (which runs on the Java VM and is thus a good way to use existing Java libraries) or Python. Start it by running the following in the Spark /bin/pyspark Or if PySpark is installed with pip in your current Spark’s primary abstracted is called a Dataset. A Dataset is a structured set of information. You can create datasets from Hadoop InputFormats (such as HDFS files) or by transforming other Datasets. Datasets behave differently in some languages. Because Python allows for dynamic typing, Datasets in Python are all Dataset[Row] on an implementation level. This leads to another key Spark DataFrame, or a Dataset with named columns. If you’re familiar with DataFrames from pandas or R, you’ll be familiar with how DataFrames work in Spark. In other languages, like Java, the difference between a Dataset and DataFrame is larger, but for now let’s proceed with Python. Let’s make a new DataFrame using the README.md file in the Spark souce directory via the command line: >>> textFile = spark.read.text(\"README.md\") Once you’ve created the DataFrame, you can perform actions against it, or transform it into another DataFrame. For more details see the API doc. >>> textFile.count() # Number of rows in this DataFrame 126 >>> textFile.first() # First row in this DataFrame Row(value=u'# Apache Spark') Now let’s transform this DataFrame to a new one. The filter function returns a new DataFrame with a subset of the lines in the file. >>> linesWithSpark = textFile.filter(textFile.value.contains(\"Spark\")) You can also chain together transformations and actions: >>> textFile.filter(textFile.value.contains(\"Spark\")).count() # How many lines contain \"Spark\"? 15 ./bin/spark-shell Spark’s primary abstracted is called a Dataset. A Dataset is a structured set of information. You can create datasets from Hadoop InputFormats (such as HDFS files) or by transforming other Datasets. Let’s make a new DataFrame using the README.md file in the Spark souce directory via the command > val textFile = spark.read.textFile(\"README.md\") [String] = [value: string] Once you’ve created the Dataset, you can perform actions and transformations against it. For more details, see the API doc. scala> textFile.count() // Number of items in this Dataset = 126 // May be different from yours as README.md will change over time, similar to other outputs scala> textFile.first() // First item in this Dataset = # Apache Spark Now let’s transform this Dataset into a new one. The filter function returns a new Dataset with a subset of the items in the file. scala> val linesWithSpark = textFile.filter(line => line.contains(\"Spark\")) [String] = [value: string] We can chain together transformations and > textFile.filter(line => line.contains(\"Spark\")).count() // How many lines contain \"Spark\"? = 15 More on Dataset Operations Dataset actions and transformations can be used for more complex computations. Let’s say we want to find the line with the most words: >>> from pyspark.sql import functions as sf >>> textFile.select(sf.size(sf.split(textFile.value, \"\\s+\")).name(\"numWords\")).agg(sf.max(sf.col(\"numWords\"))).collect() [Row(max(numWords)=15)] This first maps a line to an integer value and aliases it as “numWords”, creating a new DataFrame. agg is called on that DataFrame to find the largest word count. The arguments to select and agg are both Column, we can use df.colName to get a column from a DataFrame. We can also import pyspark.sql.functions, which provides a lot of convenient functions to build a new Column from an old one. One common data flow pattern is MapReduce, as popularized by Hadoop. Spark can implement MapReduce flows easily: >>> wordCounts = textFile.select(sf.explode(sf.split(textFile.value, \"\\s+\")).alias(\"word\")).groupBy(\"word\").count() Here, we use the explode function in select, to transform a Dataset of lines to a Dataset of words, and then combine groupBy and count to compute the per-word counts in the file as a DataFrame of 2 columns: “word” and “count”. To collect the word counts in our shell, we can call collect: >>> wordCounts.collect() [Row(word=u'online', count=1), Row(word=u'graphs', count=1), ...] scala> textFile.map(line => line.split(\" \").size).reduce((a, b) => if (a > b) a else b) = 15 This first maps a line to an integer value, creating a new Dataset. reduce is called on that Dataset to find the largest word count. The arguments to map and reduce are Scala function literals (closures), and can use any language feature or Scala/Java library. For example, we can easily call functions declared elsewhere. We’ll use Math.max() function to make this code easier to > import java.lang.Math import java.lang.Math scala> textFile.map(line => line.split(\" \").size).reduce((a, b) => Math.max(a, b)) = 15 One common data flow pattern is MapReduce, as popularized by Hadoop. Spark can implement MapReduce flows > val wordCounts = textFile.flatMap(line => line.split(\" \")).groupByKey(identity).count() [(String, Long)] = [value: string, count(1): bigint] Here, we call flatMap to transform a Dataset of lines to a Dataset of words, and then combine groupByKey and count to compute the per-word counts in the file as a Dataset of (String, Long) pairs. To collect the word counts in our shell, we can call > wordCounts.collect() [(String, Int)] = Array((means,1), (under,2), (this,3), (Because,1), (Python,2), (agree,1), (cluster.,1), ...) Caching Spark also supports pulling data sets into a cluster-wide in-memory cache. This is very useful when data is accessed repeatedly, such as when querying a small “hot” dataset or when running an iterative algorithm like PageRank. As a simple example, let’s mark our linesWithSpark dataset to be cached: >>> linesWithSpark.cache() >>> linesWithSpark.count() 15 >>> linesWithSpark.count() 15 It may seem silly to use Spark to explore and cache a 100-line text file. The interesting part is that these same functions can be used on very large data sets, even when they are striped across tens or hundreds of nodes. You can also do this interactively by connecting bin/pyspark to a cluster, as described in the RDD programming guide. scala> linesWithSpark.cache() = [value: string] scala> linesWithSpark.count() = 15 scala> linesWithSpark.count() = 15 It may seem silly to use Spark to explore and cache a 100-line text file. The interesting part is that these same functions can be used on very large data sets, even when they are striped across tens or hundreds of nodes. You can also do this interactively by connecting bin/spark-shell to a cluster, as described in the RDD programming guide. Self-Contained Applications Suppose we wish to write a self-contained application using the Spark API. We will walk through a simple application in Scala (with sbt), Java (with Maven), and Python (pip). Now we will show how to write an application using the Python API (PySpark). If you are building a packaged PySpark application or library you can add it to your setup.py file =[ 'pyspark==4.2.0' ] As an example, we’ll create a simple Spark application, SimpleApp.py: \"\"\"SimpleApp.py\"\"\" from pyspark.sql import SparkSession logFile = \"YOUR_SPARK_HOME/README.md\" # Should be some file on your system spark = SparkSession.builder.appName(\"SimpleApp\").getOrCreate() logData = spark.read.text(logFile).cache() numAs = logData.filter(logData.value.contains('a')).count() numBs = logData.filter(logData.value.contains('b')).count() print(\"Lines with a: %i, lines with b: %i\" % (numAs, numBs)) spark.stop() This program just counts the number of lines containing ‘a’ and the number containing ‘b’ in a text file. Note that you’ll need to replace YOUR_SPARK_HOME with the location where Spark is installed. As with the Scala and Java examples, we use a SparkSession to create Datasets. For applications that use custom classes or third-party libraries, we can also add code dependencies to spark-submit through its --py-files argument by packaging them into a Note that applications should define a main() method instead of extending scala.App. Subclasses of scala.App may not work correctly. This program just counts the number of lines containing ‘a’ and the number containing ‘b’ in the Spark README. Note that you’ll need to replace YOUR_SPARK_HOME with the location where Spark is installed. Unlike the earlier examples with the Spark shell, which initializes its own SparkSession, we initialize a SparkSession as part of the program. We call SparkSession.builder to construct a SparkSession, then set the application name, and finally call getOrCreate to get the SparkSession instance. Our application depends on the Spark API, so we’ll also include an sbt configuration file, build.sbt, which explains that Spark is a dependency. This file also adds a repository that Spark depends := \"Simple Project\" version := \"1.0\" scalaVersion := \"2.13.18\" libraryDependencies += \"org.apache.spark\" %% \"spark-sql\" % \"4.2.0\" For sbt to work correctly, we’ll need to layout SimpleApp.scala and build.sbt according to the typical directory structure. Once that is in place, we can create a JAR package containing the application’s code, then use the spark-submit script to run our program. # Your directory layout should look like this $ find . . ./build.sbt ./src ./src/main ./src/main/scala ./src/main/scala/SimpleApp.scala # Package a jar containing your application $ sbt package ... [info] Packaging {..}/{..}/target/scala-2.13/simple-project_2.13-1.0.jar # Use spark-submit to run your application $ YOUR_SPARK_HOME/bin/spark-submit \\ --class \"SimpleApp\" \\ --master \"local[4]\" \\ target/scala-2.13/simple-project_2.13-1.0.jar ... Lines with , Lines with This example will use Maven to compile an application JAR, but any similar build system will work. We’ll create a very simple Spark application, SimpleApp.java: /* SimpleApp.java */ import org.apache.spark.sql.SparkSession; import org.apache.spark.sql.Dataset; public class SimpleApp { public static void main(String[] args) { String logFile = \"YOUR_SPARK_HOME/README.md\"; // Should be some file on your system SparkSession spark = SparkSession.builder().appName(\"Simple Application\").getOrCreate(); Dataset<String> logData = spark.read().textFile(logFile).cache(); long numAs = logData.filter(s -> s.contains(\"a\")).count(); long numBs = logData.filter(s -> s.contains(\"b\")).count(); System.out.println(\"Lines with a: \" + numAs + \", lines with b: \" + numBs); spark.stop(); } } This program just counts the number of lines containing ‘a’ and the number containing ‘b’ in the Spark README. Note that you’ll need to replace YOUR_SPARK_HOME with the location where Spark is installed. Unlike the earlier examples with the Spark shell, which initializes its own SparkSession, we initialize a SparkSession as part of the program. To build the program, we also write a Maven pom.xml file that lists Spark as a dependency. Note that Spark artifacts are tagged with a Scala version. <project> <groupId>edu.berkeley</groupId> <artifactId>simple-project</artifactId> <modelVersion>4.0.0</modelVersion> <name>Simple Project</name> <packaging>jar</packaging> <version>1.0</version> <dependencies> <dependency> <!-- Spark dependency --> <groupId>org.apache.spark</groupId> <artifactId>spark-sql_2.13</artifactId> <version>4.2.0</version> <scope>provided</scope> </dependency> </dependencies> </project> We lay out these files according to the canonical Maven directory structure: $ find . ./pom.xml ./src ./src/main ./src/main/java ./src/main/java/SimpleApp.java Now, we can package the application using Maven and execute it with ./bin/spark-submit. # Package a JAR containing your application $ mvn package ... [INFO] Building jar: {..}/{..}/target/simple-project-1.0.jar # Use spark-submit to run your application $ YOUR_SPARK_HOME/bin/spark-submit \\ --class \"SimpleApp\" \\ --master \"local[4]\" \\ target/simple-project-1.0.jar ... Lines with , Lines with Other dependency management tools such as Conda and pip can be also used for custom classes or third-party libraries. See also Python Package Management. Where to Go from Here Congratulations on running your first Spark application! For an in-depth overview of the API, start with the RDD programming guide and the SQL programming guide, or see “Programming Guides” menu for other components. For running applications on a cluster, head to the deployment overview. Finally, Spark includes several samples in the examples directory (Python, Scala, Java, R). You can run them as follows: # For Python examples, use spark-submit /bin/spark-submit examples/src/main/python/pi.py # For Scala and Java, use /bin/run-example SparkPi # For R examples, use spark-submit /bin/spark-submit examples/src/main/r/dataframe.R\n\nExample:\n```text\n./bin/pyspark\n```\n\nExample:\n```text\npyspark\n```\n\nExample:\n```python\n>>> textFile = spark.read.text(\"README.md\")\n```\n\nExample:\n```python\n>>> textFile.count()  # Number of rows in this DataFrame\n126\n\n>>> textFile.first()  # First row in this DataFrame\nRow(value=u'# Apache Spark')\n```\n\nExample:\n```python\n>>> linesWithSpark = textFile.filter(textFile.value.contains(\"Spark\"))\n```\n\nExample:\n```python\n>>> textFile.filter(textFile.value.contains(\"Spark\")).count()  # How many lines contain \"Spark\"?\n15\n```\n\nExample:\n```text\n./bin/spark-shell\n```\n\nExample:\n```scala\nscala> val textFile = spark.read.textFile(\"README.md\")\ntextFile: org.apache.spark.sql.Dataset[String] = [value: string]\n```\n\nExample:\n```scala\nscala> textFile.count() // Number of items in this Dataset\nres0: Long = 126 // May be different from yours as README.md will change over time, similar to other outputs\n\nscala> textFile.first() // First item in this Dataset\nres1: String = # Apache Spark\n```\n\nExample:\n```scala\nscala> val linesWithSpark = textFile.filter(line => line.contains(\"Spark\"))\nlinesWithSpark: org.apache.spark.sql.Dataset[String] = [value: string]\n```\n\nExample:\n```scala\nscala> textFile.filter(line => line.contains(\"Spark\")).count() // How many lines contain \"Spark\"?\nres3: Long = 15\n```\n\nExample:\n```python\n>>> from pyspark.sql import functions as sf\n>>> textFile.select(sf.size(sf.split(textFile.value, \"\\s+\")).name(\"numWords\")).agg(sf.max(sf.col(\"numWords\"))).collect()\n[Row(max(numWords)=15)]\n```\n\nExample:\n```python\n>>> wordCounts = textFile.select(sf.explode(sf.split(textFile.value, \"\\s+\")).alias(\"word\")).groupBy(\"word\").count()\n```\n\nExample:\n```python\n>>> wordCounts.collect()\n[Row(word=u'online', count=1), Row(word=u'graphs', count=1), ...]\n```\n\nExample:\n```scala\nscala> textFile.map(line => line.split(\" \").size).reduce((a, b) => if (a > b) a else b)\nres4: Int = 15\n```\n\nExample:\n```scala\nscala> import java.lang.Math\nimport java.lang.Math\n\nscala> textFile.map(line => line.split(\" \").size).reduce((a, b) => Math.max(a, b))\nres5: Int = 15\n```\n\nExample:\n```scala\nscala> val wordCounts = textFile.flatMap(line => line.split(\" \")).groupByKey(identity).count()\nwordCounts: org.apache.spark.sql.Dataset[(String, Long)] = [value: string, count(1): bigint]\n```\n\nExample:\n```scala\nscala> wordCounts.collect()\nres6: Array[(String, Int)] = Array((means,1), (under,2), (this,3), (Because,1), (Python,2), (agree,1), (cluster.,1), ...)\n```\n\nExample:\n```python\n>>> linesWithSpark.cache()\n\n>>> linesWithSpark.count()\n15\n\n>>> linesWithSpark.count()\n15\n```\n\nExample:\n```scala\nscala> linesWithSpark.cache()\nres7: linesWithSpark.type = [value: string]\n\nscala> linesWithSpark.count()\nres8: Long = 15\n\nscala> linesWithSpark.count()\nres9: Long = 15\n```\n\nExample:\n```python\ninstall_requires=[\n        'pyspark==4.2.0'\n    ]\n```\n\nExample:\n```python\n\"\"\"SimpleApp.py\"\"\"\nfrom pyspark.sql import SparkSession\n\nlogFile = \"YOUR_SPARK_HOME/README.md\"  # Should be some file on your system\nspark = SparkSession.builder.appName(\"SimpleApp\").getOrCreate()\nlogData = spark.read.text(logFile).cache()\n\nnumAs = logData.filter(logData.value.contains('a')).count()\nnumBs = logData.filter(logData.value.contains('b')).count()\n\nprint(\"Lines with a: %i, lines with b: %i\" % (numAs, numBs))\n\nspark.stop()\n```\n\nExample:\n```bash\n# Use spark-submit to run your application\n$ YOUR_SPARK_HOME/bin/spark-submit \\\n  --master \"local[4]\" \\\n  SimpleApp.py\n...\nLines with a: 46, Lines with b: 23\n```\n\nExample:\n```bash\n# Use the Python interpreter to run your application\n$ python SimpleApp.py\n...\nLines with a: 46, Lines with b: 23\n```\n\nExample:\n```scala\n/* SimpleApp.scala */\nimport org.apache.spark.sql.SparkSession\n\nobject SimpleApp {\n  def main(args: Array[String]): Unit = {\n    val logFile = \"YOUR_SPARK_HOME/README.md\" // Should be some file on your system\n    val spark = SparkSession.builder.appName(\"Simple Application\").getOrCreate()\n    val logData = spark.read.textFile(logFile).cache()\n    val numAs = logData.filter(line => line.contains(\"a\")).count()\n    val numBs = logData.filter(line => line.contains(\"b\")).count()\n    println(s\"Lines with a: $numAs, Lines with b: $numBs\")\n    spark.stop()\n  }\n}\n```\n\nExample:\n```scala\nname := \"Simple Project\"\n\nversion := \"1.0\"\n\nscalaVersion := \"2.13.18\"\n\nlibraryDependencies += \"org.apache.spark\" %% \"spark-sql\" % \"4.2.0\"\n```\n\nExample:\n```bash\n# Your directory layout should look like this\n$ find .\n.\n./build.sbt\n./src\n./src/main\n./src/main/scala\n./src/main/scala/SimpleApp.scala\n\n# Package a jar containing your application\n$ sbt package\n...\n[info] Packaging {..}/{..}/target/scala-2.13/simple-project_2.13-1.0.jar\n\n# Use spark-submit to run your application\n$ YOUR_SPARK_HOME/bin/spark-submit \\\n  --class \"SimpleApp\" \\\n  --master \"local[4]\" \\\n  target/scala-2.13/simple-project_2.13-1.0.jar\n...\nLines with a: 46, Lines with b: 23\n```\n\nExample:\n```java\n/* SimpleApp.java */\nimport org.apache.spark.sql.SparkSession;\nimport org.apache.spark.sql.Dataset;\n\npublic class SimpleApp {\n  public static void main(String[] args) {\n    String logFile = \"YOUR_SPARK_HOME/README.md\"; // Should be some file on your system\n    SparkSession spark = SparkSession.builder().appName(\"Simple Application\").getOrCreate();\n    Dataset<String> logData = spark.read().textFile(logFile).cache();\n\n    long numAs = logData.filter(s -> s.contains(\"a\")).count();\n    long numBs = logData.filter(s -> s.contains(\"b\")).count();\n\n    System.out.println(\"Lines with a: \" + numAs + \", lines with b: \" + numBs);\n\n    spark.stop();\n  }\n}\n```\n\nExample:\n```xml\n<project>\n  <groupId>edu.berkeley</groupId>\n  <artifactId>simple-project</artifactId>\n  <modelVersion>4.0.0</modelVersion>\n  <name>Simple Project</name>\n  <packaging>jar</packaging>\n  <version>1.0</version>\n  <dependencies>\n    <dependency> <!-- Spark dependency -->\n      <groupId>org.apache.spark</groupId>\n      <artifactId>spark-sql_2.13</artifactId>\n      <version>4.2.0</version>\n      <scope>provided</scope>\n    </dependency>\n  </dependencies>\n</project>\n```\n\nExample:\n```bash\n$ find .\n./pom.xml\n./src\n./src/main\n./src/main/java\n./src/main/java/SimpleApp.java\n```\n\nExample:\n```bash\n# Package a JAR containing your application\n$ mvn package\n...\n[INFO] Building jar: {..}/{..}/target/simple-project-1.0.jar\n\n# Use spark-submit to run your application\n$ YOUR_SPARK_HOME/bin/spark-submit \\\n  --class \"SimpleApp\" \\\n  --master \"local[4]\" \\\n  target/simple-project-1.0.jar\n...\nLines with a: 46, Lines with b: 23\n```\n\nExample:\n```bash\n# For Python examples, use spark-submit directly:\n./bin/spark-submit examples/src/main/python/pi.py\n\n# For Scala and Java, use run-example:\n./bin/run-example SparkPi\n\n# For R examples, use spark-submit directly:\n./bin/spark-submit examples/src/main/r/dataframe.R\n```\n\n<|endofdoc|>","metadata":{"transformedAt":"2026-08-18T15:21:42.958Z","totalSectionsIncluded":1,"totalCodeBlocksIncluded":32,"totalLines":314,"estimatedTokens":5170}}8{"id":"doc-pyspark_overview_pyspark_4_2_0_documentation-d2e72363","source":"documentation","title":"PySpark Overview — PySpark 4.2.0 documentation","url":"https://spark.apache.org/docs/latest/api/python/index.html","text":"Site Navigation Overview Getting Started Tutorials User Guide API Reference Development More Migration Guides GitHub PyPI PySpark Overview# 11, 2026 Useful Notebook | GitHub | Issues | Examples | Community | Stack Overflow | Dev Mailing List | User Mailing List PySpark is the Python API for Apache Spark. It enables you to perform real-time, large-scale data processing in a distributed environment using Python. It also provides a PySpark shell for interactively analyzing your data. PySpark combines Python’s learnability and ease of use with the power of Apache Spark to enable processing and analysis of data at any size for everyone familiar with Python. PySpark supports all of Spark’s features such as Spark SQL, DataFrames, Structured Streaming, Machine Learning (MLlib), Pipelines and Spark Core. Python Spark Connect Client Spark Connect is a client-server architecture within Apache Spark that enables remote connectivity to Spark clusters from any application. PySpark provides the client for the Spark Connect server, allowing Spark to be used as a service. Connect Live Connect Spark Connect Overview Eager vs Connect vs Spark Classic Spark SQL and DataFrames Spark SQL is Apache Spark’s module for working with structured data. It allows you to seamlessly mix SQL queries with Spark programs. With PySpark DataFrames you can efficiently read, write, transform, and analyze data using Python and SQL. Whether you use Python or SQL, the same underlying execution engine is used so you will always leverage the full power of Spark. Live Spark SQL API Reference Pandas API on Spark Pandas API on Spark allows you to scale your pandas workload to any size by running it distributed across multiple nodes. If you are already familiar with pandas and want to leverage Spark for big data, pandas API on Spark makes you immediately productive and lets you migrate your applications without modifying the code. You can have a single codebase that works both with pandas (tests, smaller datasets) and with Spark (production, distributed datasets) and you can switch between the pandas API and the Pandas API on Spark easily and without overhead. Pandas API on Spark aims to make the transition from pandas to Spark easy but if you are new to Spark or deciding which API to use, we recommend using PySpark (see Spark SQL and DataFrames). API on Spark Live API on Spark Pandas API on Spark Reference Structured Streaming Structured Streaming is a scalable and fault-tolerant stream processing engine built on the Spark SQL engine. You can express your streaming computation the same way you would express a batch computation on static data. The Spark SQL engine will take care of running it incrementally and continuously and updating the final result as streaming data continues to arrive. Structured Streaming Programming Guide Structured Streaming API Reference Machine Learning (MLlib) Built on top of Spark, MLlib is a scalable machine learning library that provides a uniform set of high-level APIs that help users create and tune practical machine learning pipelines. Machine Learning Library (MLlib) Programming Guide Machine Learning (MLlib) API Reference Declarative Pipelines Spark Declarative Pipelines (SDP) is a declarative framework for building reliable, maintainable, and testable data pipelines on Spark. SDP simplifies ETL development by allowing you to focus on the transformations you want to apply to your data, rather than the mechanics of pipeline execution. Pipelines API Reference Spark Core and RDDs Spark Core is the underlying general execution engine for the Spark platform that all other functionality is built on top of. It provides RDDs (Resilient Distributed Datasets) and in-memory computing capabilities. Note that the RDD API is a low-level API which can be difficult to use and you do not get the benefit of Spark’s automatic query optimization capabilities. We recommend using DataFrames (see Spark SQL and DataFrames above) instead of RDDs as it allows you to express what you want more easily and lets Spark automatically construct the most efficient query for you. Spark Core API Reference Spark Streaming (Legacy) Spark Streaming is an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of live data streams. Note that Spark Streaming is the previous generation of Spark’s streaming engine. It is a legacy project and it is no longer being updated. There is a newer and easier to use streaming engine in Spark called Structured Streaming which you should use for your streaming applications and pipelines. Spark Streaming Programming Guide (Legacy) Spark Streaming API Reference (Legacy) next Getting Started Show Source\n\n<|endofdoc|>","metadata":{"transformedAt":"2026-08-18T15:21:42.959Z","totalSectionsIncluded":1,"totalCodeBlocksIncluded":0,"totalLines":3,"estimatedTokens":1184}}9{"id":"doc-r_front_end_for_apache_spark_sparkr-e32c419a","source":"documentation","title":"R Front End for Apache Spark • SparkR","url":"https://spark.apache.org/docs/latest/api/R/index.html","text":"R on Spark (deprecated) SparkR is an R package that provides a light-weight frontend to use Spark from R. Installing sparkR Libraries of sparkR need to be created in $SPARK_HOME/R/lib. This can be done by running the script $SPARK_HOME/R/install-dev.sh. By default the above script uses the system wide installation of R. However, this can be changed to any user installed location of R by setting the environment variable R_HOME the full path of the base directory where R is installed, before running install-dev.sh script. Example: # where /home/username/R is where R is installed and /home/username/R/bin contains the files R and RScript export R_HOME=/home/username/R ./install-dev.sh SparkR development Build Spark Build Spark with Maven or SBT, and include the -Psparkr profile to build the R package. For example to use the default Hadoop versions you can run # Maven ./build/mvn -DskipTests -Psparkr package # SBT ./build/sbt -Psparkr package Running sparkR You can start using SparkR by launching the SparkR shell with ./bin/sparkR The sparkR script automatically creates a SparkContext with Spark by default in local mode. To specify the Spark master of a cluster for the automatically created SparkContext, you can run ./bin/sparkR --master \"local[2]\" To set other options like driver memory, executor memory etc. you can pass in the spark-submit arguments to ./bin/sparkR Using SparkR from RStudio If you wish to use SparkR from RStudio, please refer SparkR documentation. Making changes to SparkR The instructions for making contributions to Spark also apply to SparkR. If you only make R file changes (i.e. no Scala changes) then you can just re-install the R package using R/install-dev.sh and test your changes. Once you have made your changes, please include unit tests for them and run existing unit tests using the R/run-tests.sh script as described below. Generating documentation The SparkR documentation (Rd files and HTML files) are not a part of the source repository. To generate them you can run the script R/create-docs.sh. This script uses roxygen2, knitr, and rmarkdown to generate the docs and these packages need to be installed on the machine before using the script. Also, you may need to install these prerequisites. See also, R/DOCUMENTATION.md Examples, Unit tests SparkR comes with several sample programs in the examples/src/main/r directory. To run one of them, use ./bin/spark-submit <filename> <args>. For /bin/spark-submit examples/src/main/r/dataframe.R You can run R unit tests by following the instructions under Running R Tests. Running on YARN The ./bin/spark-submit can also be used to submit jobs to YARN clusters. You will need to set YARN conf dir before doing so. For example on CDH you can run export YARN_CONF_DIR=/etc/hadoop/conf ./bin/spark-submit --master yarn examples/src/main/r/dataframe.R Links View on CRAN Report a bug License Apache License (== 2.0) Citation Citing SparkR Developers The Apache Software Foundation Author, maintainer, copyright holder Developed by The Apache Software Foundation. Site built with pkgdown 2.0.1. Using preferably template.\n\nExample:\n```bash\n# where /home/username/R is where R is installed and /home/username/R/bin contains the files R and RScript\nexport R_HOME=/home/username/R\n./install-dev.sh\n```\n\nExample:\n```bash\n# Maven\n./build/mvn -DskipTests -Psparkr package\n\n# SBT\n./build/sbt -Psparkr package\n```\n\nExample:\n```text\n./bin/sparkR\n```\n\nExample:\n```text\n./bin/sparkR --master \"local[2]\"\n```\n\nExample:\n```bash\n./bin/spark-submit examples/src/main/r/dataframe.R\n```\n\nExample:\n```bash\nexport YARN_CONF_DIR=/etc/hadoop/conf\n./bin/spark-submit --master yarn examples/src/main/r/dataframe.R\n```\n\n<|endofdoc|>","metadata":{"transformedAt":"2026-08-18T15:21:42.959Z","totalSectionsIncluded":1,"totalCodeBlocksIncluded":6,"totalLines":40,"estimatedTokens":927}}10{"id":"doc-overview_spark_4_2_0_javadoc-70297883","source":"documentation","title":"Overview (Spark 4.2.0 JavaDoc)","url":"https://spark.apache.org/docs/latest/api/java/index.html","text":"Skip navigation links Overview Package Class Deprecated Index Help SEARCH:\n\nPackages Package Description org.apache.datasketches.memory.internal org.apache.parquet.filter2.predicate org.apache.spark Core Spark classes in Scala. org.apache.spark.api.java Spark Java programming APIs. org.apache.spark.api.java.function Set of interfaces to represent functions in Spark's Java API. org.apache.spark.api.plugin org.apache.spark.api.resource org.apache.spark.broadcast Spark's broadcast variables, used to broadcast immutable datasets to all nodes. org.apache.spark.graphx ALPHA COMPONENT GraphX is a graph processing framework built on top of Spark. org.apache.spark.graphx.impl org.apache.spark.graphx.lib Various analytics functions for graphs. org.apache.spark.graphx.util Collections of utilities used by graphx. org.apache.spark.input org.apache.spark.io IO codecs used for compression. org.apache.spark.launcher Library for launching Spark applications programmatically. org.apache.spark.mapred org.apache.spark.metrics org.apache.spark.metrics.source org.apache.spark.ml DataFrame-based machine learning APIs to let users quickly assemble and configure practical machine learning pipelines. org.apache.spark.ml.attribute ML attributes org.apache.spark.ml.classification org.apache.spark.ml.clustering org.apache.spark.ml.evaluation org.apache.spark.ml.feature Feature transformers The `ml.feature` package provides common feature transformers that help convert raw data or features into more suitable forms for model fitting. org.apache.spark.ml.fpm org.apache.spark.ml.image org.apache.spark.ml.linalg org.apache.spark.ml.param org.apache.spark.ml.param.shared org.apache.spark.ml.recommendation org.apache.spark.ml.regression org.apache.spark.ml.source.image org.apache.spark.ml.source.libsvm org.apache.spark.ml.stat org.apache.spark.ml.stat.distribution org.apache.spark.ml.tree org.apache.spark.ml.tuning org.apache.spark.ml.util org.apache.spark.mllib RDD-based machine learning APIs (in maintenance mode). org.apache.spark.mllib.classification org.apache.spark.mllib.clustering org.apache.spark.mllib.evaluation org.apache.spark.mllib.feature org.apache.spark.mllib.fpm org.apache.spark.mllib.linalg org.apache.spark.mllib.linalg.distributed org.apache.spark.mllib.optimization org.apache.spark.mllib.pmml org.apache.spark.mllib.random org.apache.spark.mllib.rdd org.apache.spark.mllib.recommendation org.apache.spark.mllib.regression org.apache.spark.mllib.stat org.apache.spark.mllib.stat.distribution org.apache.spark.mllib.stat.test org.apache.spark.mllib.tree org.apache.spark.mllib.tree.configuration org.apache.spark.mllib.tree.impurity org.apache.spark.mllib.tree.loss org.apache.spark.mllib.tree.model org.apache.spark.mllib.util org.apache.spark.partial org.apache.spark.paths org.apache.spark.rdd Provides implementation's of various RDDs. org.apache.spark.resource org.apache.spark.scheduler Spark's DAG scheduler. org.apache.spark.scheduler.cluster org.apache.spark.security org.apache.spark.serializer Pluggable serializers for RDD and shuffle data. org.apache.spark.shuffle.api org.apache.spark.shuffle.api.metadata org.apache.spark.sql org.apache.spark.sql.api.java Allows the execution of relational queries, including those expressed in SQL using Spark. org.apache.spark.sql.catalog org.apache.spark.sql.columnar org.apache.spark.sql.connector org.apache.spark.sql.connector.catalog org.apache.spark.sql.connector.catalog.constraints org.apache.spark.sql.connector.catalog.functions org.apache.spark.sql.connector.catalog.index org.apache.spark.sql.connector.catalog.procedures org.apache.spark.sql.connector.catalog.transactions org.apache.spark.sql.connector.distributions org.apache.spark.sql.connector.expressions org.apache.spark.sql.connector.expressions.aggregate org.apache.spark.sql.connector.expressions.filter org.apache.spark.sql.connector.join org.apache.spark.sql.connector.metric org.apache.spark.sql.connector.read org.apache.spark.sql.connector.read.colstats org.apache.spark.sql.connector.read.partitioning org.apache.spark.sql.connector.read.streaming org.apache.spark.sql.connector.util org.apache.spark.sql.connector.write org.apache.spark.sql.connector.write.streaming org.apache.spark.sql.expressions org.apache.spark.sql.expressions.javalang org.apache.spark.sql.expressions.scalalang org.apache.spark.sql.jdbc org.apache.spark.sql.protobuf org.apache.spark.sql.sources org.apache.spark.sql.streaming org.apache.spark.sql.types org.apache.spark.sql.util org.apache.spark.sql.vectorized org.apache.spark.status.api.v1 org.apache.spark.status.api.v1.sql org.apache.spark.status.api.v1.streaming org.apache.spark.status.protobuf org.apache.spark.storage org.apache.spark.streaming org.apache.spark.streaming.api.java Java APIs for spark streaming. org.apache.spark.streaming.dstream Various implementations of DStreams. org.apache.spark.streaming.kinesis org.apache.spark.streaming.receiver org.apache.spark.streaming.scheduler org.apache.spark.streaming.scheduler.rate org.apache.spark.streaming.util org.apache.spark.ui.jobs org.apache.spark.unsafe.types org.apache.spark.util Spark utilities. org.apache.spark.util.random Utilities for random number generation. org.apache.spark.util.sketch\n\n<|endofdoc|>","metadata":{"transformedAt":"2026-08-18T15:21:42.960Z","totalSectionsIncluded":2,"totalCodeBlocksIncluded":0,"totalLines":5,"estimatedTokens":1315}}11{"id":"doc-spark_declarative_pipelines_programming_guide_sp-99e82fa1","source":"documentation","title":"Spark Declarative Pipelines Programming Guide - Spark 4.2.0 Documentation","url":"https://spark.apache.org/docs/latest/declarative-pipelines-programming-guide.html","text":"Spark Declarative Pipelines Programming Guide What is Spark Declarative Pipelines (SDP)? Quick install Key Concepts Flows Datasets Pipelines Pipeline Projects The spark-pipelines Command Line Interface spark-pipelines init spark-pipelines run Refresh Selection Behavior Examples spark-pipelines dry-run Programming with SDP in Python The Spark Session in Python Pipelines Creating a Materialized View in Python Creating a Temporary View in Python Creating a Streaming Table in Python Loading Data from Streaming Sources in Python Loading Data from Batch Sources in Python Querying Tables Defined in a Pipeline in Python Creating Tables in For Loop in Python Using Multiple Flows to Write to a Single Target in Python Programming with SDP in SQL Creating a Materialized View in SQL Creating a Temporary View in SQL Creating a Streaming Table in SQL Querying Tables Defined in a Pipeline in SQL Using Multiple Flows to Write to a Single Target in SQL Writing Data to External Targets with Sinks Creating and Using Sinks in Python Creating a Kafka Sink Sink Considerations Important Considerations Python Considerations SQL Considerations What is Spark Declarative Pipelines (SDP)? Spark Declarative Pipelines (SDP) is a declarative framework for building reliable, maintainable, and testable data pipelines on Apache Spark. SDP simplifies ETL development by allowing you to focus on the transformations you want to apply to your data, rather than the mechanics of pipeline execution. SDP is designed for both batch and streaming data processing, supporting common use cases such ingestion from cloud storage (Amazon S3, Azure ADLS Gen2, Google Cloud Storage) Data ingestion from message buses (Apache Kafka, Amazon Kinesis, Google Pub/Sub, Azure EventHub) Incremental batch and streaming transformations The key advantage of SDP is its declarative approach - you define what tables should exist and what their contents should be, and SDP handles the orchestration, compute management, and error handling automatically. Quick install A quick way to install SDP is with install pyspark[pipelines] See the downloads page for more installation options. Key Concepts Flows A flow is the foundational data processing concept in SDP which supports both streaming and batch semantics. A flow reads data from a source, applies user-defined processing logic, and writes the result into a target dataset. For example, when you author a query STREAMING TABLE target_table AS SELECT * FROM STREAM source_table SDP creates the table named target_table along with a flow that reads new data from source_table and writes it to target_table. Datasets A dataset is a queryable object that’s the output of one of more flows within a pipeline. Flows in the pipeline can also read from datasets produced in the pipeline. Streaming Table – a definition of a table and one or more streaming flows written into it. Streaming tables support incremental processing of data, allowing you to process only new data as it arrives. Materialized View – a view that is precomputed into a table. A materialized view always has exactly one batch flow writing to it. Temporary View – a view that is scoped to an execution of the pipeline. It can be referenced from flows within the pipeline. It’s useful for encapsulating transformations and intermediate logical entities that multiple other elements of the pipeline depend on. Pipelines A pipeline is the primary unit of development and execution in SDP. A pipeline can contain one or more flows, streaming tables, and materialized views. While your pipeline runs, it analyzes the dependencies of your defined objects and orchestrates their order of execution and parallelization automatically. Pipeline Projects A pipeline project is a set of source files that contain code definitions of the datasets and flows that make up a pipeline. The source files can be _customer_orders\") def regional_customer_orders(region_filter=region) -> = spark.table(\"customer_orders\") nation_region = spark.table(\"nation_region\") return ( customer_orders '\") ) Using Multiple Flows to Write to a Single Target in Python You can create multiple flows that append data to the same pyspark import pipelines as dp from pyspark.sql import DataFrame # create a streaming table dp.create_streaming_table(\"customers_us\") # define the first append flow @dp.append_flow(target = \"customers_us\") def append_customers_us_west() -> spark.readStream.table(\"customers_us_west\") # define the second append flow @dp.append_flow(target = \"customers_us\") def append_customers_us_east() -> spark.readStream.table(\"customers_us_east\") Programming with SDP in SQL Creating a Materialized View in SQL The basic syntax for creating a materialized view with SQL MATERIALIZED VIEW basic_mv AS SELECT * FROM samples.nyctaxi.trips; Creating a Temporary View in SQL The basic syntax for creating a temporary view with SQL TEMPORARY VIEW basic_tv AS SELECT * FROM samples.nyctaxi.trips; Creating a Streaming Table in SQL When creating a streaming table, use the STREAM keyword to indicate streaming semantics for the STREAMING TABLE basic_st AS SELECT * FROM STREAM samples.nyctaxi.trips; Querying Tables Defined in a Pipeline in SQL You can reference other tables defined in your STREAMING TABLE orders AS SELECT * FROM STREAM orders_source; CREATE MATERIALIZED VIEW customers AS SELECT * FROM customers_source; CREATE MATERIALIZED VIEW customer_orders AS SELECT c.customer_id, o.order_number, c.state, date(timestamp(int(o.order_datetime))) order_date FROM orders o INNER JOIN customers c ON o.customer_id = c.customer_id; CREATE MATERIALIZED VIEW daily_orders_by_state AS SELECT state, order_date, count(*) order_count FROM customer_orders GROUP BY state, order_date; Using Multiple Flows to Write to a Single Target in SQL You can create multiple flows that append data to the same create a streaming table CREATE STREAMING TABLE customers_us; -- define the first append flow CREATE FLOW append_customers_us_west AS INSERT INTO customers_us SELECT * FROM STREAM(customers_us_west); -- define the second append flow CREATE FLOW append_customers_us_east AS INSERT INTO customers_us SELECT * FROM STREAM(customers_us_east); Writing Data to External Targets with Sinks Sinks in SDP provide a way to write transformed data to external destinations beyond the default streaming tables and materialized views. Sinks are particularly useful for operational use cases that require low-latency data processing, reverse ETL operations, or writing to external systems. Sinks enable a pipeline to write to any destination that a Spark Structured Streaming query can be written to, including, but not limited to, Apache Kafka and Azure Event Hubs. Creating and Using Sinks in Python Working with sinks involves two main the sink definition and implementing an append flow to write data. Creating a Kafka Sink You can create a sink that streams data to a Kafka pyspark import pipelines as dp from pyspark.sql.functions import to_json, struct dp.create_sink( name=\"kafka_sink\", format=\"kafka\", options={ \"kafka.bootstrap.servers\": \"localhost:9092\", \"topic\": \"processed_orders\" } ) @dp.append_flow(target=\"kafka_sink\") def kafka_orders_flow() -> ( spark.readStream.table(\"customer_orders\") .select( col(\"order_id\").cast(\"string\").alias(\"key\"), to_json(struct(\"*\")).alias(\"value\") ) ) Sink Considerations When working with sinks, keep the following considerations in : Sinks currently support only streaming queries through append_flow decorators Python functionality is available only through the Python API, not SQL append operations are supported; full refresh updates reset checkpoints but do not clean previously computed results Important Considerations Python Considerations SDP evaluates the code that defines a pipeline multiple times during planning and pipeline runs. Python functions that define datasets should include only the code required to define the table or view. The function used to define a dataset must return a pyspark.sql.DataFrame. Never use methods that save or write to files or tables as part of your SDP dataset code. When using the for loop pattern to define datasets in Python, ensure that the list of values passed to the for loop is always additive. Examples of Spark SQL operations that should never be used in SDP () count() pivot() toPandas() save() saveAsTable() start() toTable() SQL Considerations The PIVOT clause is not supported in SDP SQL.\n\nExample:\n```text\npip install pyspark[pipelines]\n```\n\nExample:\n```text\nCREATE STREAMING TABLE target_table AS\nSELECT * FROM STREAM source_table\n```\n\nExample:\n```text\nname: my_pipeline\nlibraries:\n  - glob:\n      include: transformations/**\nstorage: file:///absolute/path/to/storage/dir\ncatalog: my_catalog\ndatabase: my_db\nconfiguration:\n  spark.sql.shuffle.partitions: \"1000\"\n```\n\nExample:\n```text\n# Basic run with default incremental update\nspark-pipelines run\n\n# Run with specific spec file\nspark-pipelines run --spec /path/to/my-pipeline.yaml\n\n# Full refresh of specific datasets\nspark-pipelines run --full-refresh orders,customers\n\n# Full refresh of entire pipeline\nspark-pipelines run --full-refresh-all\n\n# Run with custom Spark configuration\nspark-pipelines run --conf spark.sql.shuffle.partitions=200 --driver-memory 4g\n\n# Run on remote Spark Connect server\nspark-pipelines run --remote sc://my-cluster:15002\n```\n\nExample:\n```text\nfrom pyspark import pipelines as dp\n```\n\nExample:\n```text\nfrom pyspark import pipelines as dp\n\n@dp.materialized_view\ndef my_view():\n    return spark.range(10)\n```\n\nExample:\n```text\nfrom pyspark import pipelines as dp\nfrom pyspark.sql import SparkSession\n\nspark: SparkSession\n\n@dp.materialized_view\ndef my_view():\n    return spark.range(10)\n```\n\nExample:\n```text\nfrom pyspark import pipelines as dp\nfrom pyspark.sql import DataFrame\n\n@dp.materialized_view\ndef basic_mv() -> DataFrame:\n    return spark.table(\"samples.nyctaxi.trips\")\n```\n\nExample:\n```text\nfrom pyspark import pipelines as dp\nfrom pyspark.sql import DataFrame\n\n@dp.materialized_view(name=\"trips_mv\")\ndef basic_mv() -> DataFrame:\n    return spark.table(\"samples.nyctaxi.trips\")\n```\n\nExample:\n```text\nfrom pyspark import pipelines as dp\nfrom pyspark.sql import DataFrame\n\n@dp.temporary_view\ndef basic_tv() -> DataFrame:\n    return spark.table(\"samples.nyctaxi.trips\")\n```\n\nExample:\n```text\nfrom pyspark import pipelines as dp\nfrom pyspark.sql import DataFrame\n\n@dp.table\ndef basic_st() -> DataFrame:\n    return spark.readStream.table(\"samples.nyctaxi.trips\")\n```\n\nExample:\n```text\nfrom pyspark import pipelines as dp\nfrom pyspark.sql import DataFrame\n\n@dp.table\ndef ingestion_st() -> DataFrame:\n    return (\n        spark.readStream\n        .format(\"kafka\")\n        .option(\"kafka.bootstrap.servers\", \"localhost:9092\")\n        .option(\"subscribe\", \"orders\")\n        .load()\n    )\n```\n\nExample:\n```text\nfrom pyspark import pipelines as dp\nfrom pyspark.sql import DataFrame\n\n@dp.materialized_view\ndef batch_mv() -> DataFrame:\n    return spark.read.format(\"json\").load(\"/datasets/retail-org/sales_orders\")\n```\n\nExample:\n```text\nfrom pyspark import pipelines as dp\nfrom pyspark.sql import DataFrame\nfrom pyspark.sql.functions import col\n\n@dp.table\ndef orders() -> DataFrame:\n    return (\n        spark.readStream\n        .format(\"kafka\")\n        .option(\"kafka.bootstrap.servers\", \"localhost:9092\")\n        .option(\"subscribe\", \"orders\")\n        .load()\n    )\n\n@dp.materialized_view\ndef customers() -> DataFrame:\n    return (\n        spark.read\n        .format(\"csv\")\n        .option(\"header\", True)\n        .load(\"/datasets/retail-org/customers\")\n    )\n\n@dp.materialized_view\ndef customer_orders() -> DataFrame:\n    return (\n        spark.table(\"orders\")\n        .join(\n            spark.table(\"customers\"), \"customer_id\")\n            .select(\n                \"customer_id\",\n                \"order_number\",\n                \"state\",\n                col(\"order_datetime\").cast(\"date\").alias(\"order_date\"),\n            )\n        )\n    )\n\n@dp.materialized_view\ndef daily_orders_by_state() -> DataFrame:\n    return (\n        spark.table(\"customer_orders\")\n        .groupBy(\"state\", \"order_date\")\n        .count()\n        .withColumnRenamed(\"count\", \"order_count\")\n    )\n```\n\nExample:\n```text\nfrom pyspark import pipelines as dp\nfrom pyspark.sql import DataFrame\nfrom pyspark.sql.functions import collect_list, col\n\n@dp.temporary_view()\ndef customer_orders() -> DataFrame:\n    orders = spark.table(\"samples.tpch.orders\")\n    customer = spark.table(\"samples.tpch.customer\")\n\n    return (\n        orders\n        .join(customer, orders.o_custkey == customer.c_custkey)\n        .select(\n            col(\"c_custkey\").alias(\"custkey\"),\n            col(\"c_name\").alias(\"name\"),\n            col(\"c_nationkey\").alias(\"nationkey\"),\n            col(\"c_phone\").alias(\"phone\"),\n            col(\"o_orderkey\").alias(\"orderkey\"),\n            col(\"o_orderstatus\").alias(\"orderstatus\"),\n            col(\"o_totalprice\").alias(\"totalprice\"),\n            col(\"o_orderdate\").alias(\"orderdate\"),\n        )\n    )\n\n@dp.temporary_view()\ndef nation_region() -> DataFrame:\n    nation = spark.table(\"samples.tpch.nation\")\n    region = spark.table(\"samples.tpch.region\")\n\n    return (\n        nation\n        .join(region, nation.n_regionkey == region.r_regionkey)\n        .select(\n            col(\"n_name\").alias(\"nation\"),\n            col(\"r_name\").alias(\"region\"),\n            col(\"n_nationkey\").alias(\"nationkey\"),\n        )\n    )\n\n# Extract region names from region table\nregion_list = spark.table(\"samples.tpch.region\").select(collect_list(\"r_name\")).collect()[0][0]\n\n# Iterate through region names to create new region-specific materialized views\nfor region in region_list:\n    @dp.table(name=f\"{region.lower().replace(' ', '_')}_customer_orders\")\n    def regional_customer_orders(region_filter=region) -> DataFrame:\n        customer_orders = spark.table(\"customer_orders\")\n        nation_region = spark.table(\"nation_region\")\n\n        return (\n            customer_orders\n            .join(nation_region, customer_orders.nationkey == nation_region.nationkey)\n            .select(\n                col(\"custkey\"),\n                col(\"name\"),\n                col(\"phone\"),\n                col(\"nation\"),\n                col(\"region\"),\n                col(\"orderkey\"),\n                col(\"orderstatus\"),\n                col(\"totalprice\"),\n                col(\"orderdate\"),\n            )\n            .filter(f\"region = '{region_filter}'\")\n        )\n```\n\nExample:\n```text\nfrom pyspark import pipelines as dp\nfrom pyspark.sql import DataFrame\n\n# create a streaming table\ndp.create_streaming_table(\"customers_us\")\n\n# define the first append flow\n@dp.append_flow(target = \"customers_us\")\ndef append_customers_us_west() -> DataFrame:\n    return spark.readStream.table(\"customers_us_west\")\n\n# define the second append flow\n@dp.append_flow(target = \"customers_us\")\ndef append_customers_us_east() -> DataFrame:\n    return spark.readStream.table(\"customers_us_east\")\n```\n\nExample:\n```text\nCREATE MATERIALIZED VIEW basic_mv\nAS SELECT * FROM samples.nyctaxi.trips;\n```\n\nExample:\n```text\nCREATE TEMPORARY VIEW basic_tv\nAS SELECT * FROM samples.nyctaxi.trips;\n```\n\nExample:\n```text\nCREATE STREAMING TABLE basic_st\nAS SELECT * FROM STREAM samples.nyctaxi.trips;\n```\n\nExample:\n```text\nCREATE STREAMING TABLE orders\nAS SELECT * FROM STREAM orders_source;\n\nCREATE MATERIALIZED VIEW customers\nAS SELECT * FROM customers_source;\n\nCREATE MATERIALIZED VIEW customer_orders\nAS SELECT\n  c.customer_id,\n  o.order_number,\n  c.state,\n  date(timestamp(int(o.order_datetime))) order_date\nFROM orders o\nINNER JOIN customers c\nON o.customer_id = c.customer_id;\n\nCREATE MATERIALIZED VIEW daily_orders_by_state\nAS SELECT state, order_date, count(*) order_count\nFROM customer_orders\nGROUP BY state, order_date;\n```\n\nExample:\n```text\n-- create a streaming table\nCREATE STREAMING TABLE customers_us;\n\n-- define the first append flow\nCREATE FLOW append_customers_us_west\nAS INSERT INTO customers_us\nSELECT * FROM STREAM(customers_us_west);\n\n-- define the second append flow\nCREATE FLOW append_customers_us_east\nAS INSERT INTO customers_us\nSELECT * FROM STREAM(customers_us_east);\n```\n\nExample:\n```text\nfrom pyspark import pipelines as dp\nfrom pyspark.sql.functions import to_json, struct\n\ndp.create_sink(\n    name=\"kafka_sink\",\n    format=\"kafka\",\n    options={\n        \"kafka.bootstrap.servers\": \"localhost:9092\",\n        \"topic\": \"processed_orders\"\n    }\n)\n\n@dp.append_flow(target=\"kafka_sink\")\ndef kafka_orders_flow() -> DataFrame:\n    return (\n        spark.readStream.table(\"customer_orders\")\n        .select(\n            col(\"order_id\").cast(\"string\").alias(\"key\"),\n            to_json(struct(\"*\")).alias(\"value\")\n        )\n    )\n```\n\n<|endofdoc|>","metadata":{"transformedAt":"2026-08-18T15:21:42.971Z","totalSectionsIncluded":1,"totalCodeBlocksIncluded":22,"totalLines":361,"estimatedTokens":4180}}12{"id":"doc-hardware_provisioning_spark_4_2_0_documentation-ac0a8b48","source":"documentation","title":"Hardware Provisioning - Spark 4.2.0 Documentation","url":"https://spark.apache.org/docs/latest/hardware-provisioning.html","text":"Hardware Provisioning A common question received by Spark developers is how to configure hardware for it. While the right hardware will depend on the situation, we make the following recommendations. Storage Systems Because most Spark jobs will likely have to read input data from an external storage system (e.g. the Hadoop File System, or HBase), it is important to place it as close to this system as possible. We recommend the at all possible, run Spark on the same nodes as HDFS. The simplest way is to set up a Spark standalone mode cluster on the same nodes, and configure Spark and Hadoop’s memory and CPU usage to avoid interference (for Hadoop, the relevant options are mapred.child.java.opts for the per-task memory and mapreduce.tasktracker.map.tasks.maximum and mapreduce.tasktracker.reduce.tasks.maximum for number of tasks). Alternatively, you can run Hadoop and Spark on a common cluster manager like Hadoop YARN. If this is not possible, run Spark on different nodes in the same local-area network as HDFS. For low-latency data stores like HBase, it may be preferable to run computing jobs on different nodes than the storage system to avoid interference. Local Disks While Spark can perform a lot of its computation in memory, it still uses local disks to store data that doesn’t fit in RAM, as well as to preserve intermediate output between stages. We recommend having 4-8 disks per node, configured without RAID (just as separate mount points). In Linux, mount the disks with the noatime option to reduce unnecessary writes. In Spark, configure the spark.local.dir variable to be a comma-separated list of the local disks. If you are running HDFS, it’s fine to use the same disks as HDFS. Memory In general, Spark can run well with anywhere from 8 GiB to hundreds of gigabytes of memory per machine. In all cases, we recommend allocating only at most 75% of the memory for Spark; leave the rest for the operating system and buffer cache. How much memory you will need will depend on your application. To determine how much your application uses for a certain dataset size, load part of your dataset in a Spark RDD and use the Storage tab of Spark’s monitoring UI (http://<driver-node>:4040) to see its size in memory. Note that memory usage is greatly affected by storage level and serialization format – see the tuning guide for tips on how to reduce it. Finally, note that the Java VM does not always behave well with more than 200 GiB of RAM. If you purchase machines with more RAM than this, you can launch multiple executors in a single node. In Spark’s standalone mode, a worker is responsible for launching multiple executors according to its available memory and cores, and each executor will be launched in a separate Java VM. Network In our experience, when the data is in memory, a lot of Spark applications are network-bound. Using a 10 Gigabit or higher network is the best way to make these applications faster. This is especially true for “distributed reduce” applications such as group-bys, reduce-bys, and SQL joins. In any given application, you can see how much data Spark shuffles across the network from the application’s monitoring UI (http://<driver-node>:4040). CPU Cores Spark scales well to tens of CPU cores per machine because it performs minimal sharing between threads. You should likely provision at least 8-16 cores per machine. Depending on the CPU cost of your workload, you may also need data is in memory, most applications are either CPU- or network-bound.\n\n<|endofdoc|>","metadata":{"transformedAt":"2026-08-18T15:21:42.971Z","totalSectionsIncluded":1,"totalCodeBlocksIncluded":0,"totalLines":3,"estimatedTokens":882}}13{"id":"doc-spark_4_2_0_scaladoc_org_apache_spark-83520ce4","source":"documentation","title":"Spark 4.2.0 ScalaDoc - org.apache.spark","url":"https://spark.apache.org/docs/latest/api/scala/org/apache/spark/index.html","text":"Spark 4.2.0 ScalaDoc < Back\n\nPackages  package rootDefinition Classesroot  package orgDefinition Classesroot  package apacheDefinition Classesorg  package datasketchesDefinition Classesapache  package parquetDefinition Classesapache  package sparkCore Spark functionality.Core Spark functionality. org.apache.spark.SparkContext serves as the main entry point to Spark, while org.apache.spark.rdd.RDD is the data type representing a distributed collection, and provides most parallel operations.In addition, org.apache.spark.rdd.PairRDDFunctions contains operations available only on RDDs of key-value pairs, such as groupByKey and join; org.apache.spark.rdd.DoubleRDDFunctions contains operations available only on RDDs of Doubles; and org.apache.spark.rdd.SequenceFileRDDFunctions contains operations available on RDDs that can be saved as SequenceFiles. These operations are automatically available on any RDD of the right type (e.g. RDD[(Int, Int)] through implicit conversions.Java programmers should reference the org.apache.spark.api.java package for Spark programming APIs in Java.Classes and methods marked with Experimental are user-facing features which have not been officially adopted by the Spark project. These are subject to change or removal in minor releases.Classes and methods marked with Developer API are intended for advanced users want to extend Spark through lower level interfaces. These are subject to changes or removal in minor releases. Definition Classesapache  package api  package broadcastSpark's broadcast variables, used to broadcast immutable datasets to all nodes.  package graphxALPHA COMPONENT GraphX is a graph processing framework built on top of Spark.  package input  package ioIO codecs used for compression.IO codecs used for compression. See org.apache.spark.io.CompressionCodec.  package launcher  package mapred  package metrics  package mlDataFrame-based machine learning APIs to let users quickly assemble and configure practical machine learning pipelines.  package mllibRDD-based machine learning APIs (in maintenance mode).RDD-based machine learning APIs (in maintenance mode).The spark.mllib package is in maintenance mode as of the Spark 2.0.0 release to encourage migration to the DataFrame-based APIs under the org.apache.spark.ml package. While in maintenance mode,no new features in the RDD-based spark.mllib package will be accepted, unless they block implementing new features in the DataFrame-based spark.ml package;bug fixes in the RDD-based APIs will still be accepted.The developers will continue adding more features to the DataFrame-based APIs in the 2.x series to reach feature parity with the RDD-based APIs. And once we reach feature parity, this package will be deprecated. See alsoSPARK-4591 to track the progress of feature parity  package partialSupport for approximate results.Support for approximate results. This provides convenient api and also implementation for approximate calculation. See alsoorg.apache.spark.rdd.RDD.countApprox  package paths  package rddProvides several RDD implementations.Provides several RDD implementations. See org.apache.spark.rdd.RDD.  package resource  package schedulerSpark's scheduling components.Spark's scheduling components. This includes the org.apache.spark.scheduler.DAGScheduler and lower level org.apache.spark.scheduler.TaskScheduler.  package security  package serializerPluggable serializers for RDD and shuffle data.Pluggable serializers for RDD and shuffle data. See alsoorg.apache.spark.serializer.Serializer  package shuffle  package sqlAllows the execution of relational queries, including those expressed in SQL using Spark.  package status  package storage  package streamingSpark Streaming functionality.Spark Streaming functionality. org.apache.spark.streaming.StreamingContext serves as the main entry point to Spark Streaming, while org.apache.spark.streaming.dstream.DStream is the data type representing a continuous sequence of RDDs, representing a continuous stream of data.In addition, org.apache.spark.streaming.dstream.PairDStreamFunctions contains operations available only on DStreams of key-value pairs, such as groupByKey and reduceByKey. These operations are automatically available on any DStream of the right type (e.g. DStream[(Int, Int)] through implicit conversions.For the Java API of Spark Streaming, take a look at the org.apache.spark.streaming.api.java.JavaStreamingContext which serves as the entry point, and the org.apache.spark.streaming.api.java.JavaDStream and the org.apache.spark.streaming.api.java.JavaPairDStream which have the DStream functionality.  package ui  package unsafe  package utilSpark utilities. Aggregator BarrierTaskContext BarrierTaskInfo BreakingChangeInfo ComplexFutureAction ContextAwareIterator Dependency ErrorClassesJsonReader ExceptionFailure ExecutorLostFailure FetchFailed FutureAction HashPartitioner InterruptibleIterator JobExecutionStatus JobSubmitter MitigationConfig NarrowDependency OneToOneDependency Partition PartitionEvaluator PartitionEvaluatorFactory Partitioner QueryContext QueryContextType RangeDependency RangePartitioner ReadOnlySparkConf Resubmitted SerializableWritable ShuffleDependency ShuffleStatusNotFoundException SimpleFutureAction SparkConf SparkContext SparkEnv SparkException SparkExecutorInfo SparkFiles SparkFirehoseListener SparkJobInfo SparkStageInfo SparkStatusTracker SparkThrowable StringSubstitutor Success TaskCommitDenied TaskContext TaskEndReason TaskFailedReason TaskKilled TaskKilledException TaskResultLost UnknownReason WritableConverter WritableFactoryporg.apachespark package sparkCore Spark functionality. org.apache.spark.SparkContext serves as the main entry point to Spark, while org.apache.spark.rdd.RDD is the data type representing a distributed collection, and provides most parallel operations.In addition, org.apache.spark.rdd.PairRDDFunctions contains operations available only on RDDs of key-value pairs, such as groupByKey and join; org.apache.spark.rdd.DoubleRDDFunctions contains operations available only on RDDs of Doubles; and org.apache.spark.rdd.SequenceFileRDDFunctions contains operations available on RDDs that can be saved as SequenceFiles. These operations are automatically available on any RDD of the right type (e.g. RDD[(Int, Int)] through implicit conversions.Java programmers should reference the org.apache.spark.api.java package for Spark programming APIs in Java.Classes and methods marked with Experimental are user-facing features which have not been officially adopted by the Spark project. These are subject to change or removal in minor releases.Classes and methods marked with Developer API are intended for advanced users want to extend Spark through lower level interfaces. These are subject to changes or removal in minor releases. Sourcepackage.scalaLinear SupertypesAnyRef, AnyOrderingAlphabeticBy InheritanceInheritedsparkAnyRefAnyHide AllShow AllVisibilityPublicProtectedPackage Members  package api  package broadcastSpark's broadcast variables, used to broadcast immutable datasets to all nodes.  package graphxALPHA COMPONENT GraphX is a graph processing framework built on top of Spark.  package input  package ioIO codecs used for compression.IO codecs used for compression. See org.apache.spark.io.CompressionCodec.  package launcher  package mapred  package metrics  package mlDataFrame-based machine learning APIs to let users quickly assemble and configure practical machine learning pipelines.  package mllibRDD-based machine learning APIs (in maintenance mode).RDD-based machine learning APIs (in maintenance mode).The spark.mllib package is in maintenance mode as of the Spark 2.0.0 release to encourage migration to the DataFrame-based APIs under the org.apache.spark.ml package. While in maintenance mode,no new features in the RDD-based spark.mllib package will be accepted, unless they block implementing new features in the DataFrame-based spark.ml package;bug fixes in the RDD-based APIs will still be accepted.The developers will continue adding more features to the DataFrame-based APIs in the 2.x series to reach feature parity with the RDD-based APIs. And once we reach feature parity, this package will be deprecated. See alsoSPARK-4591 to track the progress of feature parity  package partialSupport for approximate results.Support for approximate results. This provides convenient api and also implementation for approximate calculation. See alsoorg.apache.spark.rdd.RDD.countApprox  package paths  package rddProvides several RDD implementations.Provides several RDD implementations. See org.apache.spark.rdd.RDD.  package resource  package schedulerSpark's scheduling components.Spark's scheduling components. This includes the org.apache.spark.scheduler.DAGScheduler and lower level org.apache.spark.scheduler.TaskScheduler.  package security  package serializerPluggable serializers for RDD and shuffle data.Pluggable serializers for RDD and shuffle data. See alsoorg.apache.spark.serializer.Serializer  package shuffle  package sqlAllows the execution of relational queries, including those expressed in SQL using Spark.  package status  package storage  package streamingSpark Streaming functionality.Spark Streaming functionality. org.apache.spark.streaming.StreamingContext serves as the main entry point to Spark Streaming, while org.apache.spark.streaming.dstream.DStream is the data type representing a continuous sequence of RDDs, representing a continuous stream of data.In addition, org.apache.spark.streaming.dstream.PairDStreamFunctions contains operations available only on DStreams of key-value pairs, such as groupByKey and reduceByKey. These operations are automatically available on any DStream of the right type (e.g. DStream[(Int, Int)] through implicit conversions.For the Java API of Spark Streaming, take a look at the org.apache.spark.streaming.api.java.JavaStreamingContext which serves as the entry point, and the org.apache.spark.streaming.api.java.JavaDStream and the org.apache.spark.streaming.api.java.JavaPairDStream which have the DStream functionality.  package ui  package unsafe  package utilSpark utilities.Type Members  case class Aggregator[K, V, C](createCombiner: (V) => C, mergeValue: (C, V) => C, mergeCombiners: (C, C) => C) extends Product with Serializable:: DeveloperApi :: A set of functions used to aggregate data.:: DeveloperApi :: A set of functions used to aggregate data. createCombinerfunction to create the initial value of the aggregation.mergeValuefunction to merge a new value into the aggregation result.mergeCombinersfunction to merge outputs from multiple mergeValue function.Annotations@DeveloperApi()  class BarrierTaskContext extends TaskContext with Logging:: Experimental :: A TaskContext with extra contextual info and tooling for tasks in a barrier stage.:: Experimental :: A TaskContext with extra contextual info and tooling for tasks in a barrier stage. Use BarrierTaskContext#get to obtain the barrier context for a running barrier task. Annotations@Experimental() @Since(\"2.4.0\")  class BarrierTaskInfo extends AnyRef:: Experimental :: Carries all task infos of a barrier task.:: Experimental :: Carries all task infos of a barrier task. Annotations@Experimental() @Since(\"2.4.0\")  class BreakingChangeInfo extends AnyRefAdditional information if the error was caused by a breaking change.  class ComplexFutureAction[T] extends FutureAction[T]A FutureAction for actions that could trigger multiple Spark jobs.A FutureAction for actions that could trigger multiple Spark jobs. Examples include take, takeSample. Cancellation works by setting the cancelled flag to true and cancelling any pending jobs. Annotations@DeveloperApi()  abstract class Dependency[T] extends Serializable:: DeveloperApi :: Base class for dependencies.:: DeveloperApi :: Base class for dependencies. Annotations@DeveloperApi()  class ErrorClassesJsonReader extends AnyRefA reader to load error information from one or more JSON files.A reader to load error information from one or more JSON files. Note that, if one error appears in more than one JSON files, the latter wins. Please read common/utils/src/main/resources/error/README.md for more details. Annotations@DeveloperApi()  case class ExceptionFailure(className: String, , [StackTraceElement], , [ThrowableSerializationWrapper], [AccumulableInfo] = Seq.empty, [AccumulatorV2[_, _]] = Nil, [Long] = Seq.empty) extends TaskFailedReason with Product with Serializable:: DeveloperApi :: Task failed due to a runtime exception.:: DeveloperApi :: Task failed due to a runtime exception. This is the most common failure case and also captures user program exceptions.stackTrace contains the stack trace of the exception itself. It still exists for backward compatibility. It's better to use this(e: Throwable, [TaskMetrics]) to create ExceptionFailure as it will handle the backward compatibility properly.fullStackTrace is a better representation of the stack trace because it contains the whole stack trace including the exception and its causesexception is the actual exception that caused the task to fail. It may be None in the case that the exception is not in fact serializable. If a task fails more than once (due to retries), exception is that one that caused the last failure. Annotations@DeveloperApi()  case class ExecutorLostFailure(execId: String, = true, [String]) extends TaskFailedReason with Product with Serializable:: DeveloperApi :: The task failed because the executor that it was running on was lost.:: DeveloperApi :: The task failed because the executor that it was running on was lost. This may happen because the task crashed the JVM. Annotations@DeveloperApi()  case class FetchFailed(bmAddress: BlockManagerId, , , , , ) extends TaskFailedReason with Product with Serializable:: DeveloperApi :: Task failed to fetch shuffle data from a remote node.:: DeveloperApi :: Task failed to fetch shuffle data from a remote node. Probably means we have lost the remote executors the task is trying to fetch from, and thus need to rerun the previous stage. Annotations@DeveloperApi()  trait FutureAction[T] extends Future[T]A future for the result of an action to support cancellation.A future for the result of an action to support cancellation. This is an extension of the Scala Future interface to support cancellation.  class HashPartitioner extends PartitionerA org.apache.spark.Partitioner that implements hash-based partitioning using Java's Object.hashCode.A org.apache.spark.Partitioner that implements hash-based partitioning using Java's Object.hashCode.Java arrays have hashCodes that are based on the arrays' identities rather than their contents, so attempting to partition an RDD[Array[_]] or RDD[(Array[_], _)] using a HashPartitioner will produce an unexpected or incorrect result.  class InterruptibleIterator[+T] extends Iterator[T]:: DeveloperApi :: An iterator that wraps around an existing iterator to provide task killing functionality.:: DeveloperApi :: An iterator that wraps around an existing iterator to provide task killing functionality. It works by checking the interrupted flag in TaskContext. Annotations@DeveloperApi()  sealed final class JobExecutionStatus extends Enum[JobExecutionStatus]  trait JobSubmitter extends AnyRefHandle via which a \"run\" function passed to a ComplexFutureAction can submit jobs for execution.Handle via which a \"run\" function passed to a ComplexFutureAction can submit jobs for execution. Annotations@DeveloperApi()  class MitigationConfig extends AnyRefA spark config flag that can be used to mitigate a breaking change.  abstract class NarrowDependency[T] extends Dependency[T]:: DeveloperApi :: Base class for dependencies where each partition of the child RDD depends on a small number of partitions of the parent RDD.:: DeveloperApi :: Base class for dependencies where each partition of the child RDD depends on a small number of partitions of the parent RDD. Narrow dependencies allow for pipelined execution. Annotations@DeveloperApi()  class OneToOneDependency[T] extends NarrowDependency[T]:: DeveloperApi :: Represents a one-to-one dependency between partitions of the parent and child RDDs.:: DeveloperApi :: Represents a one-to-one dependency between partitions of the parent and child RDDs. Annotations@DeveloperApi()  trait Partition extends SerializableAn identifier for a partition in an RDD.  trait PartitionEvaluator[T, U] extends AnyRefAn evaluator for computing RDD partitions.An evaluator for computing RDD partitions. Spark serializes and sends PartitionEvaluatorFactory to executors, and then creates PartitionEvaluator via the factory at the executor side. Annotations@DeveloperApi() @Since(\"3.5.0\")  trait PartitionEvaluatorFactory[T, U] extends SerializableA factory to create PartitionEvaluator.A factory to create PartitionEvaluator. Spark serializes and sends PartitionEvaluatorFactory to executors, and then creates PartitionEvaluator via the factory at the executor side. Annotations@DeveloperApi() @Since(\"3.5.0\")  abstract class Partitioner extends SerializableAn object that defines how the elements in a key-value pair RDD are partitioned by key.An object that defines how the elements in a key-value pair RDD are partitioned by key. Maps each key to a partition ID, from 0 to numPartitions - 1.Note that, partitioner must be deterministic, i.e. it must return the same partition id given the same partition key.  trait QueryContext extends AnyRefQuery context of a SparkThrowable.Query context of a SparkThrowable. It helps users understand where error occur while executing queries. Annotations@Evolving() Since3.4.0  sealed final class QueryContextType extends Enum[QueryContextType]The type of QueryContext.The type of QueryContext. Annotations@Evolving() Since4.0.0  class RangeDependency[T] extends NarrowDependency[T]:: DeveloperApi :: Represents a one-to-one dependency between ranges of partitions in the parent and child RDDs.:: DeveloperApi :: Represents a one-to-one dependency between ranges of partitions in the parent and child RDDs.Annotations@DeveloperApi()  class RangePartitioner[K, V] extends PartitionerA org.apache.spark.Partitioner that partitions sortable records by range into roughly equal ranges.A org.apache.spark.Partitioner that partitions sortable records by range into roughly equal ranges. The ranges are determined by sampling the content of the RDD passed in. NoteThe actual number of partitions created by the RangePartitioner might not be the same as the partitions parameter, in the case where the number of sampled records is less than the value of partitions.  trait ReadOnlySparkConf extends AnyRef  class SerializableWritable[T <: Writable] extends SerializableAnnotations@DeveloperApi()  class ShuffleDependency[K, V, C] extends Dependency[Product2[K, V]] with Logging:: DeveloperApi :: Represents a dependency on the output of a shuffle stage.:: DeveloperApi :: Represents a dependency on the output of a shuffle stage. Note that in the case of shuffle, the RDD is transient since we don't need it on the executor side. Annotations@DeveloperApi()  case class ShuffleStatusNotFoundException(shuffleId: Int, ) extends SparkException with Product with Serializable  class SimpleFutureAction[T] extends FutureAction[T]A FutureAction holding the result of an action that triggers a single job.A FutureAction holding the result of an action that triggers a single job. Examples include count, collect, reduce. Annotations@DeveloperApi()  class SparkConf extends ReadOnlySparkConf with Cloneable with Logging with SerializableConfiguration for a Spark application.Configuration for a Spark application. Used to set various Spark parameters as key-value pairs.Most of the time, you would create a SparkConf object with new SparkConf(), which will load values from any spark.* Java system properties set in your application as well. In this case, parameters you set directly on the SparkConf object take priority over system properties.For unit tests, you can also call new SparkConf(false) to skip loading external settings and get the same configuration no matter what the system properties are.All setter methods in this class support chaining. For example, you can write new SparkConf().setMaster(\"local\").setAppName(\"My app\"). NoteOnce a SparkConf object is passed to Spark, it is cloned and can no longer be modified by the user. Spark does not support modifying the configuration at runtime.  class SparkContext extends LoggingMain entry point for Spark functionality.Main entry point for Spark functionality. A SparkContext represents the connection to a Spark cluster, and can be used to create RDDs, accumulators and broadcast variables on that cluster. NoteOnly one SparkContext should be active per JVM. You must stop() the active SparkContext before creating a new one.  class SparkEnv extends Logging:: DeveloperApi :: Holds all the runtime environment objects for a running Spark instance (either master or worker), including the serializer, RpcEnv, block manager, map output tracker, etc.:: DeveloperApi :: Holds all the runtime environment objects for a running Spark instance (either master or worker), including the serializer, RpcEnv, block manager, map output tracker, etc. Currently Spark code finds the SparkEnv through a global variable, so all the threads can access the same SparkEnv. It can be accessed by SparkEnv.get (e.g. after creating a SparkContext). Annotations@DeveloperApi()  class SparkException extends Exception with SparkThrowable  trait SparkExecutorInfo extends SerializableExposes information about Spark Executors.Exposes information about Spark Executors.This interface is not designed to be implemented outside of Spark. We may add additional methods which may break binary compatibility with outside implementations.  class SparkFirehoseListener extends SparkListenerInterfaceClass that allows users to receive all SparkListener events.Class that allows users to receive all SparkListener events. Users should override the onEvent method.This is a concrete Java class in order to ensure that we don't forget to update it when adding new methods to to add a method will result in a compilation error (if this was a concrete Scala class, default implementations of new event handlers would be inherited from the SparkListener trait).Please note until Spark 3.1.0 this was missing the DevelopApi annotation, this needs to be taken into account if changing this API before a major release. Annotations@DeveloperApi()  trait SparkJobInfo extends SerializableExposes information about Spark Jobs.Exposes information about Spark Jobs.This interface is not designed to be implemented outside of Spark. We may add additional methods which may break binary compatibility with outside implementations.  trait SparkStageInfo extends SerializableExposes information about Spark Stages.Exposes information about Spark Stages.This interface is not designed to be implemented outside of Spark. We may add additional methods which may break binary compatibility with outside implementations.  class SparkStatusTracker extends AnyRefLow-level status reporting APIs for monitoring job and stage progress.Low-level status reporting APIs for monitoring job and stage progress.These APIs intentionally provide very weak consistency semantics; consumers of these APIs should be prepared to handle empty / missing information. For example, a job's stage ids may be known but the status API may not have any information about the details of those stages, so getStageInfo could potentially return None for a valid stage id.To limit memory usage, these APIs only provide information on recent jobs / stages. These APIs will provide information for the last spark.ui.retainedStages stages and spark.ui.retainedJobs jobs.NOTE: this class's constructor should be considered private and may be subject to change.  trait SparkThrowable extends AnyRefInterface mixed into Throwables thrown from Spark.Interface mixed into Throwables thrown from Spark.- For backwards compatibility, existing Throwable types can be thrown with an arbitrary error message with a null error class. See SparkException. - To promote standardization, Throwables should be thrown with an error class and message parameters to construct an error message with SparkThrowableHelper.getMessage(). New Throwable types should not accept arbitrary error messages. See SparkArithmeticException. Annotations@Evolving() Since3.2.0  class StringSubstitutor extends AnyRef  case class TaskCommitDenied(jobID: Int, , ) extends TaskFailedReason with Product with Serializable:: DeveloperApi :: Task requested the driver to commit, but was denied.:: DeveloperApi :: Task requested the driver to commit, but was denied. Annotations@DeveloperApi()  abstract class TaskContext extends SerializableContextual information about a task which can be read or mutated during execution.Contextual information about a task which can be read or mutated during execution. To access the TaskContext for a running task, ()  sealed trait TaskEndReason extends AnyRef:: DeveloperApi :: Various possible reasons why a task ended.:: DeveloperApi :: Various possible reasons why a task ended. The low-level TaskScheduler is supposed to retry tasks several times for \"ephemeral\" failures, and only report back failures that require some old stages to be resubmitted, such as shuffle map fetch failures. Annotations@DeveloperApi()  sealed trait TaskFailedReason extends TaskEndReason:: DeveloperApi :: Various possible reasons why a task failed.:: DeveloperApi :: Various possible reasons why a task failed. Annotations@DeveloperApi()  case class TaskKilled(reason: String, [AccumulableInfo] = Seq.empty, [AccumulatorV2[_, _]] = Nil, [Long] = Seq.empty) extends TaskFailedReason with Product with Serializable:: DeveloperApi :: Task was killed intentionally and needs to be rescheduled.:: DeveloperApi :: Task was killed intentionally and needs to be rescheduled. Annotations@DeveloperApi()  class TaskKilledException extends RuntimeException:: DeveloperApi :: Exception thrown when a task is explicitly killed (i.e., task failure is expected).:: DeveloperApi :: Exception thrown when a task is explicitly killed (i.e., task failure is expected). Annotations@DeveloperApi() Deprecated Type Members  class ContextAwareIterator[+T] extends Iterator[T]:: DeveloperApi :: A TaskContext aware iterator.:: DeveloperApi :: A TaskContext aware iterator.As the Python evaluation consumes the parent iterator in a separate thread, it could consume more data from the parent even after the task ends and the parent is closed. If an off-heap access exists in the parent iterator, it could cause segmentation fault which crashes the executor. Thus, we should use ContextAwareIterator to stop consuming after the task ends. Annotations@DeveloperApi() @deprecated Deprecated(Since version 4.0.0) Only usage for Python evaluation is now extinctSince3.1.0Value Members  val  val  val  val  val  val  val  val  object BarrierTaskContext extends SerializableAnnotations@Experimental() @Since(\"2.4.0\")  object Partitioner extends Serializable  case object Resubmitted extends TaskFailedReason with Product with Serializable:: DeveloperApi :: A org.apache.spark.scheduler.ShuffleMapTask that completed successfully earlier, but we lost the executor before the stage completed.:: DeveloperApi :: A org.apache.spark.scheduler.ShuffleMapTask that completed successfully earlier, but we lost the executor before the stage completed. This means Spark needs to reschedule the task to be re-executed on a different executor. Annotations@DeveloperApi()  object ShuffleDependency extends Serializable  object SparkContext extends LoggingThe SparkContext object contains a number of implicit conversions and parameters for use with various Spark features.  object SparkEnv extends Logging  object SparkException extends Serializable  object SparkFilesResolves paths to files added through SparkContext.addFile().  case object Success extends TaskEndReason with Product with Serializable:: DeveloperApi :: Task succeeded.:: DeveloperApi :: Task succeeded. Annotations@DeveloperApi()  object TaskContext extends Serializable  case object TaskResultLost extends TaskFailedReason with Product with Serializable:: DeveloperApi :: The task finished successfully, but the result was lost from the executor's block manager before it was fetched.:: DeveloperApi :: The task finished successfully, but the result was lost from the executor's block manager before it was fetched. Annotations@DeveloperApi()  case object UnknownReason extends TaskFailedReason with Product with Serializable:: DeveloperApi :: We don't know why the task ended -- for example, because of a ClassNotFound exception when deserializing the task result.:: DeveloperApi :: We don't know why the task ended -- for example, because of a ClassNotFound exception when deserializing the task result. Annotations@DeveloperApi()  object WritableConverter extends Serializable  object WritableFactory extends SerializableInherited from AnyRefInherited from AnyUngrouped\n\nExample:\n```text\norg.apache.spark.TaskContext.get()\n```\n\n<|endofdoc|>","metadata":{"transformedAt":"2026-08-18T15:21:42.974Z","totalSectionsIncluded":2,"totalCodeBlocksIncluded":1,"totalLines":10,"estimatedTokens":7328}}14{"id":"doc-migration_guide_spark_4_2_0_documentation-1078fd2b","source":"documentation","title":"Migration Guide - Spark 4.2.0 Documentation","url":"https://spark.apache.org/docs/latest/migration-guide.html","text":"Migration Guide This page documents sections of the migration guide for each component in order for users to migrate effectively. Spark Core SQL, Datasets, and DataFrame Structured Streaming MLlib (Machine Learning) PySpark (Python on Spark) SparkR (R on Spark)\n\n<|endofdoc|>","metadata":{"transformedAt":"2026-08-18T15:21:43.025Z","totalSectionsIncluded":1,"totalCodeBlocksIncluded":0,"totalLines":3,"estimatedTokens":69}}15{"id":"doc-sparkr_r_on_spark_spark_4_2_0_documentation-2b06c9fe","source":"documentation","title":"SparkR (R on Spark) - Spark 4.2.0 Documentation","url":"https://spark.apache.org/docs/latest/sparkr.html","text":"SparkR (R on Spark) Overview SparkDataFrame Starting Starting Up from RStudio Creating SparkDataFrames From local data frames From Data Sources From Hive tables SparkDataFrame Operations Selecting rows, columns Grouping, Aggregation Operating on Columns Applying User-Defined Function Run a given function on a large dataset using dapply or dapplyCollect dapply dapplyCollect Run a given function on a large dataset grouping by input column(s) and using gapply or gapplyCollect gapply gapplyCollect Run local R functions distributed using spark.lapply spark.lapply Eager execution Running SQL Queries from SparkR Machine Learning Algorithms Classification Regression Tree Clustering Collaborative Filtering Frequent Pattern Mining Statistics Model persistence Data type mapping between R and Spark Structured Streaming Apache Arrow in SparkR Ensure Arrow Installed Enabling for Conversion to/from R DataFrame, dapply and gapply Supported SQL Types R Function Name Conflicts Migration Guide SparkR is deprecated from Apache Spark 4.0.0 and will be removed in a future version. Overview SparkR is an R package that provides a light-weight frontend to use Apache Spark from R. In Spark 4.2.0, SparkR provides a distributed data frame implementation that supports operations like selection, filtering, aggregation etc. (similar to R data frames, dplyr) but on large datasets. SparkR also supports distributed machine learning using MLlib. SparkDataFrame A SparkDataFrame is a distributed collection of data organized into named columns. It is conceptually equivalent to a table in a relational database or a data frame in R, but with richer optimizations under the hood. SparkDataFrames can be constructed from a wide array of sources such data files, tables in Hive, external databases, or existing local R data frames. All of the examples on this page use sample data included in R or the Spark distribution and can be run using the ./bin/sparkR shell. Starting The entry point into SparkR is the SparkSession which connects your R program to a Spark cluster. You can create a SparkSession using sparkR.session and pass in options such as the application name, any spark packages depended on, etc. Further, you can also work with SparkDataFrames via SparkSession. If you are working from the sparkR shell, the SparkSession should already be created for you, and you would not need to call sparkR.session. sparkR.session() Starting Up from RStudio You can also start SparkR from RStudio. You can connect your R program to a Spark cluster from RStudio, R shell, Rscript or other R IDEs. To start, make sure SPARK_HOME is set in environment (you can check Sys.getenv), load the SparkR package, and call sparkR.session as below. It will check for the Spark installation, and, if not found, it will be downloaded and cached automatically. Alternatively, you can also run install.spark manually. In addition to calling sparkR.session, you could also specify certain Spark driver properties. Normally these Application properties and Runtime Environment cannot be set programmatically, as the driver JVM process would have been started, in this case SparkR takes care of this for you. To set them, pass them as you would other configuration properties in the sparkConfig argument to sparkR.session(). if (nchar(Sys.getenv(\"SPARK_HOME\")) < 1) { Sys.setenv(SPARK_HOME = \"/home/spark\") } library(SparkR, lib.loc = c(file.path(Sys.getenv(\"SPARK_HOME\"), \"R\", \"lib\"))) sparkR.session(master = \"local[*]\", sparkConfig = list(spark.driver.memory = \"2g\")) The following Spark driver properties can be set in sparkConfig with sparkR.session from NameProperty groupspark-submit equivalent spark.master Application Properties --master spark.kerberos.keytab Application Properties --keytab spark.kerberos.principal Application Properties --principal spark.driver.memory Application Properties --driver-memory spark.driver.extraClassPath Runtime Environment --driver-class-path spark.driver.extraJavaOptions Runtime Environment --driver-java-options spark.driver.extraLibraryPath Runtime Environment --driver-library-path Creating SparkDataFrames With a SparkSession, applications can create SparkDataFrames from a local R data frame, from a Hive table, or from other data sources. From local data frames The simplest way to create a data frame is to convert a local R data frame into a SparkDataFrame. Specifically, we can use as.DataFrame or createDataFrame and pass in the local R data frame to create a SparkDataFrame. As an example, the following creates a SparkDataFrame based using the faithful dataset from R. df <- as.DataFrame(faithful) # Displays the first part of the SparkDataFrame head(df) ## eruptions waiting ##1 3.600 79 ##2 1.800 54 ##3 3.333 74 From Data Sources SparkR supports operating on a variety of data sources through the SparkDataFrame interface. This section describes the general methods for loading and saving data using Data Sources. You can check the Spark SQL programming guide for more specific options that are available for the built-in data sources. The general method for creating SparkDataFrames from data sources is read.df. This method takes in the path for the file to load and the type of data source, and the currently active SparkSession will be used automatically. SparkR supports reading JSON, CSV and Parquet files natively, and through packages available from sources like Third Party Projects, you can find data source connectors for popular file formats like Avro. These packages can either be added by specifying --packages with spark-submit or sparkR commands, or if initializing SparkSession with sparkPackages parameter when in an interactive R shell or from RStudio. sparkR.session(sparkPackages = \"org.apache.spark:spark-avro_2.13:4.2.0\") We can see how to use data sources using an example JSON input file. Note that the file that is used here is not a typical JSON file. Each line in the file must contain a separate, self-contained valid JSON object. For more information, please see JSON Lines text format, also called newline-delimited JSON. As a consequence, a regular multi-line JSON file will most often fail. people <- read.df(\"./examples/src/main/resources/people.json\", \"json\") head(people) ## age name ##1 NA Michael ##2 30 Andy ##3 19 Justin # SparkR automatically infers the schema from the JSON file printSchema(people) # root # |-- (nullable = true) # |-- (nullable = true) # Similarly, multiple files can be read with read.json people <- read.json(c(\"./examples/src/main/resources/people.json\", \"./examples/src/main/resources/people2.json\")) The data sources API natively supports CSV formatted input files. For more information please refer to SparkR read.df API documentation. df <- read.df(csvPath, \"csv\", header = \"true\", inferSchema = \"true\", na.strings = \"NA\") The data sources API can also be used to save out SparkDataFrames into multiple file formats. For example, we can save the SparkDataFrame from the previous example to a Parquet file using write.df. write.df(people, path = \"people.parquet\", source = \"parquet\", mode = \"overwrite\") From Hive tables You can also create SparkDataFrames from Hive tables. To do this we will need to create a SparkSession with Hive support which can access tables in the Hive MetaStore. Note that Spark should have been built with Hive support and more details can be found in the SQL programming guide. In SparkR, by default it will attempt to create a SparkSession with Hive support enabled (enableHiveSupport = TRUE). sparkR.session() sql(\"CREATE TABLE IF NOT EXISTS src (key INT, value STRING)\") sql(\"LOAD DATA LOCAL INPATH 'examples/src/main/resources/kv1.txt' INTO TABLE src\") # Queries can be expressed in HiveQL. results <- sql(\"FROM src SELECT key, value\") # results is now a SparkDataFrame head(results) ## key value ## 1 238 val_238 ## 2 86 val_86 ## 3 311 val_311 SparkDataFrame Operations SparkDataFrames support a number of functions to do structured data processing. Here we include some basic examples and a complete list can be found in the API rows, columns # Create the SparkDataFrame df <- as.DataFrame(faithful) # Get basic information about the SparkDataFrame df ## SparkDataFrame[eruptions:double, ] # Select only the \"eruptions\" column head(select(df, df$eruptions)) ## eruptions ##1 3.600 ##2 1.800 ##3 3.333 # You can also pass in column name as strings head(select(df, \"eruptions\")) # Filter the SparkDataFrame to only retain rows with wait times shorter than 50 mins head(filter(df, df$waiting < 50)) ## eruptions waiting ##1 1.750 47 ##2 1.750 47 ##3 1.867 48 Grouping, Aggregation SparkR data frames support a number of commonly used functions to aggregate data after grouping. For example, we can compute a histogram of the waiting time in the faithful dataset as shown below # We use the `n` operator to count the number of times each waiting time appears head(summarize(groupBy(df, df$waiting), count = n(df$waiting))) ## waiting count ##1 70 4 ##2 67 1 ##3 69 2 # We can also sort the output from the aggregation to get the most common waiting times waiting_counts <- summarize(groupBy(df, df$waiting), count = n(df$waiting)) head(arrange(waiting_counts, desc(waiting_counts$count))) ## waiting count ##1 78 15 ##2 83 14 ##3 81 13 In addition to standard aggregations, SparkR supports OLAP cube operators (agg(cube(df, \"cyl\", \"disp\", \"gear\"), avg(df$mpg))) ## cyl disp gear avg(mpg) ##1 NA 140.8 4 22.8 ##2 4 75.7 4 30.4 ##3 8 400.0 3 19.2 ##4 8 318.0 3 15.5 ##5 NA 351.0 NA 15.8 ##6 NA 275.8 NA 16.3 and (agg(rollup(df, \"cyl\", \"disp\", \"gear\"), avg(df$mpg))) ## cyl disp gear avg(mpg) ##1 4 75.7 4 30.4 ##2 8 400.0 3 19.2 ##3 8 318.0 3 15.5 ##4 4 78.7 NA 32.4 ##5 8 304.0 3 15.2 ##6 4 79.0 NA 27.3 Operating on Columns SparkR also provides a number of functions that can be directly applied to columns for data processing and during aggregation. The example below shows the use of basic arithmetic functions. # Convert waiting time from hours to seconds. # Note that we can assign this to a new column in the same SparkDataFrame df$waiting_secs <- df$waiting * 60 head(df) ## eruptions waiting waiting_secs ##1 3.600 79 4740 ##2 1.800 54 3240 ##3 3.333 74 4440 Applying User-Defined Function In SparkR, we support several kinds of User-Defined a given function on a large dataset using dapply or dapplyCollect dapply Apply a function to each partition of a SparkDataFrame. The function to be applied to each partition of the SparkDataFrame and should have only one parameter, to which a data.frame corresponds to each partition will be passed. The output of function should be a data.frame. Schema specifies the row format of the resulting a SparkDataFrame. It must match to data types of returned value. # Convert waiting time from hours to seconds. # Note that we can apply UDF to DataFrame. schema <- structType(structField(\"eruptions\", \"double\"), structField(\"waiting\", \"double\"), structField(\"waiting_secs\", \"double\")) df1 <- dapply(df, function(x) { x <- cbind(x, x$waiting * 60) }, schema) head(collect(df1)) ## eruptions waiting waiting_secs ##1 3.600 79 4740 ##2 1.800 54 3240 ##3 3.333 74 4440 ##4 2.283 62 3720 ##5 4.533 85 5100 ##6 2.883 55 3300 dapplyCollect Like dapply, apply a function to each partition of a SparkDataFrame and collect the result back. The output of function should be a data.frame. But, Schema is not required to be passed. Note that dapplyCollect can fail if the output of UDF run on all the partition cannot be pulled to the driver and fit in driver memory. # Convert waiting time from hours to seconds. # Note that we can apply UDF to DataFrame and return a R's data.frame ldf <- dapplyCollect( df, function(x) { x <- cbind(x, \"waiting_secs\" = x$waiting * 60) }) head(ldf, 3) ## eruptions waiting waiting_secs ##1 3.600 79 4740 ##2 1.800 54 3240 ##3 3.333 74 4440 Run a given function on a large dataset grouping by input column(s) and using gapply or gapplyCollect gapply Apply a function to each group of a SparkDataFrame. The function is to be applied to each group of the SparkDataFrame and should have only two key and R data.frame corresponding to that key. The groups are chosen from SparkDataFrames column(s). The output of function should be a data.frame. Schema specifies the row format of the resulting SparkDataFrame. It must represent R function’s output schema on the basis of Spark data types. The column names of the returned data.frame are set by user. # Determine six waiting times with the largest eruption time in minutes. schema <- structType(structField(\"waiting\", \"double\"), structField(\"max_eruption\", \"double\")) result <- gapply( df, \"waiting\", function(key, x) { y <- data.frame(key, max(x$eruptions)) }, schema) head(collect(arrange(result, \"max_eruption\", decreasing = TRUE))) ## waiting max_eruption ##1 64 5.100 ##2 69 5.067 ##3 71 5.033 ##4 87 5.000 ##5 63 4.933 ##6 89 4.900 gapplyCollect Like gapply, applies a function to each partition of a SparkDataFrame and collect the result back to R data.frame. The output of the function should be a data.frame. But, the schema is not required to be passed. Note that gapplyCollect can fail if the output of UDF run on all the partition cannot be pulled to the driver and fit in driver memory. # Determine six waiting times with the largest eruption time in minutes. result <- gapplyCollect( df, \"waiting\", function(key, x) { y <- data.frame(key, max(x$eruptions)) colnames(y) <- c(\"waiting\", \"max_eruption\") y }) head(result[order(result$max_eruption, decreasing = TRUE), ]) ## waiting max_eruption ##1 64 5.100 ##2 69 5.067 ##3 71 5.033 ##4 87 5.000 ##5 63 4.933 ##6 89 4.900 Run local R functions distributed using spark.lapply spark.lapply Similar to lapply in native R, spark.lapply runs a function over a list of elements and distributes the computations with Spark. Applies a function in a manner that is similar to doParallel or lapply to elements of a list. The results of all the computations should fit in a single machine. If that is not the case they can do something like df <- createDataFrame(list) and then use dapply # Perform distributed training of multiple models with spark.lapply. Here, we pass # a read-only list of arguments which specifies family the generalized linear model should be. families <- c(\"gaussian\", \"poisson\") train <- function(family) { model <- glm(Sepal.Length ~ Sepal.Width + Species, iris, family = family) summary(model) } # Return a list of model's summaries model.summaries <- spark.lapply(families, train) # Print the summary of each model print(model.summaries) Eager execution If eager execution is enabled, the data will be returned to R client immediately when the SparkDataFrame is created. By default, eager execution is not enabled and can be enabled by setting the configuration property spark.sql.repl.eagerEval.enabled to true when the SparkSession is started up. Maximum number of rows and maximum number of characters per column of data to display can be controlled by spark.sql.repl.eagerEval.maxNumRows and spark.sql.repl.eagerEval.truncate configuration properties, respectively. These properties are only effective when eager execution is enabled. If these properties are not set explicitly, by default, data up to 20 rows and up to 20 characters per column will be showed. # Start up spark session with eager execution enabled sparkR.session(master = \"local[*]\", sparkConfig = list(spark.sql.repl.eagerEval.enabled = \"true\", spark.sql.repl.eagerEval.maxNumRows = as.integer(10))) # Create a grouped and sorted SparkDataFrame df <- createDataFrame(faithful) df2 <- arrange(summarize(groupBy(df, df$waiting), count = n(df$waiting)), \"waiting\") # Similar to R data.frame, displays the data returned, instead of SparkDataFrame class string df2 ##+-------+-----+ ##|waiting|count| ##+-------+-----+ ##| 43.0| 1| ##| 45.0| 3| ##| 46.0| 5| ##| 47.0| 4| ##| 48.0| 3| ##| 49.0| 5| ##| 50.0| 5| ##| 51.0| 6| ##| 52.0| 5| ##| 53.0| 7| ##+-------+-----+ ##only showing top 10 rows Note that to enable eager execution in sparkR shell, add spark.sql.repl.eagerEval.enabled=true configuration property to the --conf option. Running SQL Queries from SparkR A SparkDataFrame can also be registered as a temporary view in Spark SQL and that allows you to run SQL queries over its data. The sql function enables applications to run SQL queries programmatically and returns the result as a SparkDataFrame. # Load a JSON file people <- read.df(\"./examples/src/main/resources/people.json\", \"json\") # Register this SparkDataFrame as a temporary view. createOrReplaceTempView(people, \"people\") # SQL statements can be run by using the sql method teenagers <- sql(\"SELECT name FROM people WHERE age >= 13 AND age <= 19\") head(teenagers) ## name ##1 Justin Machine Learning Algorithms SparkR supports the following machine learning algorithms spark.logit: Logistic Regression spark.mlp: Multilayer Perceptron (MLP) spark.naiveBayes: Naive Bayes spark.svmLinear: Linear Support Vector Machine spark.fmClassifier: Factorization Machines classifier Regression spark.survreg: Accelerated Failure Time (AFT) Survival Model spark.glm or Linear Model (GLM) spark.isoreg: Isotonic Regression spark.lm: Linear Regression spark.fmRegressor: Factorization Machines regressor Tree spark.decisionTree: Decision Tree for Regression and Classification spark.gbt: Gradient Boosted Trees for Regression and Classification spark.randomForest: Random Forest for Regression and Classification Clustering spark.bisectingKmeans: Bisecting k-means spark.gaussianMixture: Gaussian Mixture Model (GMM) spark.kmeans: K-Means spark.lda: Latent Dirichlet Allocation (LDA) spark.powerIterationClustering (PIC): Power Iteration Clustering (PIC) Collaborative Filtering spark.als: Alternating Least Squares (ALS) Frequent Pattern Mining spark.fpGrowth : FP-growth spark.prefixSpan : PrefixSpan Statistics spark.kstest: Kolmogorov-Smirnov Test Under the hood, SparkR uses MLlib to train the model. Please refer to the corresponding section of MLlib user guide for example code. Users can call summary to print a summary of the fitted model, predict to make predictions on new data, and write.ml/read.ml to save/load fitted models. SparkR supports a subset of the available R formula operators for model fitting, including ‘~’, ‘.’, ‘:’, ‘+’, and ‘-‘. Model persistence The following example shows how to save/load a MLlib model by SparkR. training <- read.df(\"data/mllib/sample_multiclass_classification_data.txt\", source = \"libsvm\") # Fit a generalized linear model of family \"gaussian\" with spark.glm df_list <- randomSplit(training, c(7,3), 2) gaussianDF <- df_list[[1]] gaussianTestDF <- df_list[[2]] gaussianGLM <- spark.glm(gaussianDF, label ~ features, family = \"gaussian\") # Save and then load a fitted MLlib model modelPath <- tempfile(pattern = \"ml\", fileext = \".tmp\") write.ml(gaussianGLM, modelPath) gaussianGLM2 <- read.ml(modelPath) # Check model summary summary(gaussianGLM2) # Check model prediction gaussianPredictions <- predict(gaussianGLM2, gaussianTestDF) head(gaussianPredictions) unlink(modelPath) Find full example code at \"examples/src/main/r/ml/ml.R\" in the Spark repo. Data type mapping between R and Spark RSpark byte byte integer integer float float double double numeric double character string string string binary binary raw binary logical boolean POSIXct timestamp POSIXlt timestamp Date date array array list array env map Structured Streaming SparkR supports the Structured Streaming API. Structured Streaming is a scalable and fault-tolerant stream processing engine built on the Spark SQL engine. For more information see the R API on the Structured Streaming Programming Guide. Apache Arrow in SparkR Apache Arrow is an in-memory columnar data format that is used in Spark to efficiently transfer data between JVM and R processes. See also PySpark optimization done, PySpark Usage Guide for Pandas with Apache Arrow. This guide targets to explain how to use Arrow optimization in SparkR with some key points. Ensure Arrow Installed Arrow R library is available on CRAN and it can be installed as below. Rscript -e 'install.packages(\"arrow\", repos=\"https://cloud.r-project.org/\")' Please refer the official documentation of Apache Arrow for more details. Note that you must ensure that Arrow R package is installed and available on all cluster nodes. The current supported minimum version is 1.0.0; however, this might change between the minor releases since Arrow optimization in SparkR is experimental. Enabling for Conversion to/from R DataFrame, dapply and gapply Arrow optimization is available when converting a Spark DataFrame to an R DataFrame using the call collect(spark_df), when creating a Spark DataFrame from an R DataFrame with createDataFrame(r_df), when applying an R native function to each partition via dapply(...) and when applying an R native function to grouped data via gapply(...). To use Arrow when executing these, users need to set the Spark configuration ‘spark.sql.execution.arrow.sparkr.enabled’ to ‘true’ first. This is disabled by default. Whether the optimization is enabled or not, SparkR produces the same results. In addition, the conversion between Spark DataFrame and R DataFrame falls back automatically to non-Arrow optimization implementation when the optimization fails for any reasons before the actual computation. # Start up spark session with Arrow optimization enabled sparkR.session(master = \"local[*]\", sparkConfig = list(spark.sql.execution.arrow.sparkr.enabled = \"true\")) # Converts Spark DataFrame from an R DataFrame spark_df <- createDataFrame(mtcars) # Converts Spark DataFrame to an R DataFrame collect(spark_df) # Apply an R native function to each partition. collect(dapply(spark_df, function(rdf) { data.frame(rdf$gear + 1) }, structType(\"gear double\"))) # Apply an R native function to grouped data. collect(gapply(spark_df, \"gear\", function(key, group) { data.frame(gear = key[[1]], disp = mean(group$disp) > group$disp) }, structType(\"gear double, disp boolean\"))) Note that even with Arrow, collect(spark_df) results in the collection of all records in the DataFrame to the driver program and should be done on a small subset of the data. In addition, the specified output schema in gapply(...) and dapply(...) should be matched to the R DataFrame’s returned by the given function. Supported SQL Types Currently, all Spark SQL data types are supported by Arrow-based conversion except FloatType, BinaryType, ArrayType, StructType and MapType. R Function Name Conflicts When loading and attaching a new package in R, it is possible to have a name conflict, where a function is masking another function. The following functions are masked by the SparkR functionHow to Access cov in stats::cov(x, y = NULL, use = \"everything\", method = c(\"pearson\", \"kendall\", \"spearman\")) filter in stats::filter(x, filter, method = c(\"convolution\", \"recursive\"), sides = 2, circular = FALSE, init) sample in base::sample(x, size, replace = FALSE, prob = NULL) Since part of SparkR is modeled on the dplyr package, certain functions in SparkR share the same names with those in dplyr. Depending on the load order of the two packages, some functions from the package loaded first are masked by those in the package loaded after. In such case, prefix such calls with the package name, for instance, SparkR::cume_dist(x) or dplyr::cume_dist(x). You can inspect the search path in R with search() Migration Guide The migration guide is now archived on this page.\n\nExample:\n```r\nsparkR.session()\n```\n\nExample:\n```r\nif (nchar(Sys.getenv(\"SPARK_HOME\")) < 1) {\n  Sys.setenv(SPARK_HOME = \"/home/spark\")\n}\nlibrary(SparkR, lib.loc = c(file.path(Sys.getenv(\"SPARK_HOME\"), \"R\", \"lib\")))\nsparkR.session(master = \"local[*]\", sparkConfig = list(spark.driver.memory = \"2g\"))\n```\n\nExample:\n```r\ndf <- as.DataFrame(faithful)\n\n# Displays the first part of the SparkDataFrame\nhead(df)\n##  eruptions waiting\n##1     3.600      79\n##2     1.800      54\n##3     3.333      74\n```\n\nExample:\n```r\nsparkR.session(sparkPackages = \"org.apache.spark:spark-avro_2.13:4.2.0\")\n```\n\nExample:\n```r\npeople <- read.df(\"./examples/src/main/resources/people.json\", \"json\")\nhead(people)\n##  age    name\n##1  NA Michael\n##2  30    Andy\n##3  19  Justin\n\n# SparkR automatically infers the schema from the JSON file\nprintSchema(people)\n# root\n#  |-- age: long (nullable = true)\n#  |-- name: string (nullable = true)\n\n# Similarly, multiple files can be read with read.json\npeople <- read.json(c(\"./examples/src/main/resources/people.json\", \"./examples/src/main/resources/people2.json\"))\n```\n\nExample:\n```r\ndf <- read.df(csvPath, \"csv\", header = \"true\", inferSchema = \"true\", na.strings = \"NA\")\n```\n\nExample:\n```r\nwrite.df(people, path = \"people.parquet\", source = \"parquet\", mode = \"overwrite\")\n```\n\nExample:\n```r\nsparkR.session()\n\nsql(\"CREATE TABLE IF NOT EXISTS src (key INT, value STRING)\")\nsql(\"LOAD DATA LOCAL INPATH 'examples/src/main/resources/kv1.txt' INTO TABLE src\")\n\n# Queries can be expressed in HiveQL.\nresults <- sql(\"FROM src SELECT key, value\")\n\n# results is now a SparkDataFrame\nhead(results)\n##  key   value\n## 1 238 val_238\n## 2  86  val_86\n## 3 311 val_311\n```\n\nExample:\n```r\n# Create the SparkDataFrame\ndf <- as.DataFrame(faithful)\n\n# Get basic information about the SparkDataFrame\ndf\n## SparkDataFrame[eruptions:double, waiting:double]\n\n# Select only the \"eruptions\" column\nhead(select(df, df$eruptions))\n##  eruptions\n##1     3.600\n##2     1.800\n##3     3.333\n\n# You can also pass in column name as strings\nhead(select(df, \"eruptions\"))\n\n# Filter the SparkDataFrame to only retain rows with wait times shorter than 50 mins\nhead(filter(df, df$waiting < 50))\n##  eruptions waiting\n##1     1.750      47\n##2     1.750      47\n##3     1.867      48\n```\n\nExample:\n```r\n# We use the `n` operator to count the number of times each waiting time appears\nhead(summarize(groupBy(df, df$waiting), count = n(df$waiting)))\n##  waiting count\n##1      70     4\n##2      67     1\n##3      69     2\n\n# We can also sort the output from the aggregation to get the most common waiting times\nwaiting_counts <- summarize(groupBy(df, df$waiting), count = n(df$waiting))\nhead(arrange(waiting_counts, desc(waiting_counts$count)))\n##   waiting count\n##1      78    15\n##2      83    14\n##3      81    13\n```\n\nExample:\n```r\nhead(agg(cube(df, \"cyl\", \"disp\", \"gear\"), avg(df$mpg)))\n##  cyl  disp gear avg(mpg)\n##1  NA 140.8    4     22.8\n##2   4  75.7    4     30.4\n##3   8 400.0    3     19.2\n##4   8 318.0    3     15.5\n##5  NA 351.0   NA     15.8\n##6  NA 275.8   NA     16.3\n```\n\nExample:\n```r\nhead(agg(rollup(df, \"cyl\", \"disp\", \"gear\"), avg(df$mpg)))\n##  cyl  disp gear avg(mpg)\n##1   4  75.7    4     30.4\n##2   8 400.0    3     19.2\n##3   8 318.0    3     15.5\n##4   4  78.7   NA     32.4\n##5   8 304.0    3     15.2\n##6   4  79.0   NA     27.3\n```\n\nExample:\n```r\n# Convert waiting time from hours to seconds.\n# Note that we can assign this to a new column in the same SparkDataFrame\ndf$waiting_secs <- df$waiting * 60\nhead(df)\n##  eruptions waiting waiting_secs\n##1     3.600      79         4740\n##2     1.800      54         3240\n##3     3.333      74         4440\n```\n\nExample:\n```r\n# Convert waiting time from hours to seconds.\n# Note that we can apply UDF to DataFrame.\nschema <- structType(structField(\"eruptions\", \"double\"), structField(\"waiting\", \"double\"),\n                     structField(\"waiting_secs\", \"double\"))\ndf1 <- dapply(df, function(x) { x <- cbind(x, x$waiting * 60) }, schema)\nhead(collect(df1))\n##  eruptions waiting waiting_secs\n##1     3.600      79         4740\n##2     1.800      54         3240\n##3     3.333      74         4440\n##4     2.283      62         3720\n##5     4.533      85         5100\n##6     2.883      55         3300\n```\n\nExample:\n```r\n# Convert waiting time from hours to seconds.\n# Note that we can apply UDF to DataFrame and return a R's data.frame\nldf <- dapplyCollect(\n         df,\n         function(x) {\n           x <- cbind(x, \"waiting_secs\" = x$waiting * 60)\n         })\nhead(ldf, 3)\n##  eruptions waiting waiting_secs\n##1     3.600      79         4740\n##2     1.800      54         3240\n##3     3.333      74         4440\n```\n\nExample:\n```r\n# Determine six waiting times with the largest eruption time in minutes.\nschema <- structType(structField(\"waiting\", \"double\"), structField(\"max_eruption\", \"double\"))\nresult <- gapply(\n    df,\n    \"waiting\",\n    function(key, x) {\n        y <- data.frame(key, max(x$eruptions))\n    },\n    schema)\nhead(collect(arrange(result, \"max_eruption\", decreasing = TRUE)))\n\n##    waiting   max_eruption\n##1      64       5.100\n##2      69       5.067\n##3      71       5.033\n##4      87       5.000\n##5      63       4.933\n##6      89       4.900\n```\n\nExample:\n```r\n# Determine six waiting times with the largest eruption time in minutes.\nresult <- gapplyCollect(\n    df,\n    \"waiting\",\n    function(key, x) {\n        y <- data.frame(key, max(x$eruptions))\n        colnames(y) <- c(\"waiting\", \"max_eruption\")\n        y\n    })\nhead(result[order(result$max_eruption, decreasing = TRUE), ])\n\n##    waiting   max_eruption\n##1      64       5.100\n##2      69       5.067\n##3      71       5.033\n##4      87       5.000\n##5      63       4.933\n##6      89       4.900\n```\n\nExample:\n```r\n# Perform distributed training of multiple models with spark.lapply. Here, we pass\n# a read-only list of arguments which specifies family the generalized linear model should be.\nfamilies <- c(\"gaussian\", \"poisson\")\ntrain <- function(family) {\n  model <- glm(Sepal.Length ~ Sepal.Width + Species, iris, family = family)\n  summary(model)\n}\n# Return a list of model's summaries\nmodel.summaries <- spark.lapply(families, train)\n\n# Print the summary of each model\nprint(model.summaries)\n```\n\nExample:\n```r\n# Start up spark session with eager execution enabled\nsparkR.session(master = \"local[*]\",\n               sparkConfig = list(spark.sql.repl.eagerEval.enabled = \"true\",\n                                  spark.sql.repl.eagerEval.maxNumRows = as.integer(10)))\n\n# Create a grouped and sorted SparkDataFrame\ndf <- createDataFrame(faithful)\ndf2 <- arrange(summarize(groupBy(df, df$waiting), count = n(df$waiting)), \"waiting\")\n\n# Similar to R data.frame, displays the data returned, instead of SparkDataFrame class string\ndf2\n\n##+-------+-----+\n##|waiting|count|\n##+-------+-----+\n##|   43.0|    1|\n##|   45.0|    3|\n##|   46.0|    5|\n##|   47.0|    4|\n##|   48.0|    3|\n##|   49.0|    5|\n##|   50.0|    5|\n##|   51.0|    6|\n##|   52.0|    5|\n##|   53.0|    7|\n##+-------+-----+\n##only showing top 10 rows\n```\n\nExample:\n```r\n# Load a JSON file\npeople <- read.df(\"./examples/src/main/resources/people.json\", \"json\")\n\n# Register this SparkDataFrame as a temporary view.\ncreateOrReplaceTempView(people, \"people\")\n\n# SQL statements can be run by using the sql method\nteenagers <- sql(\"SELECT name FROM people WHERE age >= 13 AND age <= 19\")\nhead(teenagers)\n##    name\n##1 Justin\n```\n\nExample:\n```text\ntraining <- read.df(\"data/mllib/sample_multiclass_classification_data.txt\", source = \"libsvm\")\n# Fit a generalized linear model of family \"gaussian\" with spark.glm\ndf_list <- randomSplit(training, c(7,3), 2)\ngaussianDF <- df_list[[1]]\ngaussianTestDF <- df_list[[2]]\ngaussianGLM <- spark.glm(gaussianDF, label ~ features, family = \"gaussian\")\n\n# Save and then load a fitted MLlib model\nmodelPath <- tempfile(pattern = \"ml\", fileext = \".tmp\")\nwrite.ml(gaussianGLM, modelPath)\ngaussianGLM2 <- read.ml(modelPath)\n\n# Check model summary\nsummary(gaussianGLM2)\n\n# Check model prediction\ngaussianPredictions <- predict(gaussianGLM2, gaussianTestDF)\nhead(gaussianPredictions)\n\nunlink(modelPath)\n```\n\nExample:\n```text\nRscript -e 'install.packages(\"arrow\", repos=\"https://cloud.r-project.org/\")'\n```\n\nExample:\n```r\n# Start up spark session with Arrow optimization enabled\nsparkR.session(master = \"local[*]\",\n               sparkConfig = list(spark.sql.execution.arrow.sparkr.enabled = \"true\"))\n\n# Converts Spark DataFrame from an R DataFrame\nspark_df <- createDataFrame(mtcars)\n\n# Converts Spark DataFrame to an R DataFrame\ncollect(spark_df)\n\n# Apply an R native function to each partition.\ncollect(dapply(spark_df, function(rdf) { data.frame(rdf$gear + 1) }, structType(\"gear double\")))\n\n# Apply an R native function to grouped data.\ncollect(gapply(spark_df,\n               \"gear\",\n               function(key, group) {\n                 data.frame(gear = key[[1]], disp = mean(group$disp) > group$disp)\n               },\n               structType(\"gear double, disp boolean\")))\n```\n\nExample:\n```text\nstats::cov(x, y = NULL, use = \"everything\",\n           method = c(\"pearson\", \"kendall\", \"spearman\"))\n```\n\nExample:\n```text\nstats::filter(x, filter, method = c(\"convolution\", \"recursive\"),\n              sides = 2, circular = FALSE, init)\n```\n\n<|endofdoc|>","metadata":{"transformedAt":"2026-08-18T15:21:43.028Z","totalSectionsIncluded":1,"totalCodeBlocksIncluded":25,"totalLines":366,"estimatedTokens":8198}}16{"id":"doc-graphx_spark_4_2_0_documentation-61070d63","source":"documentation","title":"GraphX - Spark 4.2.0 Documentation","url":"https://spark.apache.org/docs/latest/graphx-programming-guide.html","text":"GraphX Programming Guide Overview Getting Started The Property Graph Example Property Graph Graph Operators Summary List of Operators Property Operators Structural Operators Join Operators Neighborhood Aggregation Aggregate Messages (aggregateMessages) Map Reduce Triplets Transition Guide (Legacy) Computing Degree Information Collecting Neighbors Caching and Uncaching Pregel API Graph Builders Vertex and Edge RDDs VertexRDDs EdgeRDDs Optimized Representation Graph Algorithms PageRank Connected Components Triangle Counting Examples Overview GraphX is a new component in Spark for graphs and graph-parallel computation. At a high level, GraphX extends the Spark RDD by introducing a new Graph directed multigraph with properties attached to each vertex and edge. To support graph computation, GraphX exposes a set of fundamental operators (e.g., subgraph, joinVertices, and aggregateMessages) as well as an optimized variant of the Pregel API. In addition, GraphX includes a growing collection of graph algorithms and builders to simplify graph analytics tasks. Getting Started To get started you first need to import Spark and GraphX into your project, as org.apache.spark._ import org.apache.spark.graphx._ // To make some of the examples work we will also need RDD import org.apache.spark.rdd.RDD If you are not using the Spark shell you will also need a SparkContext. To learn more about getting started with Spark refer to the Spark Quick Start Guide. The Property Graph The property graph is a directed multigraph with user defined objects attached to each vertex and edge. A directed multigraph is a directed graph with potentially multiple parallel edges sharing the same source and destination vertex. The ability to support parallel edges simplifies modeling scenarios where there can be multiple relationships (e.g., co-worker and friend) between the same vertices. Each vertex is keyed by a unique 64-bit long identifier (VertexId). GraphX does not impose any ordering constraints on the vertex identifiers. Similarly, edges have corresponding source and destination vertex identifiers. The property graph is parameterized over the vertex (VD) and edge (ED) types. These are the types of the objects associated with each vertex and edge respectively. GraphX optimizes the representation of vertex and edge types when they are primitive data types (e.g., int, double, etc…) reducing the in memory footprint by storing them in specialized arrays. In some cases it may be desirable to have vertices with different property types in the same graph. This can be accomplished through inheritance. For example to model users and products as a bipartite graph we might do the VertexProperty() case class UserProperty(val ) extends VertexProperty case class ProductProperty(val , val ) extends VertexProperty // The graph might then have the [VertexProperty, String] = null Like RDDs, property graphs are immutable, distributed, and fault-tolerant. Changes to the values or structure of the graph are accomplished by producing a new graph with the desired changes. Note that substantial parts of the original graph (i.e., unaffected structure, attributes, and indices) are reused in the new graph reducing the cost of this inherently functional data structure. The graph is partitioned across the executors using a range of vertex partitioning heuristics. As with RDDs, each partition of the graph can be recreated on a different machine in the event of a failure. Logically the property graph corresponds to a pair of typed collections (RDDs) encoding the properties for each vertex and edge. As a consequence, the graph class contains members to access the vertices and edges of the Graph[VD, ED] { val [VD] val [ED] } The classes VertexRDD[VD] and EdgeRDD[ED] extend and are optimized versions of RDD[(VertexId, VD)] and RDD[Edge[ED]] respectively. Both VertexRDD[VD] and EdgeRDD[ED] provide additional functionality built around graph computation and leverage internal optimizations. We discuss the VertexRDDVertexRDD and EdgeRDDEdgeRDD API in greater detail in the section on vertex and edge RDDs but for now they can be thought of as simply RDDs of the [(VertexId, VD)] and RDD[Edge[ED]]. Example Property Graph Suppose we want to construct a property graph consisting of the various collaborators on the GraphX project. The vertex property might contain the username and occupation. We could annotate edges with a string describing the relationships between resulting graph would have the type [(String, String), String] There are numerous ways to construct a property graph from raw files, RDDs, and even synthetic generators and these are discussed in more detail in the section on graph builders. Probably the most general method is to use the Graph object. For example the following code constructs a graph from a collection of RDDs: // Assume the SparkContext has already been constructed val // Create an RDD for the vertices val [(VertexId, (String, String))] = sc.parallelize(Seq((3L, (\"rxin\", \"student\")), (7L, (\"jgonzal\", \"postdoc\")), (5L, (\"franklin\", \"prof\")), (2L, (\"istoica\", \"prof\")))) // Create an RDD for edges val [Edge[String]] = sc.parallelize(Seq(Edge(3L, 7L, \"collab\"), Edge(5L, 3L, \"advisor\"), Edge(2L, 5L, \"colleague\"), Edge(5L, 7L, \"pi\"))) // Define a default user in case there are relationship with missing user val defaultUser = (\"John Doe\", \"Missing\") // Build the initial Graph val graph = Graph(users, relationships, defaultUser) In the above example we make use of the Edge case class. Edges have a srcId and a dstId corresponding to the source and destination vertex identifiers. In addition, the Edge class has an attr member which stores the edge property. We can deconstruct a graph into the respective vertex and edge views by using the graph.vertices and graph.edges members respectively. val [(String, String), String] // Constructed from above // Count all users which are postdocs graph.vertices.filter { case (id, (name, pos)) => pos == \"postdoc\" }.count // Count all the edges where src > dst graph.edges.filter(e => e.srcId > e.dstId).count Note that graph.vertices returns an VertexRDD[(String, String)] which extends RDD[(VertexId, (String, String))] and so we use the scala case expression to deconstruct the tuple. On the other hand, graph.edges returns an EdgeRDD containing Edge[String] objects. We could have also used the case class type constructor as in the { case Edge(src, dst, prop) => src > dst }.count In addition to the vertex and edge views of the property graph, GraphX also exposes a triplet view. The triplet view logically joins the vertex and edge properties yielding an RDD[EdgeTriplet[VD, ED]] containing instances of the EdgeTriplet class. This join can be expressed in the following SQL src.id, dst.id, src.attr, e.attr, dst.attr FROM edges AS e LEFT JOIN vertices AS src, vertices AS dst ON e.srcId = src.Id AND e.dstId = dst.Id or graphically EdgeTriplet class extends the Edge class by adding the srcAttr and dstAttr members which contain the source and destination properties respectively. We can use the triplet view of a graph to render a collection of strings describing relationships between users. val [(String, String), String] // Constructed from above // Use the triplets view to create an RDD of facts. val [String] = graph.triplets.map(triplet => triplet.srcAttr._1 + \" is the \" + triplet.attr + \" of \" + triplet.dstAttr._1) facts.collect.foreach(println(_)) Graph Operators Just as RDDs have basic operations like map, filter, and reduceByKey, property graphs also have a collection of basic operators that take user defined functions and produce new graphs with transformed properties and structure. The core operators that have optimized implementations are defined in Graph and convenient operators that are expressed as a compositions of the core operators are defined in GraphOps. However, thanks to Scala implicits the operators in GraphOps are automatically available as members of Graph. For example, we can compute the in-degree of each vertex (defined in GraphOps) by the [(String, String), String] // Use the implicit GraphOps.inDegrees operator val [Int] = graph.inDegrees The reason for differentiating between core graph operations and GraphOps is to be able to support different graph representations in the future. Each graph representation must provide implementations of the core operations and reuse many of the useful operations defined in GraphOps. Summary List of Operators The following is a quick summary of the functionality defined in both Graph and GraphOps but presented as members of Graph for simplicity. Note that some function signatures have been simplified (e.g., default arguments and type constraints removed) and some more advanced functionality has been removed so please consult the API docs for the official list of operations. /** Summary of the functionality in the property graph */ class Graph[VD, ED] { // Information about the Graph =================================================================== val val val [Int] val [Int] val [Int] // Views of the graph as collections ============================================================= val [VD] val [ED] val [EdgeTriplet[VD, ED]] // Functions for caching graphs ================================================================== def persist(newLevel: StorageLevel = StorageLevel.MEMORY_ONLY): Graph[VD, ED] def cache(): Graph[VD, ED] def unpersistVertices(blocking: Boolean = false): Graph[VD, ED] // Change the partitioning heuristic ============================================================ def partitionBy(partitionStrategy: PartitionStrategy): Graph[VD, ED] // Transform vertex and edge attributes ========================================================== def mapVertices[VD2](map: (VertexId, VD) => VD2): Graph[VD2, ED] def mapEdges[ED2](map: Edge[ED] => ED2): Graph[VD, ED2] def mapEdges[ED2](map: (PartitionID, Iterator[Edge[ED]]) => Iterator[ED2]): Graph[VD, ED2] def mapTriplets[ED2](map: EdgeTriplet[VD, ED] => ED2): Graph[VD, ED2] def mapTriplets[ED2](map: (PartitionID, Iterator[EdgeTriplet[VD, ED]]) => Iterator[ED2]) : Graph[VD, ED2] // Modify the graph structure ==================================================================== def [VD, ED] def subgraph( [VD,ED] => Boolean = (x => true), vpred: (VertexId, VD) => Boolean = ((v, d) => true)) : Graph[VD, ED] def mask[VD2, ED2](other: Graph[VD2, ED2]): Graph[VD, ED] def groupEdges(merge: (ED, ED) => ED): Graph[VD, ED] // Join RDDs with the graph ====================================================================== def joinVertices[U](table: RDD[(VertexId, U)])(mapFunc: (VertexId, VD, U) => VD): Graph[VD, ED] def outerJoinVertices[U, VD2](other: RDD[(VertexId, U)]) (mapFunc: (VertexId, VD, Option[U]) => VD2) : Graph[VD2, ED] // Aggregate information about adjacent triplets ================================================= def collectNeighborIds(edgeDirection: EdgeDirection): VertexRDD[Array[VertexId]] def collectNeighbors(edgeDirection: EdgeDirection): VertexRDD[Array[(VertexId, VD)]] def aggregateMessages[Msg: ClassTag]( [VD, ED, Msg] => Unit, mergeMsg: (Msg, Msg) => Msg, = TripletFields.All) : VertexRDD[A] // Iterative graph-parallel computation ========================================================== def pregel[A](initialMsg: A, , )( vprog: (VertexId, VD, A) => VD, [VD, ED] => Iterator[(VertexId, A)], mergeMsg: (A, A) => A) : Graph[VD, ED] // Basic graph algorithms ======================================================================== def pageRank(tol: Double, = 0.15): Graph[Double, Double] def connectedComponents(): Graph[VertexId, ED] def triangleCount(): Graph[Int, ED] def stronglyConnectedComponents(numIter: Int): Graph[VertexId, ED] } Property Operators Like the RDD map operator, the property graph contains the Graph[VD, ED] { def mapVertices[VD2](map: (VertexId, VD) => VD2): Graph[VD2, ED] def mapEdges[ED2](map: Edge[ED] => ED2): Graph[VD, ED2] def mapTriplets[ED2](map: EdgeTriplet[VD, ED] => ED2): Graph[VD, ED2] } Each of these operators yields a new graph with the vertex or edge properties modified by the user defined map function. Note that in each case the graph structure is unaffected. This is a key feature of these operators which allows the resulting graph to reuse the structural indices of the original graph. The following snippets are logically equivalent, but the first one does not preserve the structural indices and would not benefit from the GraphX system newVertices = graph.vertices.map { case (id, attr) => (id, mapUdf(id, attr)) } val newGraph = Graph(newVertices, graph.edges) Instead, use mapVertices to preserve the newGraph = graph.mapVertices((id, attr) => mapUdf(id, attr)) These operators are often used to initialize the graph for a particular computation or project away unnecessary properties. For example, given a graph with the out degrees as the vertex properties (we describe how to construct such a graph later), we initialize it for PageRank: // Given a graph where the vertex property is the out degree val [Int, String] = graph.outerJoinVertices(graph.outDegrees)((vid, _, degOpt) => degOpt.getOrElse(0)) // Construct a graph where each edge contains the weight // and each vertex is the initial PageRank val [Double, Double] = inputGraph.mapTriplets(triplet => 1.0 / triplet.srcAttr).mapVertices((id, _) => 1.0) Structural Operators Currently GraphX supports only a simple set of commonly used structural operators and we expect to add more in the future. The following is a list of the basic structural operators. class Graph[VD, ED] { def [VD, ED] def subgraph(epred: EdgeTriplet[VD,ED] => Boolean, vpred: (VertexId, VD) => Boolean): Graph[VD, ED] def mask[VD2, ED2](other: Graph[VD2, ED2]): Graph[VD, ED] def groupEdges(merge: (ED, ED) => ED): Graph[VD,ED] } The reverse operator returns a new graph with all the edge directions reversed. This can be useful when, for example, trying to compute the inverse PageRank. Because the reverse operation does not modify vertex or edge properties or change the number of edges, it can be implemented efficiently without data movement or duplication. The subgraph operator takes vertex and edge predicates and returns the graph containing only the vertices that satisfy the vertex predicate (evaluate to true) and edges that satisfy the edge predicate and connect vertices that satisfy the vertex predicate. The subgraph operator can be used in number of situations to restrict the graph to the vertices and edges of interest or eliminate broken links. For example in the following code we remove broken links: // Create an RDD for the vertices val [(VertexId, (String, String))] = sc.parallelize(Seq((3L, (\"rxin\", \"student\")), (7L, (\"jgonzal\", \"postdoc\")), (5L, (\"franklin\", \"prof\")), (2L, (\"istoica\", \"prof\")), (4L, (\"peter\", \"student\")))) // Create an RDD for edges val [Edge[String]] = sc.parallelize(Seq(Edge(3L, 7L, \"collab\"), Edge(5L, 3L, \"advisor\"), Edge(2L, 5L, \"colleague\"), Edge(5L, 7L, \"pi\"), Edge(4L, 0L, \"student\"), Edge(5L, 0L, \"colleague\"))) // Define a default user in case there are relationship with missing user val defaultUser = (\"John Doe\", \"Missing\") // Build the initial Graph val graph = Graph(users, relationships, defaultUser) // Notice that there is a user 0 (for which we have no information) connected to users // 4 (peter) and 5 (franklin). graph.triplets.map( triplet => triplet.srcAttr._1 + \" is the \" + triplet.attr + \" of \" + triplet.dstAttr._1 ).collect.foreach(println(_)) // Remove missing vertices as well as the edges to connected to them val validGraph = graph.subgraph(vpred = (id, attr) => attr._2 != \"Missing\") // The valid subgraph will disconnect users 4 and 5 by removing user 0 validGraph.vertices.collect.foreach(println(_)) validGraph.triplets.map( triplet => triplet.srcAttr._1 + \" is the \" + triplet.attr + \" of \" + triplet.dstAttr._1 ).collect.foreach(println(_)) Note in the above example only the vertex predicate is provided. The subgraph operator defaults to true if the vertex or edge predicates are not provided. The mask operator constructs a subgraph by returning a graph that contains the vertices and edges that are also found in the input graph. This can be used in conjunction with the subgraph operator to restrict a graph based on the properties in another related graph. For example, we might run connected components using the graph with missing vertices and then restrict the answer to the valid subgraph. // Run Connected Components val ccGraph = graph.connectedComponents() // No longer contains missing field // Remove missing vertices as well as the edges to connected to them val validGraph = graph.subgraph(vpred = (id, attr) => attr._2 != \"Missing\") // Restrict the answer to the valid subgraph val validCCGraph = ccGraph.mask(validGraph) The groupEdges operator merges parallel edges (i.e., duplicate edges between pairs of vertices) in the multigraph. In many numerical applications, parallel edges can be added (their weights combined) into a single edge thereby reducing the size of the graph. Join Operators In many cases it is necessary to join data from external collections (RDDs) with graphs. For example, we might have extra user properties that we want to merge with an existing graph or we might want to pull vertex properties from one graph into another. These tasks can be accomplished using the join operators. Below we list the key join Graph[VD, ED] { def joinVertices[U](table: RDD[(VertexId, U)])(map: (VertexId, VD, U) => VD) : Graph[VD, ED] def outerJoinVertices[U, VD2](table: RDD[(VertexId, U)])(map: (VertexId, VD, Option[U]) => VD2) : Graph[VD2, ED] } The joinVertices operator joins the vertices with the input RDD and returns a new graph with the vertex properties obtained by applying the user defined map function to the result of the joined vertices. Vertices without a matching value in the RDD retain their original value. Note that if the RDD contains more than one value for a given vertex only one will be used. It is therefore recommended that the input RDD be made unique using the following which will also pre-index the resulting values to substantially accelerate the subsequent join. val [(VertexId, Double)] val [Double] = graph.vertices.aggregateUsingIndex(nonUnique, (a,b) => a + b) val joinedGraph = graph.joinVertices(uniqueCosts)( (id, oldCost, extraCost) => oldCost + extraCost) The more general outerJoinVertices behaves similarly to joinVertices except that the user defined map function is applied to all vertices and can change the vertex property type. Because not all vertices may have a matching value in the input RDD the map function takes an Option type. For example, we can set up a graph for PageRank by initializing vertex properties with their outDegree. val [Int] = graph.outDegrees val degreeGraph = graph.outerJoinVertices(outDegrees) { (id, oldAttr, outDegOpt) => outDegOpt match { case Some(outDeg) => outDeg case None => 0 // No outDegree means zero outDegree } } You may have noticed the multiple parameter lists (e.g., f(a)(b)) curried function pattern used in the above examples. While we could have equally written f(a)(b) as f(a,b) this would mean that type inference on b would not depend on a. As a consequence, the user would need to provide type annotation for the user defined joinedGraph = graph.joinVertices(uniqueCosts, (id: VertexId, , ) => oldCost + extraCost) Neighborhood Aggregation A key step in many graph analytics tasks is aggregating information about the neighborhood of each vertex. For example, we might want to know the number of followers each user has or the average age of the followers of each user. Many iterative graph algorithms (e.g., PageRank, Shortest Path, and connected components) repeatedly aggregate properties of neighboring vertices (e.g., current PageRank Value, shortest path to the source, and smallest reachable vertex id). To improve performance the primary aggregation operator changed from graph.mapReduceTriplets to the new graph.AggregateMessages. While the changes in the API are relatively small, we provide a transition guide below. Aggregate Messages (aggregateMessages) The core aggregation operation in GraphX is aggregateMessages. This operator applies a user defined sendMsg function to each edge triplet in the graph and then uses the mergeMsg function to aggregate those messages at their destination vertex. class Graph[VD, ED] { def aggregateMessages[Msg: ClassTag]( [VD, ED, Msg] => Unit, mergeMsg: (Msg, Msg) => Msg, = TripletFields.All) : VertexRDD[Msg] } The user defined sendMsg function takes an EdgeContext, which exposes the source and destination attributes along with the edge attribute and functions (sendToSrc, and sendToDst) to send messages to the source and destination attributes. Think of sendMsg as the map function in map-reduce. The user defined mergeMsg function takes two messages destined to the same vertex and yields a single message. Think of mergeMsg as the reduce function in map-reduce. The aggregateMessages operator returns a VertexRDD[Msg] containing the aggregate message (of type Msg) destined to each vertex. Vertices that did not receive a message are not included in the returned VertexRDDVertexRDD. In addition, aggregateMessages takes an optional tripletsFields which indicates what data is accessed in the EdgeContext (i.e., the source vertex attribute but not the destination vertex attribute). The possible options for the tripletsFields are defined in TripletFields and the default value is TripletFields.All which indicates that the user defined sendMsg function may access any of the fields in the EdgeContext. The tripletFields argument can be used to notify GraphX that only part of the EdgeContext will be needed allowing GraphX to select an optimized join strategy. For example if we are computing the average age of the followers of each user we would only require the source field and so we would use TripletFields.Src to indicate that we only require the source field In earlier versions of GraphX we used byte code inspection to infer the TripletFields however we have found that bytecode inspection to be slightly unreliable and instead opted for more explicit user control. In the following example we use the aggregateMessages operator to compute the average age of the more senior followers of each user. import org.apache.spark.graphx.{Graph, VertexRDD} import org.apache.spark.graphx.util.GraphGenerators // Create a graph with \"age\" as the vertex property. // Here we use a random graph for simplicity. val [Double, Int] = GraphGenerators.logNormalGraph(sc, numVertices = 100).mapVertices( (id, _) => id.toDouble ) // Compute the number of older followers and their total age val [(Int, Double)] = graph.aggregateMessages[(Int, Double)]( triplet => { // Map Function if (triplet.srcAttr > triplet.dstAttr) { // Send message to destination vertex containing counter and age triplet.sendToDst((1, triplet.srcAttr)) } }, // Add counter and age (a, b) => (a._1 + b._1, a._2 + b._2) // Reduce Function ) // Divide total age by number of older followers to get average age of older followers val [Double] = olderFollowers.mapValues( (id, value) => value match { case (count, totalAge) => totalAge / count } ) // Display the results avgAgeOfOlderFollowers.collect().foreach(println(_)) Find full example code at \"examples/src/main/scala/org/apache/spark/examples/graphx/AggregateMessagesExample.scala\" in the Spark repo. The aggregateMessages operation performs optimally when the messages (and the sums of messages) are constant sized (e.g., floats and addition instead of lists and concatenation). Map Reduce Triplets Transition Guide (Legacy) In earlier versions of GraphX neighborhood aggregation was accomplished using the mapReduceTriplets Graph[VD, ED] { def mapReduceTriplets[Msg]( [VD, ED] => Iterator[(VertexId, Msg)], reduce: (Msg, Msg) => Msg) : VertexRDD[Msg] } The mapReduceTriplets operator takes a user defined map function which is applied to each triplet and can yield messages which are aggregated using the user defined reduce function. However, we found the user of the returned iterator to be expensive and it inhibited our ability to apply additional optimizations (e.g., local vertex renumbering). In aggregateMessages we introduced the EdgeContext which exposes the triplet fields and also functions to explicitly send messages to the source and destination vertex. Furthermore we removed bytecode inspection and instead require the user to indicate what fields in the triplet are actually required. The following code block using [Int, Float] = ... def msgFun(triplet: Triplet[Int, Float]): Iterator[(Int, String)] = { Iterator((triplet.dstId, \"Hi\")) } def reduceFun(a: String, ): String = a + \" \" + b val result = graph.mapReduceTriplets[String](msgFun, reduceFun) can be rewritten using aggregateMessages [Int, Float] = ... def msgFun(triplet: EdgeContext[Int, Float, String]) { triplet.sendToDst(\"Hi\") } def reduceFun(a: String, ): String = a + \" \" + b val result = graph.aggregateMessages[String](msgFun, reduceFun) Computing Degree Information A common aggregation task is computing the degree of each number of edges adjacent to each vertex. In the context of directed graphs it is often necessary to know the in-degree, out-degree, and the total degree of each vertex. The GraphOps class contains a collection of operators to compute the degrees of each vertex. For example in the following we compute the max in, out, and total degrees: // Define a reduce operation to compute the highest degree vertex def max(a: (VertexId, Int), b: (VertexId, Int)): (VertexId, Int) = { if (a._2 > b._2) a else b } // Compute the max degrees val maxInDegree: (VertexId, Int) = graph.inDegrees.reduce(max) val maxOutDegree: (VertexId, Int) = graph.outDegrees.reduce(max) val maxDegrees: (VertexId, Int) = graph.degrees.reduce(max) Collecting Neighbors In some cases it may be easier to express computation by collecting neighboring vertices and their attributes at each vertex. This can be easily accomplished using the collectNeighborIds and the collectNeighbors operators. class GraphOps[VD, ED] { def collectNeighborIds(edgeDirection: EdgeDirection): VertexRDD[Array[VertexId]] def collectNeighbors(edgeDirection: EdgeDirection): VertexRDD[ Array[(VertexId, VD)] ] } These operators can be quite costly as they duplicate information and require substantial communication. If possible try expressing the same computation using the aggregateMessages operator directly. Caching and Uncaching In Spark, RDDs are not persisted in memory by default. To avoid recomputation, they must be explicitly cached when using them multiple times (see the Spark Programming Guide). Graphs in GraphX behave the same way. When using a graph multiple times, make sure to call Graph.cache() on it first. In iterative computations, uncaching may also be necessary for best performance. By default, cached RDDs and graphs will remain in memory until memory pressure forces them to be evicted in LRU order. For iterative computation, intermediate results from previous iterations will fill up the cache. Though they will eventually be evicted, the unnecessary data stored in memory will slow down garbage collection. It would be more efficient to uncache intermediate results as soon as they are no longer necessary. This involves materializing (caching and forcing) a graph or RDD every iteration, uncaching all other datasets, and only using the materialized dataset in future iterations. However, because graphs are composed of multiple RDDs, it can be difficult to unpersist them correctly. For iterative computation we recommend using the Pregel API, which correctly unpersists intermediate results. Pregel API Graphs are inherently recursive data structures as properties of vertices depend on properties of their neighbors which in turn depend on properties of their neighbors. As a consequence many important graph algorithms iteratively recompute the properties of each vertex until a fixed-point condition is reached. A range of graph-parallel abstractions have been proposed to express these iterative algorithms. GraphX exposes a variant of the Pregel API. At a high level the Pregel operator in GraphX is a bulk-synchronous parallel messaging abstraction constrained to the topology of the graph. The Pregel operator executes in a series of super steps in which vertices receive the sum of their inbound messages from the previous super step, compute a new value for the vertex property, and then send messages to neighboring vertices in the next super step. Unlike Pregel, messages are computed in parallel as a function of the edge triplet and the message computation has access to both the source and destination vertex attributes. Vertices that do not receive a message are skipped within a super step. The Pregel operator terminates iteration and returns the final graph when there are no messages remaining. Note, unlike more standard Pregel implementations, vertices in GraphX can only send messages to neighboring vertices and the message construction is done in parallel using a user defined messaging function. These constraints allow additional optimization within GraphX. The following is the type signature of the Pregel operator as well as a sketch of its implementation (note: to avoid stackOverflowError due to long lineage chains, pregel support periodically checkpoint graph and messages by setting “spark.graphx.pregel.checkpointInterval” to a positive number, say 10. And set checkpoint directory as well using SparkContext.setCheckpointDir(directory: String)): class GraphOps[VD, ED] { def pregel[A] (initialMsg: A, = Int.MaxValue, = EdgeDirection.Out) (vprog: (VertexId, VD, A) => VD, [VD, ED] => Iterator[(VertexId, A)], mergeMsg: (A, A) => A) : Graph[VD, ED] = { // Receive the initial message at each vertex var g = mapVertices( (vid, vdata) => vprog(vid, vdata, initialMsg) ).cache() // compute the messages var messages = GraphXUtils.mapReduceTriplets(g, sendMsg, mergeMsg) var activeMessages = messages.count() // Loop until no messages remain or maxIterations is achieved var i = 0 while (activeMessages > 0 && i < maxIterations) { // Receive the messages and update the vertices. g = g.joinVertices(messages)(vprog).cache() val oldMessages = messages // Send new messages, skipping edges where neither side received a message. We must cache // messages so it can be materialized on the next line, allowing us to uncache the previous // iteration. messages = GraphXUtils.mapReduceTriplets( g, sendMsg, mergeMsg, Some((oldMessages, activeDirection))).cache() activeMessages = messages.count() i += 1 } g } } Notice that Pregel takes two argument lists (i.e., graph.pregel(list1)(list2)). The first argument list contains configuration parameters including the initial message, the maximum number of iterations, and the edge direction in which to send messages (by default along out edges). The second argument list contains the user defined functions for receiving messages (the vertex program vprog), computing messages (sendMsg), and combining messages mergeMsg. We can use the Pregel operator to express computation such as single source shortest path in the following example. import org.apache.spark.graphx.{Graph, VertexId} import org.apache.spark.graphx.util.GraphGenerators // A graph with edge attributes containing distances val [Long, Double] = GraphGenerators.logNormalGraph(sc, numVertices = 100).mapEdges(e => e.attr.toDouble) val = 42 // The ultimate source // Initialize the graph such that all vertices except the root have distance infinity. val initialGraph = graph.mapVertices((id, _) => if (id == sourceId) 0.0 else Double.PositiveInfinity) val sssp = initialGraph.pregel(Double.PositiveInfinity)( (id, dist, newDist) => math.min(dist, newDist), // Vertex Program triplet => { // Send Message if (triplet.srcAttr + triplet.attr < triplet.dstAttr) { Iterator((triplet.dstId, triplet.srcAttr + triplet.attr)) } else { Iterator.empty } }, (a, b) => math.min(a, b) // Merge Message ) println(sssp.vertices.collect().mkString(\"\\n\")) Find full example code at \"examples/src/main/scala/org/apache/spark/examples/graphx/SSSPExample.scala\" in the Spark repo. Graph Builders GraphX provides several ways of building a graph from a collection of vertices and edges in an RDD or on disk. None of the graph builders repartitions the graph’s edges by default; instead, edges are left in their default partitions (such as their original blocks in HDFS). Graph.groupEdges requires the graph to be repartitioned because it assumes identical edges will be colocated on the same partition, so you must call Graph.partitionBy before calling groupEdges. object GraphLoader { def edgeListFile( , , = false, = 1) : Graph[Int, Int] } GraphLoader.edgeListFile provides a way to load a graph from a list of edges on disk. It parses an adjacency list of (source vertex ID, destination vertex ID) pairs of the following form, skipping comment lines that begin with #: # This is a comment 2 1 4 1 1 2 It creates a Graph from the specified edges, automatically creating any vertices mentioned by edges. All vertex and edge attributes default to 1. The canonicalOrientation argument allows reorienting edges in the positive direction (srcId < dstId), which is required by the connected components algorithm. The minEdgePartitions argument specifies the minimum number of edge partitions to generate; there may be more edge partitions than specified if, for example, the HDFS file has more blocks. object Graph { def apply[VD, ED]( [(VertexId, VD)], [Edge[ED]], = null) : Graph[VD, ED] def fromEdges[VD, ED]( [Edge[ED]], ): Graph[VD, ED] def fromEdgeTuples[VD]( [(VertexId, VertexId)], , [PartitionStrategy] = None): Graph[VD, Int] } Graph.apply allows creating a graph from RDDs of vertices and edges. Duplicate vertices are picked arbitrarily and vertices found in the edge RDD but not the vertex RDD are assigned the default attribute. Graph.fromEdges allows creating a graph from only an RDD of edges, automatically creating any vertices mentioned by edges and assigning them the default value. Graph.fromEdgeTuples allows creating a graph from only an RDD of edge tuples, assigning the edges the value 1, and automatically creating any vertices mentioned by edges and assigning them the default value. It also supports deduplicating the edges; to deduplicate, pass Some of a PartitionStrategy as the uniqueEdges parameter (for example, uniqueEdges = Some(PartitionStrategy.RandomVertexCut)). A partition strategy is necessary to colocate identical edges on the same partition so they can be deduplicated. Vertex and Edge RDDs GraphX exposes RDD views of the vertices and edges stored within the graph. However, because GraphX maintains the vertices and edges in optimized data structures and these data structures provide additional functionality, the vertices and edges are returned as VertexRDDVertexRDD and EdgeRDDEdgeRDD respectively. In this section we review some of the additional useful functionality in these types. Note that this is just an incomplete list, please refer to the API docs for the official list of operations. VertexRDDs The VertexRDD[A] extends RDD[(VertexId, A)] and adds the additional constraint that each VertexId occurs only once. Moreover, VertexRDD[A] represents a set of vertices each with an attribute of type A. Internally, this is achieved by storing the vertex attributes in a reusable hash-map data-structure. As a consequence if two VertexRDDs are derived from the same base VertexRDDVertexRDD (e.g., by filter or mapValues) they can be joined in constant time without hash evaluations. To leverage this indexed data structure, the VertexRDDVertexRDD exposes the following additional VertexRDD[VD] extends RDD[(VertexId, VD)] { // Filter the vertex set but preserves the internal index def filter(pred: Tuple2[VertexId, VD] => Boolean): VertexRDD[VD] // Transform the values without changing the ids (preserves the internal index) def mapValues[VD2](map: VD => VD2): VertexRDD[VD2] def mapValues[VD2](map: (VertexId, VD) => VD2): VertexRDD[VD2] // Show only vertices unique to this set based on their VertexId's def minus(other: RDD[(VertexId, VD)]) // Remove vertices from this set that appear in the other set def diff(other: VertexRDD[VD]): VertexRDD[VD] // Join operators that take advantage of the internal indexing to accelerate joins (substantially) def leftJoin[VD2, VD3](other: RDD[(VertexId, VD2)])(f: (VertexId, VD, Option[VD2]) => VD3): VertexRDD[VD3] def innerJoin[U, VD2](other: RDD[(VertexId, U)])(f: (VertexId, VD, U) => VD2): VertexRDD[VD2] // Use the index on this RDD to accelerate a `reduceByKey` operation on the input RDD. def aggregateUsingIndex[VD2](other: RDD[(VertexId, VD2)], reduceFunc: (VD2, VD2) => VD2): VertexRDD[VD2] } Notice, for example, how the filter operator returns a VertexRDDVertexRDD. Filter is actually implemented using a BitSet thereby reusing the index and preserving the ability to do fast joins with other VertexRDDs. Likewise, the mapValues operators do not allow the map function to change the VertexId thereby enabling the same HashMap data structures to be reused. Both the leftJoin and innerJoin are able to identify when joining two VertexRDDs derived from the same HashMap and implement the join by linear scan rather than costly point lookups. The aggregateUsingIndex operator is useful for efficient construction of a new VertexRDDVertexRDD from an RDD[(VertexId, A)]. Conceptually, if I have constructed a VertexRDD[B] over a set of vertices, which is a super-set of the vertices in some RDD[(VertexId, A)] then I can reuse the index to both aggregate and then subsequently index the RDD[(VertexId, A)]. For [Int] = VertexRDD(sc.parallelize(0L until 100L).map(id => (id, 1))) val [(VertexId, Double)] = sc.parallelize(0L until 100L).flatMap(id => List((id, 1.0), (id, 2.0))) // There should be 200 entries in rddB rddB.count val [Double] = setA.aggregateUsingIndex(rddB, _ + _) // There should be 100 entries in setB setB.count // Joining A and B should now be fast! val [Double] = setA.innerJoin(setB)((id, a, b) => a + b) EdgeRDDs The EdgeRDD[ED], which extends RDD[Edge[ED]] organizes the edges in blocks partitioned using one of the various partitioning strategies defined in PartitionStrategy. Within each partition, edge attributes and adjacency structure, are stored separately enabling maximum reuse when changing attribute values. The three additional functions exposed by the EdgeRDDEdgeRDD are: // Transform the edge attributes while preserving the structure def mapValues[ED2](f: Edge[ED] => ED2): EdgeRDD[ED2] // Reverse the edges reusing both attributes and structure def [ED] // Join two `EdgeRDD`s partitioned using the same partitioning strategy. def innerJoin[ED2, ED3](other: EdgeRDD[ED2])(f: (VertexId, VertexId, ED, ED2) => ED3): EdgeRDD[ED3] In most applications we have found that operations on the EdgeRDDEdgeRDD are accomplished through the graph operators or rely on operations defined in the base RDD class. Optimized Representation While a detailed description of the optimizations used in the GraphX representation of distributed graphs is beyond the scope of this guide, some high-level understanding may aid in the design of scalable algorithms as well as optimal use of the API. GraphX adopts a vertex-cut approach to distributed graph than splitting graphs along edges, GraphX partitions the graph along vertices which can reduce both the communication and storage overhead. Logically, this corresponds to assigning edges to machines and allowing vertices to span multiple machines. The exact method of assigning edges depends on the PartitionStrategy and there are several tradeoffs to the various heuristics. Users can choose between different strategies by repartitioning the graph with the Graph.partitionBy operator. The default partitioning strategy is to use the initial partitioning of the edges as provided on graph construction. However, users can easily switch to 2D-partitioning or other heuristics included in GraphX. Once the edges have been partitioned the key challenge to efficient graph-parallel computation is efficiently joining vertex attributes with the edges. Because real-world graphs typically have more edges than vertices, we move vertex attributes to the edges. Because not all partitions will contain edges adjacent to all vertices we internally maintain a routing table which identifies where to broadcast vertices when implementing the join required for operations like triplets and aggregateMessages. Graph Algorithms GraphX includes a set of graph algorithms to simplify analytics tasks. The algorithms are contained in the org.apache.spark.graphx.lib package and can be accessed directly as methods on Graph via GraphOps. This section describes the algorithms and how they are used. PageRank PageRank measures the importance of each vertex in a graph, assuming an edge from u to v represents an endorsement of v’s importance by u. For example, if a Twitter user is followed by many others, the user will be ranked highly. GraphX comes with static and dynamic implementations of PageRank as methods on the PageRank object. Static PageRank runs for a fixed number of iterations, while dynamic PageRank runs until the ranks converge (i.e., stop changing by more than a specified tolerance). GraphOps allows calling these algorithms directly as methods on Graph. GraphX also includes an example social network dataset that we can run PageRank on. A set of users is given in data/graphx/users.txt, and a set of relationships between users is given in data/graphx/followers.txt. We compute the PageRank of each user as org.apache.spark.graphx.GraphLoader // Load the edges as a graph val graph = GraphLoader.edgeListFile(sc, \"data/graphx/followers.txt\") // Run PageRank val ranks = graph.pageRank(0.0001).vertices // Join the ranks with the usernames val users = sc.textFile(\"data/graphx/users.txt\").map { line => val fields = line.split(\",\") (fields(0).toLong, fields(1)) } val ranksByUsername = users.join(ranks).map { case (id, (username, rank)) => (username, rank) } // Print the result println(ranksByUsername.collect().mkString(\"\\n\")) Find full example code at \"examples/src/main/scala/org/apache/spark/examples/graphx/PageRankExample.scala\" in the Spark repo. Connected Components The connected components algorithm labels each connected component of the graph with the ID of its lowest-numbered vertex. For example, in a social network, connected components can approximate clusters. GraphX contains an implementation of the algorithm in the ConnectedComponents object, and we compute the connected components of the example social network dataset from the PageRank section as org.apache.spark.graphx.GraphLoader // Load the graph as in the PageRank example val graph = GraphLoader.edgeListFile(sc, \"data/graphx/followers.txt\") // Find the connected components val cc = graph.connectedComponents().vertices // Join the connected components with the usernames val users = sc.textFile(\"data/graphx/users.txt\").map { line => val fields = line.split(\",\") (fields(0).toLong, fields(1)) } val ccByUsername = users.join(cc).map { case (id, (username, cc)) => (username, cc) } // Print the result println(ccByUsername.collect().mkString(\"\\n\")) Find full example code at \"examples/src/main/scala/org/apache/spark/examples/graphx/ConnectedComponentsExample.scala\" in the Spark repo. Triangle Counting A vertex is part of a triangle when it has two adjacent vertices with an edge between them. GraphX implements a triangle counting algorithm in the TriangleCount object that determines the number of triangles passing through each vertex, providing a measure of clustering. We compute the triangle count of the social network dataset from the PageRank section. Note that TriangleCount requires the edges to be in canonical orientation (srcId < dstId) and the graph to be partitioned using Graph.partitionBy. import org.apache.spark.graphx.{GraphLoader, PartitionStrategy} // Load the edges in canonical order and partition the graph for triangle count val graph = GraphLoader.edgeListFile(sc, \"data/graphx/followers.txt\", true) val triCountByUsername = users.join(triCounts).map { case (id, (username, tc)) => (username, tc) } // Print the result println(triCountByUsername.collect().mkString(\"\\n\")) Find full example code at \"examples/src/main/scala/org/apache/spark/examples/graphx/TriangleCountingExample.scala\" in the Spark repo. Examples Suppose I want to build a graph from some text files, restrict the graph to important relationships and users, run page-rank on the subgraph, and then finally return attributes associated with the top users. I can do all of this in just a few lines with org.apache.spark.graphx.GraphLoader // Load my user data and parse into tuples of user id and attribute list val users = (sc.textFile(\"data/graphx/users.txt\") // Restrict the graph to users with usernames and names val subgraph = graph.subgraph(vpred = (vid, attr) => attr.size == 2) // Compute the PageRank val pagerankGraph = subgraph.pageRank(0.001) // Get the attributes of the top pagerank users val userInfoWithPageRank = subgraph.outerJoinVertices(pagerankGraph.vertices) { case (uid, attrList, Some(pr)) => (pr, attrList.toList) case (uid, attrList, None) => (0.0, attrList.toList) } println(userInfoWithPageRank.vertices.top(5)(Ordering.by(_._2._1)).mkString(\"\\n\")) Find full example code at \"examples/src/main/scala/org/apache/spark/examples/graphx/ComprehensiveExample.scala\" in the Spark repo.\n\nExample:\n```scala\nimport org.apache.spark._\nimport org.apache.spark.graphx._\n// To make some of the examples work we will also need RDD\nimport org.apache.spark.rdd.RDD\n```\n\nExample:\n```scala\nclass VertexProperty()\ncase class UserProperty(val name: String) extends VertexProperty\ncase class ProductProperty(val name: String, val price: Double) extends VertexProperty\n// The graph might then have the type:\nvar graph: Graph[VertexProperty, String] = null\n```\n\nExample:\n```scala\nclass Graph[VD, ED] {\n  val vertices: VertexRDD[VD]\n  val edges: EdgeRDD[ED]\n}\n```\n\nExample:\n```scala\nval userGraph: Graph[(String, String), String]\n```\n\nExample:\n```scala\n// Assume the SparkContext has already been constructed\nval sc: SparkContext\n// Create an RDD for the vertices\nval users: RDD[(VertexId, (String, String))] =\n  sc.parallelize(Seq((3L, (\"rxin\", \"student\")), (7L, (\"jgonzal\", \"postdoc\")),\n                       (5L, (\"franklin\", \"prof\")), (2L, (\"istoica\", \"prof\"))))\n// Create an RDD for edges\nval relationships: RDD[Edge[String]] =\n  sc.parallelize(Seq(Edge(3L, 7L, \"collab\"),    Edge(5L, 3L, \"advisor\"),\n                       Edge(2L, 5L, \"colleague\"), Edge(5L, 7L, \"pi\")))\n// Define a default user in case there are relationship with missing user\nval defaultUser = (\"John Doe\", \"Missing\")\n// Build the initial Graph\nval graph = Graph(users, relationships, defaultUser)\n```\n\nExample:\n```scala\nval graph: Graph[(String, String), String] // Constructed from above\n// Count all users which are postdocs\ngraph.vertices.filter { case (id, (name, pos)) => pos == \"postdoc\" }.count\n// Count all the edges where src > dst\ngraph.edges.filter(e => e.srcId > e.dstId).count\n```\n\nExample:\n```scala\ngraph.edges.filter { case Edge(src, dst, prop) => src > dst }.count\n```\n\nExample:\n```sql\nSELECT src.id, dst.id, src.attr, e.attr, dst.attr\nFROM edges AS e LEFT JOIN vertices AS src, vertices AS dst\nON e.srcId = src.Id AND e.dstId = dst.Id\n```\n\nExample:\n```scala\nval graph: Graph[(String, String), String] // Constructed from above\n// Use the triplets view to create an RDD of facts.\nval facts: RDD[String] =\n  graph.triplets.map(triplet =>\n    triplet.srcAttr._1 + \" is the \" + triplet.attr + \" of \" + triplet.dstAttr._1)\nfacts.collect.foreach(println(_))\n```\n\nExample:\n```scala\nval graph: Graph[(String, String), String]\n// Use the implicit GraphOps.inDegrees operator\nval inDegrees: VertexRDD[Int] = graph.inDegrees\n```\n\nExample:\n```scala\n/** Summary of the functionality in the property graph */\nclass Graph[VD, ED] {\n  // Information about the Graph ===================================================================\n  val numEdges: Long\n  val numVertices: Long\n  val inDegrees: VertexRDD[Int]\n  val outDegrees: VertexRDD[Int]\n  val degrees: VertexRDD[Int]\n  // Views of the graph as collections =============================================================\n  val vertices: VertexRDD[VD]\n  val edges: EdgeRDD[ED]\n  val triplets: RDD[EdgeTriplet[VD, ED]]\n  // Functions for caching graphs ==================================================================\n  def persist(newLevel: StorageLevel = StorageLevel.MEMORY_ONLY): Graph[VD, ED]\n  def cache(): Graph[VD, ED]\n  def unpersistVertices(blocking: Boolean = false): Graph[VD, ED]\n  // Change the partitioning heuristic  ============================================================\n  def partitionBy(partitionStrategy: PartitionStrategy): Graph[VD, ED]\n  // Transform vertex and edge attributes ==========================================================\n  def mapVertices[VD2](map: (VertexId, VD) => VD2): Graph[VD2, ED]\n  def mapEdges[ED2](map: Edge[ED] => ED2): Graph[VD, ED2]\n  def mapEdges[ED2](map: (PartitionID, Iterator[Edge[ED]]) => Iterator[ED2]): Graph[VD, ED2]\n  def mapTriplets[ED2](map: EdgeTriplet[VD, ED] => ED2): Graph[VD, ED2]\n  def mapTriplets[ED2](map: (PartitionID, Iterator[EdgeTriplet[VD, ED]]) => Iterator[ED2])\n    : Graph[VD, ED2]\n  // Modify the graph structure ====================================================================\n  def reverse: Graph[VD, ED]\n  def subgraph(\n      epred: EdgeTriplet[VD,ED] => Boolean = (x => true),\n      vpred: (VertexId, VD) => Boolean = ((v, d) => true))\n    : Graph[VD, ED]\n  def mask[VD2, ED2](other: Graph[VD2, ED2]): Graph[VD, ED]\n  def groupEdges(merge: (ED, ED) => ED): Graph[VD, ED]\n  // Join RDDs with the graph ======================================================================\n  def joinVertices[U](table: RDD[(VertexId, U)])(mapFunc: (VertexId, VD, U) => VD): Graph[VD, ED]\n  def outerJoinVertices[U, VD2](other: RDD[(VertexId, U)])\n      (mapFunc: (VertexId, VD, Option[U]) => VD2)\n    : Graph[VD2, ED]\n  // Aggregate information about adjacent triplets =================================================\n  def collectNeighborIds(edgeDirection: EdgeDirection): VertexRDD[Array[VertexId]]\n  def collectNeighbors(edgeDirection: EdgeDirection): VertexRDD[Array[(VertexId, VD)]]\n  def aggregateMessages[Msg: ClassTag](\n      sendMsg: EdgeContext[VD, ED, Msg] => Unit,\n      mergeMsg: (Msg, Msg) => Msg,\n      tripletFields: TripletFields = TripletFields.All)\n    : VertexRDD[A]\n  // Iterative graph-parallel computation ==========================================================\n  def pregel[A](initialMsg: A, maxIterations: Int, activeDirection: EdgeDirection)(\n      vprog: (VertexId, VD, A) => VD,\n      sendMsg: EdgeTriplet[VD, ED] => Iterator[(VertexId, A)],\n      mergeMsg: (A, A) => A)\n    : Graph[VD, ED]\n  // Basic graph algorithms ========================================================================\n  def pageRank(tol: Double, resetProb: Double = 0.15): Graph[Double, Double]\n  def connectedComponents(): Graph[VertexId, ED]\n  def triangleCount(): Graph[Int, ED]\n  def stronglyConnectedComponents(numIter: Int): Graph[VertexId, ED]\n}\n```\n\nExample:\n```scala\nclass Graph[VD, ED] {\n  def mapVertices[VD2](map: (VertexId, VD) => VD2): Graph[VD2, ED]\n  def mapEdges[ED2](map: Edge[ED] => ED2): Graph[VD, ED2]\n  def mapTriplets[ED2](map: EdgeTriplet[VD, ED] => ED2): Graph[VD, ED2]\n}\n```\n\nExample:\n```scala\nval newVertices = graph.vertices.map { case (id, attr) => (id, mapUdf(id, attr)) }\nval newGraph = Graph(newVertices, graph.edges)\n```\n\nExample:\n```scala\nval newGraph = graph.mapVertices((id, attr) => mapUdf(id, attr))\n```\n\nExample:\n```scala\n// Given a graph where the vertex property is the out degree\nval inputGraph: Graph[Int, String] =\n  graph.outerJoinVertices(graph.outDegrees)((vid, _, degOpt) => degOpt.getOrElse(0))\n// Construct a graph where each edge contains the weight\n// and each vertex is the initial PageRank\nval outputGraph: Graph[Double, Double] =\n  inputGraph.mapTriplets(triplet => 1.0 / triplet.srcAttr).mapVertices((id, _) => 1.0)\n```\n\nExample:\n```scala\nclass Graph[VD, ED] {\n  def reverse: Graph[VD, ED]\n  def subgraph(epred: EdgeTriplet[VD,ED] => Boolean,\n               vpred: (VertexId, VD) => Boolean): Graph[VD, ED]\n  def mask[VD2, ED2](other: Graph[VD2, ED2]): Graph[VD, ED]\n  def groupEdges(merge: (ED, ED) => ED): Graph[VD,ED]\n}\n```\n\nExample:\n```scala\n// Create an RDD for the vertices\nval users: RDD[(VertexId, (String, String))] =\n  sc.parallelize(Seq((3L, (\"rxin\", \"student\")), (7L, (\"jgonzal\", \"postdoc\")),\n                       (5L, (\"franklin\", \"prof\")), (2L, (\"istoica\", \"prof\")),\n                       (4L, (\"peter\", \"student\"))))\n// Create an RDD for edges\nval relationships: RDD[Edge[String]] =\n  sc.parallelize(Seq(Edge(3L, 7L, \"collab\"),    Edge(5L, 3L, \"advisor\"),\n                       Edge(2L, 5L, \"colleague\"), Edge(5L, 7L, \"pi\"),\n                       Edge(4L, 0L, \"student\"),   Edge(5L, 0L, \"colleague\")))\n// Define a default user in case there are relationship with missing user\nval defaultUser = (\"John Doe\", \"Missing\")\n// Build the initial Graph\nval graph = Graph(users, relationships, defaultUser)\n// Notice that there is a user 0 (for which we have no information) connected to users\n// 4 (peter) and 5 (franklin).\ngraph.triplets.map(\n  triplet => triplet.srcAttr._1 + \" is the \" + triplet.attr + \" of \" + triplet.dstAttr._1\n).collect.foreach(println(_))\n// Remove missing vertices as well as the edges to connected to them\nval validGraph = graph.subgraph(vpred = (id, attr) => attr._2 != \"Missing\")\n// The valid subgraph will disconnect users 4 and 5 by removing user 0\nvalidGraph.vertices.collect.foreach(println(_))\nvalidGraph.triplets.map(\n  triplet => triplet.srcAttr._1 + \" is the \" + triplet.attr + \" of \" + triplet.dstAttr._1\n).collect.foreach(println(_))\n```\n\nExample:\n```scala\n// Run Connected Components\nval ccGraph = graph.connectedComponents() // No longer contains missing field\n// Remove missing vertices as well as the edges to connected to them\nval validGraph = graph.subgraph(vpred = (id, attr) => attr._2 != \"Missing\")\n// Restrict the answer to the valid subgraph\nval validCCGraph = ccGraph.mask(validGraph)\n```\n\nExample:\n```scala\nclass Graph[VD, ED] {\n  def joinVertices[U](table: RDD[(VertexId, U)])(map: (VertexId, VD, U) => VD)\n    : Graph[VD, ED]\n  def outerJoinVertices[U, VD2](table: RDD[(VertexId, U)])(map: (VertexId, VD, Option[U]) => VD2)\n    : Graph[VD2, ED]\n}\n```\n\nExample:\n```scala\nval nonUniqueCosts: RDD[(VertexId, Double)]\nval uniqueCosts: VertexRDD[Double] =\n  graph.vertices.aggregateUsingIndex(nonUnique, (a,b) => a + b)\nval joinedGraph = graph.joinVertices(uniqueCosts)(\n  (id, oldCost, extraCost) => oldCost + extraCost)\n```\n\nExample:\n```scala\nval outDegrees: VertexRDD[Int] = graph.outDegrees\nval degreeGraph = graph.outerJoinVertices(outDegrees) { (id, oldAttr, outDegOpt) =>\n  outDegOpt match {\n    case Some(outDeg) => outDeg\n    case None => 0 // No outDegree means zero outDegree\n  }\n}\n```\n\nExample:\n```scala\nval joinedGraph = graph.joinVertices(uniqueCosts,\n  (id: VertexId, oldCost: Double, extraCost: Double) => oldCost + extraCost)\n```\n\nExample:\n```scala\nclass Graph[VD, ED] {\n  def aggregateMessages[Msg: ClassTag](\n      sendMsg: EdgeContext[VD, ED, Msg] => Unit,\n      mergeMsg: (Msg, Msg) => Msg,\n      tripletFields: TripletFields = TripletFields.All)\n    : VertexRDD[Msg]\n}\n```\n\nExample:\n```text\nimport org.apache.spark.graphx.{Graph, VertexRDD}\nimport org.apache.spark.graphx.util.GraphGenerators\n\n// Create a graph with \"age\" as the vertex property.\n// Here we use a random graph for simplicity.\nval graph: Graph[Double, Int] =\n  GraphGenerators.logNormalGraph(sc, numVertices = 100).mapVertices( (id, _) => id.toDouble )\n// Compute the number of older followers and their total age\nval olderFollowers: VertexRDD[(Int, Double)] = graph.aggregateMessages[(Int, Double)](\n  triplet => { // Map Function\n    if (triplet.srcAttr > triplet.dstAttr) {\n      // Send message to destination vertex containing counter and age\n      triplet.sendToDst((1, triplet.srcAttr))\n    }\n  },\n  // Add counter and age\n  (a, b) => (a._1 + b._1, a._2 + b._2) // Reduce Function\n)\n// Divide total age by number of older followers to get average age of older followers\nval avgAgeOfOlderFollowers: VertexRDD[Double] =\n  olderFollowers.mapValues( (id, value) =>\n    value match { case (count, totalAge) => totalAge / count } )\n// Display the results\navgAgeOfOlderFollowers.collect().foreach(println(_))\n```\n\nExample:\n```scala\nclass Graph[VD, ED] {\n  def mapReduceTriplets[Msg](\n      map: EdgeTriplet[VD, ED] => Iterator[(VertexId, Msg)],\n      reduce: (Msg, Msg) => Msg)\n    : VertexRDD[Msg]\n}\n```\n\nExample:\n```scala\nval graph: Graph[Int, Float] = ...\ndef msgFun(triplet: Triplet[Int, Float]): Iterator[(Int, String)] = {\n  Iterator((triplet.dstId, \"Hi\"))\n}\ndef reduceFun(a: String, b: String): String = a + \" \" + b\nval result = graph.mapReduceTriplets[String](msgFun, reduceFun)\n```\n\nExample:\n```scala\nval graph: Graph[Int, Float] = ...\ndef msgFun(triplet: EdgeContext[Int, Float, String]) {\n  triplet.sendToDst(\"Hi\")\n}\ndef reduceFun(a: String, b: String): String = a + \" \" + b\nval result = graph.aggregateMessages[String](msgFun, reduceFun)\n```\n\nExample:\n```scala\n// Define a reduce operation to compute the highest degree vertex\ndef max(a: (VertexId, Int), b: (VertexId, Int)): (VertexId, Int) = {\n  if (a._2 > b._2) a else b\n}\n// Compute the max degrees\nval maxInDegree: (VertexId, Int)  = graph.inDegrees.reduce(max)\nval maxOutDegree: (VertexId, Int) = graph.outDegrees.reduce(max)\nval maxDegrees: (VertexId, Int)   = graph.degrees.reduce(max)\n```\n\nExample:\n```scala\nclass GraphOps[VD, ED] {\n  def collectNeighborIds(edgeDirection: EdgeDirection): VertexRDD[Array[VertexId]]\n  def collectNeighbors(edgeDirection: EdgeDirection): VertexRDD[ Array[(VertexId, VD)] ]\n}\n```\n\nExample:\n```scala\nclass GraphOps[VD, ED] {\n  def pregel[A]\n      (initialMsg: A,\n       maxIter: Int = Int.MaxValue,\n       activeDir: EdgeDirection = EdgeDirection.Out)\n      (vprog: (VertexId, VD, A) => VD,\n       sendMsg: EdgeTriplet[VD, ED] => Iterator[(VertexId, A)],\n       mergeMsg: (A, A) => A)\n    : Graph[VD, ED] = {\n    // Receive the initial message at each vertex\n    var g = mapVertices( (vid, vdata) => vprog(vid, vdata, initialMsg) ).cache()\n\n    // compute the messages\n    var messages = GraphXUtils.mapReduceTriplets(g, sendMsg, mergeMsg)\n    var activeMessages = messages.count()\n    // Loop until no messages remain or maxIterations is achieved\n    var i = 0\n    while (activeMessages > 0 && i < maxIterations) {\n      // Receive the messages and update the vertices.\n      g = g.joinVertices(messages)(vprog).cache()\n      val oldMessages = messages\n      // Send new messages, skipping edges where neither side received a message. We must cache\n      // messages so it can be materialized on the next line, allowing us to uncache the previous\n      // iteration.\n      messages = GraphXUtils.mapReduceTriplets(\n        g, sendMsg, mergeMsg, Some((oldMessages, activeDirection))).cache()\n      activeMessages = messages.count()\n      i += 1\n    }\n    g\n  }\n}\n```\n\nExample:\n```text\nimport org.apache.spark.graphx.{Graph, VertexId}\nimport org.apache.spark.graphx.util.GraphGenerators\n\n// A graph with edge attributes containing distances\nval graph: Graph[Long, Double] =\n  GraphGenerators.logNormalGraph(sc, numVertices = 100).mapEdges(e => e.attr.toDouble)\nval sourceId: VertexId = 42 // The ultimate source\n// Initialize the graph such that all vertices except the root have distance infinity.\nval initialGraph = graph.mapVertices((id, _) =>\n    if (id == sourceId) 0.0 else Double.PositiveInfinity)\nval sssp = initialGraph.pregel(Double.PositiveInfinity)(\n  (id, dist, newDist) => math.min(dist, newDist), // Vertex Program\n  triplet => {  // Send Message\n    if (triplet.srcAttr + triplet.attr < triplet.dstAttr) {\n      Iterator((triplet.dstId, triplet.srcAttr + triplet.attr))\n    } else {\n      Iterator.empty\n    }\n  },\n  (a, b) => math.min(a, b) // Merge Message\n)\nprintln(sssp.vertices.collect().mkString(\"\\n\"))\n```\n\nExample:\n```scala\nobject GraphLoader {\n  def edgeListFile(\n      sc: SparkContext,\n      path: String,\n      canonicalOrientation: Boolean = false,\n      minEdgePartitions: Int = 1)\n    : Graph[Int, Int]\n}\n```\n\nExample:\n```text\n# This is a comment\n2 1\n4 1\n1 2\n```\n\nExample:\n```scala\nobject Graph {\n  def apply[VD, ED](\n      vertices: RDD[(VertexId, VD)],\n      edges: RDD[Edge[ED]],\n      defaultVertexAttr: VD = null)\n    : Graph[VD, ED]\n\n  def fromEdges[VD, ED](\n      edges: RDD[Edge[ED]],\n      defaultValue: VD): Graph[VD, ED]\n\n  def fromEdgeTuples[VD](\n      rawEdges: RDD[(VertexId, VertexId)],\n      defaultValue: VD,\n      uniqueEdges: Option[PartitionStrategy] = None): Graph[VD, Int]\n\n}\n```\n\nExample:\n```scala\nclass VertexRDD[VD] extends RDD[(VertexId, VD)] {\n  // Filter the vertex set but preserves the internal index\n  def filter(pred: Tuple2[VertexId, VD] => Boolean): VertexRDD[VD]\n  // Transform the values without changing the ids (preserves the internal index)\n  def mapValues[VD2](map: VD => VD2): VertexRDD[VD2]\n  def mapValues[VD2](map: (VertexId, VD) => VD2): VertexRDD[VD2]\n  // Show only vertices unique to this set based on their VertexId's\n  def minus(other: RDD[(VertexId, VD)])\n  // Remove vertices from this set that appear in the other set\n  def diff(other: VertexRDD[VD]): VertexRDD[VD]\n  // Join operators that take advantage of the internal indexing to accelerate joins (substantially)\n  def leftJoin[VD2, VD3](other: RDD[(VertexId, VD2)])(f: (VertexId, VD, Option[VD2]) => VD3): VertexRDD[VD3]\n  def innerJoin[U, VD2](other: RDD[(VertexId, U)])(f: (VertexId, VD, U) => VD2): VertexRDD[VD2]\n  // Use the index on this RDD to accelerate a `reduceByKey` operation on the input RDD.\n  def aggregateUsingIndex[VD2](other: RDD[(VertexId, VD2)], reduceFunc: (VD2, VD2) => VD2): VertexRDD[VD2]\n}\n```\n\nExample:\n```scala\nval setA: VertexRDD[Int] = VertexRDD(sc.parallelize(0L until 100L).map(id => (id, 1)))\nval rddB: RDD[(VertexId, Double)] = sc.parallelize(0L until 100L).flatMap(id => List((id, 1.0), (id, 2.0)))\n// There should be 200 entries in rddB\nrddB.count\nval setB: VertexRDD[Double] = setA.aggregateUsingIndex(rddB, _ + _)\n// There should be 100 entries in setB\nsetB.count\n// Joining A and B should now be fast!\nval setC: VertexRDD[Double] = setA.innerJoin(setB)((id, a, b) => a + b)\n```\n\nExample:\n```scala\n// Transform the edge attributes while preserving the structure\ndef mapValues[ED2](f: Edge[ED] => ED2): EdgeRDD[ED2]\n// Reverse the edges reusing both attributes and structure\ndef reverse: EdgeRDD[ED]\n// Join two `EdgeRDD`s partitioned using the same partitioning strategy.\ndef innerJoin[ED2, ED3](other: EdgeRDD[ED2])(f: (VertexId, VertexId, ED, ED2) => ED3): EdgeRDD[ED3]\n```\n\nExample:\n```text\nimport org.apache.spark.graphx.GraphLoader\n\n// Load the edges as a graph\nval graph = GraphLoader.edgeListFile(sc, \"data/graphx/followers.txt\")\n// Run PageRank\nval ranks = graph.pageRank(0.0001).vertices\n// Join the ranks with the usernames\nval users = sc.textFile(\"data/graphx/users.txt\").map { line =>\n  val fields = line.split(\",\")\n  (fields(0).toLong, fields(1))\n}\nval ranksByUsername = users.join(ranks).map {\n  case (id, (username, rank)) => (username, rank)\n}\n// Print the result\nprintln(ranksByUsername.collect().mkString(\"\\n\"))\n```\n\nExample:\n```text\nimport org.apache.spark.graphx.GraphLoader\n\n// Load the graph as in the PageRank example\nval graph = GraphLoader.edgeListFile(sc, \"data/graphx/followers.txt\")\n// Find the connected components\nval cc = graph.connectedComponents().vertices\n// Join the connected components with the usernames\nval users = sc.textFile(\"data/graphx/users.txt\").map { line =>\n  val fields = line.split(\",\")\n  (fields(0).toLong, fields(1))\n}\nval ccByUsername = users.join(cc).map {\n  case (id, (username, cc)) => (username, cc)\n}\n// Print the result\nprintln(ccByUsername.collect().mkString(\"\\n\"))\n```\n\nExample:\n```text\nimport org.apache.spark.graphx.{GraphLoader, PartitionStrategy}\n\n// Load the edges in canonical order and partition the graph for triangle count\nval graph = GraphLoader.edgeListFile(sc, \"data/graphx/followers.txt\", true)\n  .partitionBy(PartitionStrategy.RandomVertexCut)\n// Find the triangle count for each vertex\nval triCounts = graph.triangleCount().vertices\n// Join the triangle counts with the usernames\nval users = sc.textFile(\"data/graphx/users.txt\").map { line =>\n  val fields = line.split(\",\")\n  (fields(0).toLong, fields(1))\n}\nval triCountByUsername = users.join(triCounts).map { case (id, (username, tc)) =>\n  (username, tc)\n}\n// Print the result\nprintln(triCountByUsername.collect().mkString(\"\\n\"))\n```\n\nExample:\n```text\nimport org.apache.spark.graphx.GraphLoader\n\n// Load my user data and parse into tuples of user id and attribute list\nval users = (sc.textFile(\"data/graphx/users.txt\")\n  .map(line => line.split(\",\")).map( parts => (parts.head.toLong, parts.tail) ))\n\n// Parse the edge data which is already in userId -> userId format\nval followerGraph = GraphLoader.edgeListFile(sc, \"data/graphx/followers.txt\")\n\n// Attach the user attributes\nval graph = followerGraph.outerJoinVertices(users) {\n  case (uid, deg, Some(attrList)) => attrList\n  // Some users may not have attributes so we set them as empty\n  case (uid, deg, None) => Array.empty[String]\n}\n\n// Restrict the graph to users with usernames and names\nval subgraph = graph.subgraph(vpred = (vid, attr) => attr.size == 2)\n\n// Compute the PageRank\nval pagerankGraph = subgraph.pageRank(0.001)\n\n// Get the attributes of the top pagerank users\nval userInfoWithPageRank = subgraph.outerJoinVertices(pagerankGraph.vertices) {\n  case (uid, attrList, Some(pr)) => (pr, attrList.toList)\n  case (uid, attrList, None) => (0.0, attrList.toList)\n}\n\nprintln(userInfoWithPageRank.vertices.top(5)(Ordering.by(_._2._1)).mkString(\"\\n\"))\n```\n\n<|endofdoc|>","metadata":{"transformedAt":"2026-08-18T15:21:43.033Z","totalSectionsIncluded":1,"totalCodeBlocksIncluded":41,"totalLines":598,"estimatedTokens":16551}}17{"id":"doc-cluster_mode_overview_spark_4_2_0_documentation-8c5d7c51","source":"documentation","title":"Cluster Mode Overview - Spark 4.2.0 Documentation","url":"https://spark.apache.org/docs/latest/cluster-overview.html","text":"Cluster Mode Overview This document gives a short overview of how Spark runs on clusters, to make it easier to understand the components involved. Read through the application submission guide to learn about launching applications on a cluster. Components Spark applications run as independent sets of processes on a cluster, coordinated by the SparkContext object in your main program (called the driver program). Specifically, to run on a cluster, the SparkContext can connect to several types of cluster managers (either Spark’s own standalone cluster manager, YARN or Kubernetes), which allocate resources across applications. Once connected, Spark acquires executors on nodes in the cluster, which are processes that run computations and store data for your application. Next, it sends your application code (defined by JAR or Python files passed to SparkContext) to the executors. Finally, SparkContext sends tasks to the executors to run. There are several useful things to note about this application gets its own executor processes, which stay up for the duration of the whole application and run tasks in multiple threads. This has the benefit of isolating applications from each other, on both the scheduling side (each driver schedules its own tasks) and executor side (tasks from different applications run in different JVMs). However, it also means that data cannot be shared across different Spark applications (instances of SparkContext) without writing it to an external storage system. Spark is agnostic to the underlying cluster manager. As long as it can acquire executor processes, and these communicate with each other, it is relatively easy to run it even on a cluster manager that also supports other applications (e.g. YARN/Kubernetes). The driver program must listen for and accept incoming connections from its executors throughout its lifetime (e.g., see spark.driver.port in the network config section). As such, the driver program must be network addressable from the worker nodes. Because the driver schedules tasks on the cluster, it should be run close to the worker nodes, preferably on the same local area network. If you’d like to send requests to the cluster remotely, it’s better to open an RPC to the driver and have it submit operations from nearby than to run a driver far away from the worker nodes. Cluster Manager Types The system currently supports several cluster – a simple cluster manager included with Spark that makes it easy to set up a cluster. Hadoop YARN – the resource manager in Hadoop 3. Kubernetes – an open-source system for automating deployment, scaling, and management of containerized applications. Submitting Applications Applications can be submitted to a cluster of any type using the spark-submit script. The application submission guide describes how to do this. Monitoring Each driver program has a web UI, typically on port 4040, that displays information about running tasks, executors, and storage usage. Simply go to http://<driver-node>:4040 in a web browser to access this UI. The monitoring guide also describes other monitoring options. Job Scheduling Spark gives control over resource allocation both across applications (at the level of the cluster manager) and within applications (if multiple computations are happening on the same SparkContext). The job scheduling overview describes this in more detail. Glossary The following table summarizes terms you’ll see used to refer to cluster Application User program built on Spark. Consists of a driver program and executors on the cluster. Application jar A jar containing the user's Spark application. In some cases users will want to create an \"uber jar\" containing their application along with its dependencies. The user's jar should never include Hadoop or Spark libraries, however, these will be added at runtime. Driver program The process running the main() function of the application and creating the SparkContext Cluster manager An external service for acquiring resources on the cluster (e.g. standalone manager, YARN, Kubernetes) Deploy mode Distinguishes where the driver process runs. In \"cluster\" mode, the framework launches the driver inside of the cluster. In \"client\" mode, the submitter launches the driver outside of the cluster. Worker node Any node that can run application code in the cluster Executor A process launched for an application on a worker node, that runs tasks and keeps data in memory or disk storage across them. Each application has its own executors. Task A unit of work that will be sent to one executor Job A parallel computation consisting of multiple tasks that gets spawned in response to a Spark action (e.g. save, collect); you'll see this term used in the driver's logs. Stage Each job gets divided into smaller sets of tasks called stages that depend on each other (similar to the map and reduce stages in MapReduce); you'll see this term used in the driver's logs.\n\n<|endofdoc|>","metadata":{"transformedAt":"2026-08-18T15:21:43.037Z","totalSectionsIncluded":1,"totalCodeBlocksIncluded":0,"totalLines":3,"estimatedTokens":1240}}18{"id":"doc-using_spark_s_hadoop_free_build_spark_4_2_0_docu-d80b9aba","source":"documentation","title":"Using Spark's \"Hadoop Free\" Build - Spark 4.2.0 Documentation","url":"https://spark.apache.org/docs/latest/hadoop-provided.html","text":"Using Spark's \"Hadoop Free\" Build Spark uses Hadoop client libraries for HDFS and YARN. Starting in version Spark 1.4, the project packages “Hadoop free” builds that lets you more easily connect a single Spark binary to any Hadoop version. To use these builds, you need to modify SPARK_DIST_CLASSPATH to include Hadoop’s package jars. The most convenient place to do this is by adding an entry in conf/spark-env.sh. This page describes how to connect Spark to Hadoop for different types of distributions. Apache Hadoop For Apache distributions, you can use Hadoop’s ‘classpath’ command. For instance: ### in conf/spark-env.sh ### # If 'hadoop' binary is on your PATH export SPARK_DIST_CLASSPATH=$(hadoop classpath) # With explicit path to 'hadoop' binary export SPARK_DIST_CLASSPATH=$(/path/to/hadoop/bin/hadoop classpath) # Passing a Hadoop configuration directory export SPARK_DIST_CLASSPATH=$(hadoop --config /path/to/configs classpath) Hadoop Free Build Setup for Spark on Kubernetes To run the Hadoop free build of Spark on Kubernetes, the executor image must have the appropriate version of Hadoop binaries and the correct SPARK_DIST_CLASSPATH value set. See the example below for the relevant changes needed in the executor Dockerfile: ### Set environment variables in the executor dockerfile ### ENV SPARK_HOME=\"/opt/spark\" ENV HADOOP_HOME=\"/opt/hadoop\" ENV PATH=\"$SPARK_HOME/bin:$HADOOP_HOME/bin:$PATH\" ... #Copy your target hadoop binaries to the executor hadoop home COPY /opt/hadoop3 $HADOOP_HOME ... #Copy and use the Spark provided entrypoint.sh. It sets your SPARK_DIST_CLASSPATH using the hadoop binary in $HADOOP_HOME and starts the executor. If you choose to customize the value of SPARK_DIST_CLASSPATH here, the value will be retained in entrypoint.sh ENTRYPOINT [ \"/opt/entrypoint.sh\" ] ...\n\nExample:\n```bash\n### in conf/spark-env.sh ###\n\n# If 'hadoop' binary is on your PATH\nexport SPARK_DIST_CLASSPATH=$(hadoop classpath)\n\n# With explicit path to 'hadoop' binary\nexport SPARK_DIST_CLASSPATH=$(/path/to/hadoop/bin/hadoop classpath)\n\n# Passing a Hadoop configuration directory\nexport SPARK_DIST_CLASSPATH=$(hadoop --config /path/to/configs classpath)\n```\n\nExample:\n```bash\n### Set environment variables in the executor dockerfile ###\n\nENV SPARK_HOME=\"/opt/spark\"  \nENV HADOOP_HOME=\"/opt/hadoop\"  \nENV PATH=\"$SPARK_HOME/bin:$HADOOP_HOME/bin:$PATH\"  \n...  \n\n#Copy your target hadoop binaries to the executor hadoop home   \n\nCOPY /opt/hadoop3  $HADOOP_HOME  \n...\n\n#Copy and use the Spark provided entrypoint.sh. It sets your SPARK_DIST_CLASSPATH using the hadoop binary in $HADOOP_HOME and starts the executor. If you choose to customize the value of SPARK_DIST_CLASSPATH here, the value will be retained in entrypoint.sh\n\nENTRYPOINT [ \"/opt/entrypoint.sh\" ]\n...\n```\n\n<|endofdoc|>","metadata":{"transformedAt":"2026-08-18T15:21:43.038Z","totalSectionsIncluded":1,"totalCodeBlocksIncluded":2,"totalLines":37,"estimatedTokens":700}}19{"id":"doc-rdd_programming_guide_spark_4_2_0_documentation-ee8fb804","source":"documentation","title":"RDD Programming Guide - Spark 4.2.0 Documentation","url":"https://spark.apache.org/docs/latest/rdd-programming-guide.html","text":"RDD Programming Guide Overview Linking with Spark Initializing Spark Using the Shell Resilient Distributed Datasets (RDDs) Parallelized Collections External Datasets RDD Operations Basics Passing Functions to Spark Understanding closures Example Local vs. cluster modes Printing elements of an RDD Working with Key-Value Pairs Transformations Actions Shuffle operations Background Performance Impact RDD Persistence Which Storage Level to Choose? Removing Data Shared Variables Broadcast Variables Accumulators Deploying to a Cluster Launching Spark jobs from Java / Scala Unit Testing Where to Go from Here Overview At a high level, every Spark application consists of a driver program that runs the user’s main function and executes various parallel operations on a cluster. The main abstraction Spark provides is a resilient distributed dataset (RDD), which is a collection of elements partitioned across the nodes of the cluster that can be operated on in parallel. RDDs are created by starting with a file in the Hadoop file system (or any other Hadoop-supported file system), or an existing Scala collection in the driver program, and transforming it. Users may also ask Spark to persist an RDD in memory, allowing it to be reused efficiently across parallel operations. Finally, RDDs automatically recover from node failures. A second abstraction in Spark is shared variables that can be used in parallel operations. By default, when Spark runs a function in parallel as a set of tasks on different nodes, it ships a copy of each variable used in the function to each task. Sometimes, a variable needs to be shared across tasks, or between tasks and the driver program. Spark supports two types of shared variables, which can be used to cache a value in memory on all nodes, and accumulators, which are variables that are only “added” to, such as counters and sums. This guide shows each of these features in each of Spark’s supported languages. It is easiest to follow along with if you launch Spark’s interactive shell – either bin/spark-shell for the Scala shell or bin/pyspark for the Python one. Linking with Spark Spark 4.2.0 works with Python 3.10+. It can use the standard CPython interpreter, so C libraries like NumPy can be used. Spark applications in Python can either be run with the bin/spark-submit script which includes Spark at runtime, or by including it in your setup.py =[ 'pyspark==4.2.0' ] To run Spark applications in Python without pip installing PySpark, use the bin/spark-submit script located in the Spark directory. This script will load Spark’s Java/Scala libraries and allow you to submit applications to a cluster. You can also use bin/pyspark to launch an interactive Python shell. If you wish to access HDFS data, you need to use a build of PySpark linking to your version of HDFS. Prebuilt packages are also available on the Spark homepage for common HDFS versions. Finally, you need to import some Spark classes into your program. Add the following pyspark import SparkContext, SparkConf PySpark requires the same minor version of Python in both driver and workers. It uses the default python version in PATH, you can specify which version of Python you want to use by PYSPARK_PYTHON, for example: $ PYSPARK_PYTHON=python3.8 bin/pyspark Spark 4.2.0 is built and distributed to work with Scala 2.13 by default. (Spark can be built to work with other versions of Scala, too.) To write applications in Scala, you will need to use a compatible Scala version (e.g. 2.13.X). To write a Spark application, you need to add a Maven dependency on Spark. Spark is available through Maven Central = org.apache.spark artifactId = spark-core_2.13 version = 4.2.0 In addition, if you wish to access an HDFS cluster, you need to add a dependency on hadoop-client for your version of HDFS. groupId = org.apache.hadoop artifactId = hadoop-client version = <your-hdfs-version> Finally, you need to import some Spark classes into your program. Add the following org.apache.spark.SparkContext import org.apache.spark.SparkConf (Before Spark 1.3.0, you need to explicitly import org.apache.spark.SparkContext._ to enable essential implicit conversions.) Spark 4.2.0 supports lambda expressions for concisely writing functions, otherwise you can use the classes in the org.apache.spark.api.java.function package. Note that support for Java 7 was removed in Spark 2.2.0. To write a Spark application in Java, you need to add a dependency on Spark. Spark is available through Maven Central = org.apache.spark artifactId = spark-core_2.13 version = 4.2.0 In addition, if you wish to access an HDFS cluster, you need to add a dependency on hadoop-client for your version of HDFS. groupId = org.apache.hadoop artifactId = hadoop-client version = <your-hdfs-version> Finally, you need to import some Spark classes into your program. Add the following org.apache.spark.api.java.JavaSparkContext; import org.apache.spark.api.java.JavaRDD; import org.apache.spark.SparkConf; Initializing Spark The first thing a Spark program must do is to create a SparkContext object, which tells Spark how to access a cluster. To create a SparkContext you first need to build a SparkConf object that contains information about your application. conf = SparkConf().setAppName(appName).setMaster(master) sc = SparkContext(conf=conf) The first thing a Spark program must do is to create a SparkContext object, which tells Spark how to access a cluster. To create a SparkContext you first need to build a SparkConf object that contains information about your application. Only one SparkContext should be active per JVM. You must stop() the active SparkContext before creating a new one. val conf = new SparkConf().setAppName(appName).setMaster(master) new SparkContext(conf) The first thing a Spark program must do is to create a JavaSparkContext object, which tells Spark how to access a cluster. To create a SparkContext you first need to build a SparkConf object that contains information about your application. SparkConf conf = new SparkConf().setAppName(appName).setMaster(master); JavaSparkContext sc = new JavaSparkContext(conf); The appName parameter is a name for your application to show on the cluster UI. master is a Spark or YARN cluster URL, or a special “local” string to run in local mode. In practice, when running on a cluster, you will not want to hardcode master in the program, but rather launch the application with spark-submit and receive it there. However, for local testing and unit tests, you can pass “local” to run Spark in-process. Using the Shell In the PySpark shell, a special interpreter-aware SparkContext is already created for you, in the variable called sc. Making your own SparkContext will not work. You can set which master the context connects to using the --master argument, and you can add Python # assume Elasticsearch is running on localhost defaults >>> rdd = sc.newAPIHadoopRDD(\"org.elasticsearch.hadoop.mr.EsInputFormat\", \"org.apache.hadoop.io.NullWritable\", \"org.elasticsearch.hadoop.mr.LinkedMapWritable\", conf=conf) >>> rdd.first() # the result is a MapWritable that is converted to a Python dict (u'Elasticsearch ID', {u'field1': True, u'field2': u'Some Text', u'field3': 12345}) Note that, if the InputFormat simply depends on a Hadoop configuration and/or input path, and the key and value classes can easily be converted according to the above table, then this approach should work well for such cases. If you have custom serialized binary data (such as loading data from Cassandra / HBase), then you will first need to transform that data on the Scala/Java side to something which can be handled by pickle’s pickler. A Converter trait is provided for this. Simply extend this trait and implement your transformation code in the convert method. Remember to ensure that this class, along with any dependencies required to access your InputFormat, are packaged into your Spark job jar and included on the PySpark classpath. See the Python examples and the Converter examples for examples of using Cassandra / HBase InputFormat and OutputFormat with custom converters. Spark can create distributed datasets from any storage source supported by Hadoop, including your local file system, HDFS, Cassandra, HBase, Amazon S3, etc. Spark supports text files, SequenceFiles, and any other Hadoop InputFormat. Text file RDDs can be created using SparkContext’s textFile method. This method takes a URI for the file (either a local path on the machine, or a hdfs://, s3a://, etc URI) and reads it as a collection of lines. Here is an example > val distFile = sc.textFile(\"data.txt\") [String] = data.txt MapPartitionsRDD[10] at textFile at <console>:26 Once created, distFile can be acted on by dataset operations. For example, we can add up the sizes of all the lines using the map and reduce operations as (s => s.length).reduce((a, b) => a + b). Some notes on reading files with using a path on the local filesystem, the file must also be accessible at the same path on worker nodes. Either copy the file to all workers or use a network-mounted shared file system. All of Spark’s file-based input methods, including textFile, support running on directories, compressed files, and wildcards as well. For example, you can use textFile(\"/my/directory\"), textFile(\"/my/directory/*.txt\"), and textFile(\"/my/directory/*.gz\"). When multiple files are read, the order of the partitions depends on the order the files are returned from the filesystem. It may or may not, for example, follow the lexicographic ordering of the files by path. Within a partition, elements are ordered according to their order in the underlying file. The textFile method also takes an optional second argument for controlling the number of partitions of the file. By default, Spark creates one partition for each block of the file (blocks being 128MB by default in HDFS), but you can also ask for a higher number of partitions by passing a larger value. Note that you cannot have fewer partitions than blocks. Apart from text files, Spark’s Scala API also supports several other data lets you read a directory containing multiple small text files, and returns each of them as (filename, content) pairs. This is in contrast with textFile, which would return one record per line in each file. Partitioning is determined by data locality which, in some cases, may result in too few partitions. For those cases, wholeTextFiles provides an optional second argument for controlling the minimal number of partitions. For SequenceFiles, use SparkContext’s sequenceFile[K, V] method where K and V are the types of key and values in the file. These should be subclasses of Hadoop’s Writable interface, like IntWritable and Text. In addition, Spark allows you to specify native types for a few common Writables; for example, sequenceFile[Int, String] will automatically read IntWritables and Texts. For other Hadoop InputFormats, you can use the SparkContext.hadoopRDD method, which takes an arbitrary JobConf and input format class, key class and value class. Set these the same way you would for a Hadoop job with your input source. You can also use SparkContext.newAPIHadoopRDD for InputFormats based on the “new” MapReduce API (org.apache.hadoop.mapreduce). RDD.saveAsObjectFile and SparkContext.objectFile support saving an RDD in a simple format consisting of serialized Java objects. While this is not as efficient as specialized formats like Avro, it offers an easy way to save any RDD. Spark can create distributed datasets from any storage source supported by Hadoop, including your local file system, HDFS, Cassandra, HBase, Amazon S3, etc. Spark supports text files, SequenceFiles, and any other Hadoop InputFormat. Text file RDDs can be created using SparkContext’s textFile method. This method takes a URI for the file (either a local path on the machine, or a hdfs://, s3a://, etc URI) and reads it as a collection of lines. Here is an example <String> distFile = sc.textFile(\"data.txt\"); Once created, distFile can be acted on by dataset operations. For example, we can add up the sizes of all the lines using the map and reduce operations as (s -> s.length()).reduce((a, b) -> a + b). Some notes on reading files with using a path on the local filesystem, the file must also be accessible at the same path on worker nodes. Either copy the file to all workers or use a network-mounted shared file system. All of Spark’s file-based input methods, including textFile, support running on directories, compressed files, and wildcards as well. For example, you can use textFile(\"/my/directory\"), textFile(\"/my/directory/*.txt\"), and textFile(\"/my/directory/*.gz\"). The textFile method also takes an optional second argument for controlling the number of partitions of the file. By default, Spark creates one partition for each block of the file (blocks being 128MB by default in HDFS), but you can also ask for a higher number of partitions by passing a larger value. Note that you cannot have fewer partitions than blocks. Apart from text files, Spark’s Java API also supports several other data lets you read a directory containing multiple small text files, and returns each of them as (filename, content) pairs. This is in contrast with textFile, which would return one record per line in each file. For SequenceFiles, use SparkContext’s sequenceFile[K, V] method where K and V are the types of key and values in the file. These should be subclasses of Hadoop’s Writable interface, like IntWritable and Text. For other Hadoop InputFormats, you can use the JavaSparkContext.hadoopRDD method, which takes an arbitrary JobConf and input format class, key class and value class. Set these the same way you would for a Hadoop job with your input source. You can also use JavaSparkContext.newAPIHadoopRDD for InputFormats based on the “new” MapReduce API (org.apache.hadoop.mapreduce). JavaRDD.saveAsObjectFile and JavaSparkContext.objectFile support saving an RDD in a simple format consisting of serialized Java objects. While this is not as efficient as specialized formats like Avro, it offers an easy way to save any RDD. RDD Operations RDDs support two types of , which create a new dataset from an existing one, and actions, which return a value to the driver program after running a computation on the dataset. For example, map is a transformation that passes each dataset element through a function and returns a new RDD representing the results. On the other hand, reduce is an action that aggregates all the elements of the RDD using some function and returns the final result to the driver program (although there is also a parallel reduceByKey that returns a distributed dataset). All transformations in Spark are lazy, in that they do not compute their results right away. Instead, they just remember the transformations applied to some base dataset (e.g. a file). The transformations are only computed when an action requires a result to be returned to the driver program. This design enables Spark to run more efficiently. For example, we can realize that a dataset created through map will be used in a reduce and return only the result of the reduce to the driver, rather than the larger mapped dataset. By default, each transformed RDD may be recomputed each time you run an action on it. However, you may also persist an RDD in memory using the persist (or cache) method, in which case Spark will keep the elements around on the cluster for much faster access the next time you query it. There is also support for persisting RDDs on disk, or replicated across multiple nodes. Basics To illustrate RDD basics, consider the simple program = sc.textFile(\"data.txt\") lineLengths = lines.map(lambda (s)) totalLength = lineLengths.reduce(lambda a, + b) The first line defines a base RDD from an external file. This dataset is not loaded in memory or otherwise acted is merely a pointer to the file. The second line defines lineLengths as the result of a map transformation. Again, lineLengths is not immediately computed, due to laziness. Finally, we run reduce, which is an action. At this point Spark breaks the computation into tasks to run on separate machines, and each machine runs both its part of the map and a local reduction, returning only its answer to the driver program. If we also wanted to use lineLengths again later, we could () before the reduce, which would cause lineLengths to be saved in memory after the first time it is computed. To illustrate RDD basics, consider the simple program lines = sc.textFile(\"data.txt\") val lineLengths = lines.map(s => s.length) val totalLength = lineLengths.reduce((a, b) => a + b) The first line defines a base RDD from an external file. This dataset is not loaded in memory or otherwise acted is merely a pointer to the file. The second line defines lineLengths as the result of a map transformation. Again, lineLengths is not immediately computed, due to laziness. Finally, we run reduce, which is an action. At this point Spark breaks the computation into tasks to run on separate machines, and each machine runs both its part of the map and a local reduction, returning only its answer to the driver program. If we also wanted to use lineLengths again later, we could () before the reduce, which would cause lineLengths to be saved in memory after the first time it is computed. To illustrate RDD basics, consider the simple program <String> lines = sc.textFile(\"data.txt\"); JavaRDD<Integer> lineLengths = lines.map(s -> s.length()); int totalLength = lineLengths.reduce((a, b) -> a + b); The first line defines a base RDD from an external file. This dataset is not loaded in memory or otherwise acted is merely a pointer to the file. The second line defines lineLengths as the result of a map transformation. Again, lineLengths is not immediately computed, due to laziness. Finally, we run reduce, which is an action. At this point Spark breaks the computation into tasks to run on separate machines, and each machine runs both its part of the map and a local reduction, returning only its answer to the driver program. If we also wanted to use lineLengths again later, we could (StorageLevel.MEMORY_ONLY()); before the reduce, which would cause lineLengths to be saved in memory after the first time it is computed. Passing Functions to Spark Spark’s API relies heavily on passing functions in the driver program to run on the cluster. There are three recommended ways to do expressions, for simple functions that can be written as an expression. (Lambdas do not support multi-statement functions or statements that do not return a value.) Local defs inside the function calling into Spark, for longer code. Top-level functions in a module. For example, to pass a longer function than can be supported using a lambda, consider the code below: \"\"\"MyScript.py\"\"\" if __name__ == \"__main__\": def myFunc(s): words = s.split(\" \") return len(words) sc = SparkContext(...) sc.textFile(\"file.txt\").map(myFunc) Note that while it is also possible to pass a reference to a method in a class instance (as opposed to a singleton object), this requires sending the object that contains that class along with the method. For example, MyClass(object): def func(self, s): return s def doStuff(self, rdd): return rdd.map(self.func) Here, if we create a new MyClass and call doStuff on it, the map inside there references the func method of that MyClass instance, so the whole object needs to be sent to the cluster. In a similar way, accessing fields of the outer object will reference the whole MyClass(object): def __init__(self): self.field = \"Hello\" def doStuff(self, rdd): return rdd.map(lambda + s) To avoid this issue, the simplest way is to copy field into a local variable instead of accessing it doStuff(self, rdd): field = self.field return rdd.map(lambda + s) Spark’s API relies heavily on passing functions in the driver program to run on the cluster. There are two recommended ways to do function syntax, which can be used for short pieces of code. Static methods in a global singleton object. For example, you can define object MyFunctions and then pass MyFunctions.func1, as MyFunctions { def func1(s: String): String = { ... } } myRdd.map(MyFunctions.func1) Note that while it is also possible to pass a reference to a method in a class instance (as opposed to a singleton object), this requires sending the object that contains that class along with the method. For example, MyClass { def func1(s: String): String = { ... } def doStuff(rdd: RDD[String]): RDD[String] = { rdd.map(func1) } } Here, if we create a new MyClass instance and call doStuff on it, the map inside there references the func1 method of that MyClass instance, so the whole object needs to be sent to the cluster. It is similar to writing rdd.map(x => this.func1(x)). In a similar way, accessing fields of the outer object will reference the whole MyClass { val field = \"Hello\" def doStuff(rdd: RDD[String]): RDD[String] = { rdd.map(x => field + x) } } is equivalent to writing rdd.map(x => this.field + x), which references all of this. To avoid this issue, the simplest way is to copy field into a local variable instead of accessing it doStuff(rdd: RDD[String]): RDD[String] = { val field_ = this.field rdd.map(x => field_ + x) } Spark’s API relies heavily on passing functions in the driver program to run on the cluster. In Java, functions are represented by classes implementing the interfaces in the org.apache.spark.api.java.function package. There are two ways to create such the Function interfaces in your own class, either as an anonymous inner class or a named one, and pass an instance of it to Spark. Use lambda expressions to concisely define an implementation. While much of this guide uses lambda syntax for conciseness, it is easy to use all the same APIs in long-form. For example, we could have written our code above as <String> lines = sc.textFile(\"data.txt\"); JavaRDD<Integer> lineLengths = lines.map(new Function<String, Integer>() { public Integer call(String s) { return s.length(); } }); int totalLength = lineLengths.reduce(new Function2<Integer, Integer, Integer>() { public Integer call(Integer a, Integer b) { return a + b; } }); Or, if writing the functions inline is GetLength implements Function<String, Integer> { public Integer call(String s) { return s.length(); } } class Sum implements Function2<Integer, Integer, Integer> { public Integer call(Integer a, Integer b) { return a + b; } } JavaRDD<String> lines = sc.textFile(\"data.txt\"); JavaRDD<Integer> lineLengths = lines.map(new GetLength()); int totalLength = lineLengths.reduce(new Sum()); Note that anonymous inner classes in Java can also access variables in the enclosing scope as long as they are marked final. Spark will ship copies of these variables to each worker node as it does for other languages. Understanding closures One of the harder things about Spark is understanding the scope and life cycle of variables and methods when executing code across a cluster. RDD operations that modify variables outside of their scope can be a frequent source of confusion. In the example below we’ll look at code that uses foreach() to increment a counter, but similar issues can occur for other operations as well. Example Consider the naive RDD element sum below, which may behave differently depending on whether execution is happening within the same JVM. A common example of this is when running Spark in local mode (--master = \"local[n]\") versus deploying a Spark application to a cluster (e.g. via spark-submit to YARN): counter = 0 rdd = sc.parallelize(data) # 't do this!! def increment_counter(x): global counter counter += x rdd.foreach(increment_counter) print(\"Counter value: \", counter) var counter = 0 var rdd = sc.parallelize(data) // 't do this!! rdd.foreach(x => counter += x) println(\"Counter value: \" + counter) int counter = 0; JavaRDD<Integer> rdd = sc.parallelize(data); // 't do this!! rdd.foreach(x -> counter += x); println(\"Counter value: \" + counter); Local vs. cluster modes The behavior of the above code is undefined, and may not work as intended. To execute jobs, Spark breaks up the processing of RDD operations into tasks, each of which is executed by an executor. Prior to execution, Spark computes the task’s closure. The closure is those variables and methods which must be visible for the executor to perform its computations on the RDD (in this case foreach()). This closure is serialized and sent to each executor. The variables within the closure sent to each executor are now copies and thus, when counter is referenced within the foreach function, it’s no longer the counter on the driver node. There is still a counter in the memory of the driver node but this is no longer visible to the executors! The executors only see the copy from the serialized closure. Thus, the final value of counter will still be zero since all operations on counter were referencing the value within the serialized closure. In local mode, in some circumstances, the foreach function will actually execute within the same JVM as the driver and will reference the same original counter, and may actually update it. To ensure well-defined behavior in these sorts of scenarios one should use an Accumulator. Accumulators in Spark are used specifically to provide a mechanism for safely updating a variable when execution is split up across worker nodes in a cluster. The Accumulators section of this guide discusses these in more detail. In general, closures - constructs like loops or locally defined methods, should not be used to mutate some global state. Spark does not define or guarantee the behavior of mutations to objects referenced from outside of closures. Some code that does this may work in local mode, but that’s just by accident and such code will not behave as expected in distributed mode. Use an Accumulator instead if some global aggregation is needed. Printing elements of an RDD Another common idiom is attempting to print out the elements of an RDD using rdd.foreach(println) or rdd.map(println). On a single machine, this will generate the expected output and print all the RDD’s elements. However, in cluster mode, the output to stdout being called by the executors is now writing to the executor’s stdout instead, not the one on the driver, so stdout on the driver won’t show these! To print all elements on the driver, one can use the collect() method to first bring the RDD to the driver node ().foreach(println). This can cause the driver to run out of memory, though, because collect() fetches the entire RDD to a single machine; if you only need to print a few elements of the RDD, a safer approach is to use the take(): rdd.take(100).foreach(println). Working with Key-Value Pairs While most Spark operations work on RDDs containing any type of objects, a few special operations are only available on RDDs of key-value pairs. The most common ones are distributed “shuffle” operations, such as grouping or aggregating the elements by a key. In Python, these operations work on RDDs containing built-in Python tuples such as (1, 2). Simply create such tuples and then call your desired operation. For example, the following code uses the reduceByKey operation on key-value pairs to count how many times each line of text occurs in a = sc.textFile(\"data.txt\") pairs = lines.map(lambda s: (s, 1)) counts = pairs.reduceByKey(lambda a, + b) We could also use counts.sortByKey(), for example, to sort the pairs alphabetically, and finally counts.collect() to bring them back to the driver program as a list of objects. While most Spark operations work on RDDs containing any type of objects, a few special operations are only available on RDDs of key-value pairs. The most common ones are distributed “shuffle” operations, such as grouping or aggregating the elements by a key. In Scala, these operations are automatically available on RDDs containing Tuple2 objects (the built-in tuples in the language, created by simply writing (a, b)). The key-value pair operations are available in the PairRDDFunctions class, which automatically wraps around an RDD of tuples. For example, the following code uses the reduceByKey operation on key-value pairs to count how many times each line of text occurs in a lines = sc.textFile(\"data.txt\") val pairs = lines.map(s => (s, 1)) val counts = pairs.reduceByKey((a, b) => a + b) We could also use counts.sortByKey(), for example, to sort the pairs alphabetically, and finally counts.collect() to bring them back to the driver program as an array of objects. using custom objects as the key in key-value pair operations, you must be sure that a custom equals() method is accompanied with a matching hashCode() method. For full details, see the contract outlined in the Object.hashCode() documentation. While most Spark operations work on RDDs containing any type of objects, a few special operations are only available on RDDs of key-value pairs. The most common ones are distributed “shuffle” operations, such as grouping or aggregating the elements by a key. In Java, key-value pairs are represented using the scala.Tuple2 class from the Scala standard library. You can simply call new Tuple2(a, b) to create a tuple, and access its fields later with tuple._1() and tuple._2(). RDDs of key-value pairs are represented by the JavaPairRDD class. You can construct JavaPairRDDs from JavaRDDs using special versions of the map operations, like mapToPair and flatMapToPair. The JavaPairRDD will have both standard RDD functions and special key-value ones. For example, the following code uses the reduceByKey operation on key-value pairs to count how many times each line of text occurs in a <String> lines = sc.textFile(\"data.txt\"); JavaPairRDD<String, Integer> pairs = lines.mapToPair(s -> new Tuple2(s, 1)); JavaPairRDD<String, Integer> counts = pairs.reduceByKey((a, b) -> a + b); We could also use counts.sortByKey(), for example, to sort the pairs alphabetically, and finally counts.collect() to bring them back to the driver program as an array of objects. using custom objects as the key in key-value pair operations, you must be sure that a custom equals() method is accompanied with a matching hashCode() method. For full details, see the contract outlined in the Object.hashCode() documentation. Transformations The following table lists some of the common transformations supported by Spark. Refer to the RDD API doc (Python, Scala, Java, R) and pair RDD functions doc (Scala, Java) for details. TransformationMeaning map(func) Return a new distributed dataset formed by passing each element of the source through a function func. filter(func) Return a new dataset formed by selecting those elements of the source on which func returns true. flatMap(func) Similar to map, but each input item can be mapped to 0 or more output items (so func should return a Seq rather than a single item). mapPartitions(func) Similar to map, but runs separately on each partition (block) of the RDD, so func must be of type Iterator<T> => Iterator<U> when running on an RDD of type T. mapPartitionsWithIndex(func) Similar to mapPartitions, but also provides func with an integer value representing the index of the partition, so func must be of type (Int, Iterator<T>) => Iterator<U> when running on an RDD of type T. sample(withReplacement, fraction, seed) Sample a fraction fraction of the data, with or without replacement, using a given random number generator seed. union(otherDataset) Return a new dataset that contains the union of the elements in the source dataset and the argument. intersection(otherDataset) Return a new RDD that contains the intersection of elements in the source dataset and the argument. distinct([numPartitions])) Return a new dataset that contains the distinct elements of the source dataset. groupByKey([numPartitions]) When called on a dataset of (K, V) pairs, returns a dataset of (K, Iterable<V>) pairs. you are grouping in order to perform an aggregation (such as a sum or average) over each key, using reduceByKey or aggregateByKey will yield much better performance. default, the level of parallelism in the output depends on the number of partitions of the parent RDD. You can pass an optional numPartitions argument to set a different number of tasks. reduceByKey(func, [numPartitions]) When called on a dataset of (K, V) pairs, returns a dataset of (K, V) pairs where the values for each key are aggregated using the given reduce function func, which must be of type (V,V) => V. Like in groupByKey, the number of reduce tasks is configurable through an optional second argument. aggregateByKey(zeroValue)(seqOp, combOp, [numPartitions]) When called on a dataset of (K, V) pairs, returns a dataset of (K, U) pairs where the values for each key are aggregated using the given combine functions and a neutral \"zero\" value. Allows an aggregated value type that is different than the input value type, while avoiding unnecessary allocations. Like in groupByKey, the number of reduce tasks is configurable through an optional second argument. sortByKey([ascending], [numPartitions]) When called on a dataset of (K, V) pairs where K implements Ordered, returns a dataset of (K, V) pairs sorted by keys in ascending or descending order, as specified in the boolean ascending argument. join(otherDataset, [numPartitions]) When called on datasets of type (K, V) and (K, W), returns a dataset of (K, (V, W)) pairs with all pairs of elements for each key. Outer joins are supported through leftOuterJoin, rightOuterJoin, and fullOuterJoin. cogroup(otherDataset, [numPartitions]) When called on datasets of type (K, V) and (K, W), returns a dataset of (K, (Iterable<V>, Iterable<W>)) tuples. This operation is also called groupWith. cartesian(otherDataset) When called on datasets of types T and U, returns a dataset of (T, U) pairs (all pairs of elements). pipe(command, [envVars]) Pipe each partition of the RDD through a shell command, e.g. a Perl or bash script. RDD elements are written to the process's stdin and lines output to its stdout are returned as an RDD of strings. coalesce(numPartitions) Decrease the number of partitions in the RDD to numPartitions. Useful for running operations more efficiently after filtering down a large dataset. repartition(numPartitions) Reshuffle the data in the RDD randomly to create either more or fewer partitions and balance it across them. This always shuffles all data over the network. repartitionAndSortWithinPartitions(partitioner) Repartition the RDD according to the given partitioner and, within each resulting partition, sort records by their keys. This is more efficient than calling repartition and then sorting within each partition because it can push the sorting down into the shuffle machinery. Actions The following table lists some of the common actions supported by Spark. Refer to the RDD API doc (Python, Scala, Java, R) and pair RDD functions doc (Scala, Java) for details. ActionMeaning reduce(func) Aggregate the elements of the dataset using a function func (which takes two arguments and returns one). The function should be commutative and associative so that it can be computed correctly in parallel. collect() Return all the elements of the dataset as an array at the driver program. This is usually useful after a filter or other operation that returns a sufficiently small subset of the data. count() Return the number of elements in the dataset. first() Return the first element of the dataset (similar to take(1)). take(n) Return an array with the first n elements of the dataset. takeSample(withReplacement, num, [seed]) Return an array with a random sample of num elements of the dataset, with or without replacement, optionally pre-specifying a random number generator seed. takeOrdered(n, [ordering]) Return the first n elements of the RDD using either their natural order or a custom comparator. saveAsTextFile(path) Write the elements of the dataset as a text file (or set of text files) in a given directory in the local filesystem, HDFS or any other Hadoop-supported file system. Spark will call toString on each element to convert it to a line of text in the file. saveAsSequenceFile(path) (Java and Scala) Write the elements of the dataset as a Hadoop SequenceFile in a given path in the local filesystem, HDFS or any other Hadoop-supported file system. This is available on RDDs of key-value pairs that implement Hadoop's Writable interface. In Scala, it is also available on types that are implicitly convertible to Writable (Spark includes conversions for basic types like Int, Double, String, etc). saveAsObjectFile(path) (Java and Scala) Write the elements of the dataset in a simple format using Java serialization, which can then be loaded using SparkContext.objectFile(). countByKey() Only available on RDDs of type (K, V). Returns a hashmap of (K, Int) pairs with the count of each key. foreach(func) Run a function func on each element of the dataset. This is usually done for side effects such as updating an Accumulator or interacting with external storage systems. variables other than Accumulators outside of the foreach() may result in undefined behavior. See Understanding closures for more details. The Spark RDD API also exposes asynchronous versions of some actions, like foreachAsync for foreach, which immediately return a FutureAction to the caller instead of blocking on completion of the action. This can be used to manage or wait for the asynchronous execution of the action. Shuffle operations Certain operations within Spark trigger an event known as the shuffle. The shuffle is Spark’s mechanism for re-distributing data so that it’s grouped differently across partitions. This typically involves copying data across executors and machines, making the shuffle a complex and costly operation. Background To understand what happens during the shuffle, we can consider the example of the reduceByKey operation. The reduceByKey operation generates a new RDD where all values for a single key are combined into a tuple - the key and the result of executing a reduce function against all values associated with that key. The challenge is that not all values for a single key necessarily reside on the same partition, or even the same machine, but they must be co-located to compute the result. In Spark, data is generally not distributed across partitions to be in the necessary place for a specific operation. During computations, a single task will operate on a single partition - thus, to organize all the data for a single reduceByKey reduce task to execute, Spark needs to perform an all-to-all operation. It must read from all partitions to find all the values for all keys, and then bring together values across partitions to compute the final result for each key - this is called the shuffle. Although the set of elements in each partition of newly shuffled data will be deterministic, and so is the ordering of partitions themselves, the ordering of these elements is not. If one desires predictably ordered data following shuffle then it’s possible to to sort each partition using, for example, ); broadcastVar.value(); // returns [1, 2, 3] After the broadcast variable is created, it should be used instead of the value v in any functions run on the cluster so that v is not shipped to the nodes more than once. In addition, the object v should not be modified after it is broadcast in order to ensure that all nodes get the same value of the broadcast variable (e.g. if the variable is shipped to a new node later). To release the resources that the broadcast variable copied onto executors, call // Then, create an Accumulator of this myVectorAcc = new VectorAccumulatorV2 // Then, register it into spark (myVectorAcc, \"MyVectorAcc1\") Note that, when programmers define their own type of AccumulatorV2, the resulting type can be different than that of the elements added. A numeric accumulator can be created by calling SparkContext.longAccumulator() or SparkContext.doubleAccumulator() to accumulate values of type Long or Double, respectively. Tasks running on a cluster can then add to it using the add method. However, they cannot read its value. Only the driver program can read the accumulator’s value, using its value method. The code below shows an accumulator being used to add up the elements of an accum = jsc.sc().longAccumulator(); sc.parallelize(Arrays.asList(1, 2, 3, 4)).foreach(x -> accum.add(x)); // ... // 10/09/29 :08 INFO finished in 0.317106 s accum.value(); // returns 10 While this code used the built-in support for accumulators of type Long, programmers can also create their own types by subclassing AccumulatorV2. The AccumulatorV2 abstract class has several methods which one has to for resetting the accumulator to zero, add for adding another value into the accumulator, merge for merging another same-type accumulator into this one. Other methods that must be overridden are contained in the API documentation. For example, supposing we had a MyVector class representing mathematical vectors, we could VectorAccumulatorV2 implements AccumulatorV2<MyVector, MyVector> { private MyVector myVector = MyVector.createZeroVector(); public void reset() { myVector.reset(); } public void add(MyVector v) { myVector.add(v); } ... } // Then, create an Accumulator of this myVectorAcc = new VectorAccumulatorV2(); // Then, register it into spark ().register(myVectorAcc, \"MyVectorAcc1\"); Note that, when programmers define their own type of AccumulatorV2, the resulting type can be different than that of the elements added. a Spark task finishes, Spark will try to merge the accumulated updates in this task to an accumulator. If it fails, Spark will ignore the failure and still mark the task successful and continue to run other tasks. Hence, a buggy accumulator will not impact a Spark job, but it may not get updated correctly although a Spark job is successful. For accumulator updates performed inside actions only, Spark guarantees that each task’s update to the accumulator will only be applied once, i.e. restarted tasks will not update the value. In transformations, users should be aware of that each task’s update may be applied more than once if tasks or job stages are re-executed. Accumulators do not change the lazy evaluation model of Spark. If they are being updated within an operation on an RDD, their value is only updated once that RDD is computed as part of an action. Consequently, accumulator updates are not guaranteed to be executed when made within a lazy transformation like map(). The below code fragment demonstrates this = sc.accumulator(0) def g(x): accum.add(x) return f(x) data.map(g) # Here, accum is still 0 because no actions have caused the `map` to be computed. val accum = sc.longAccumulator data.map { x => accum.add(x); x } // Here, accum is still 0 because no actions have caused the map operation to be computed. LongAccumulator accum = jsc.sc().longAccumulator(); data.map(x -> { accum.add(x); return f(x); }); // Here, accum is still 0 because no actions have caused the `map` to be computed. Deploying to a Cluster The application submission guide describes how to submit applications to a cluster. In short, once you package your application into a JAR (for Java/Scala) or a set of .py or .zip files (for Python), the bin/spark-submit script lets you submit it to any supported cluster manager. Launching Spark jobs from Java / Scala The org.apache.spark.launcher package provides classes for launching Spark jobs as child processes using a simple Java API. Unit Testing Spark is friendly to unit testing with any popular unit test framework. Simply create a SparkContext in your test with the master URL set to local, run your operations, and then call SparkContext.stop() to tear it down. Make sure you stop the context within a finally block or the test framework’s tearDown method, as Spark does not support two contexts running concurrently in the same program. Where to Go from Here You can see some example Spark programs on the Spark website. In addition, Spark includes several samples in the examples directory (Python, Scala, Java, R). You can run Java and Scala examples by passing the class name to Spark’s bin/run-example script; for /bin/run-example SparkPi For Python examples, use spark-submit /bin/spark-submit examples/src/main/python/pi.py For R examples, use spark-submit /bin/spark-submit examples/src/main/r/dataframe.R For help on optimizing your programs, the configuration and tuning guides provide information on best practices. They are especially important for making sure that your data is stored in memory in an efficient format. For help on deploying, the cluster mode overview describes the components involved in distributed operation and supported cluster managers. Finally, full API documentation is available in Python, Scala, Java and R.\n\nExample:\n```python\ninstall_requires=[\n        'pyspark==4.2.0'\n    ]\n```\n\nExample:\n```python\nfrom pyspark import SparkContext, SparkConf\n```\n\nExample:\n```bash\n$ PYSPARK_PYTHON=python3.8 bin/pyspark\n```\n\nExample:\n```text\ngroupId = org.apache.spark\nartifactId = spark-core_2.13\nversion = 4.2.0\n```\n\nExample:\n```text\ngroupId = org.apache.hadoop\nartifactId = hadoop-client\nversion = <your-hdfs-version>\n```\n\nExample:\n```scala\nimport org.apache.spark.SparkContext\nimport org.apache.spark.SparkConf\n```\n\nExample:\n```java\nimport org.apache.spark.api.java.JavaSparkContext;\nimport org.apache.spark.api.java.JavaRDD;\nimport org.apache.spark.SparkConf;\n```\n\nExample:\n```python\nconf = SparkConf().setAppName(appName).setMaster(master)\nsc = SparkContext(conf=conf)\n```\n\nExample:\n```scala\nval conf = new SparkConf().setAppName(appName).setMaster(master)\nnew SparkContext(conf)\n```\n\nExample:\n```java\nSparkConf conf = new SparkConf().setAppName(appName).setMaster(master);\nJavaSparkContext sc = new JavaSparkContext(conf);\n```\n\nExample:\n```bash\n$ ./bin/pyspark --master \"local[4]\"\n```\n\nExample:\n```bash\n$ ./bin/pyspark --master \"local[4]\" --py-files code.py\n```\n\nExample:\n```bash\n$ PYSPARK_DRIVER_PYTHON=ipython ./bin/pyspark\n```\n\nExample:\n```bash\n$ PYSPARK_DRIVER_PYTHON=jupyter PYSPARK_DRIVER_PYTHON_OPTS=notebook ./bin/pyspark\n```\n\nExample:\n```bash\n$ ./bin/spark-shell --master \"local[4]\"\n```\n\nExample:\n```bash\n$ ./bin/spark-shell --master \"local[4]\" --jars code.jar\n```\n\nExample:\n```bash\n$ ./bin/spark-shell --master \"local[4]\" --packages \"org.example:example:0.1\"\n```\n\nExample:\n```python\ndata = [1, 2, 3, 4, 5]\ndistData = sc.parallelize(data)\n```\n\nExample:\n```scala\nval data = Array(1, 2, 3, 4, 5)\nval distData = sc.parallelize(data)\n```\n\nExample:\n```java\nList<Integer> data = Arrays.asList(1, 2, 3, 4, 5);\nJavaRDD<Integer> distData = sc.parallelize(data);\n```\n\nExample:\n```python\n>>> distFile = sc.textFile(\"data.txt\")\n```\n\nExample:\n```python\n>>> rdd = sc.parallelize(range(1, 4)).map(lambda x: (x, \"a\" * x))\n>>> rdd.saveAsSequenceFile(\"path/to/file\")\n>>> sorted(sc.sequenceFile(\"path/to/file\").collect())\n[(1, u'a'), (2, u'aa'), (3, u'aaa')]\n```\n\nExample:\n```python\n$ ./bin/pyspark --jars /path/to/elasticsearch-hadoop.jar\n>>> conf = {\"es.resource\" : \"index/type\"}  # assume Elasticsearch is running on localhost defaults\n>>> rdd = sc.newAPIHadoopRDD(\"org.elasticsearch.hadoop.mr.EsInputFormat\",\n                             \"org.apache.hadoop.io.NullWritable\",\n                             \"org.elasticsearch.hadoop.mr.LinkedMapWritable\",\n                             conf=conf)\n>>> rdd.first()  # the result is a MapWritable that is converted to a Python dict\n(u'Elasticsearch ID',\n {u'field1': True,\n  u'field2': u'Some Text',\n  u'field3': 12345})\n```\n\nExample:\n```scala\nscala> val distFile = sc.textFile(\"data.txt\")\ndistFile: org.apache.spark.rdd.RDD[String] = data.txt MapPartitionsRDD[10] at textFile at <console>:26\n```\n\nExample:\n```java\nJavaRDD<String> distFile = sc.textFile(\"data.txt\");\n```\n\nExample:\n```python\nlines = sc.textFile(\"data.txt\")\nlineLengths = lines.map(lambda s: len(s))\ntotalLength = lineLengths.reduce(lambda a, b: a + b)\n```\n\nExample:\n```python\nlineLengths.persist()\n```\n\nExample:\n```scala\nval lines = sc.textFile(\"data.txt\")\nval lineLengths = lines.map(s => s.length)\nval totalLength = lineLengths.reduce((a, b) => a + b)\n```\n\nExample:\n```java\nJavaRDD<String> lines = sc.textFile(\"data.txt\");\nJavaRDD<Integer> lineLengths = lines.map(s -> s.length());\nint totalLength = lineLengths.reduce((a, b) -> a + b);\n```\n\nExample:\n```java\nlineLengths.persist(StorageLevel.MEMORY_ONLY());\n```\n\nExample:\n```python\n\"\"\"MyScript.py\"\"\"\nif __name__ == \"__main__\":\n    def myFunc(s):\n        words = s.split(\" \")\n        return len(words)\n\n    sc = SparkContext(...)\n    sc.textFile(\"file.txt\").map(myFunc)\n```\n\nExample:\n```python\nclass MyClass(object):\n    def func(self, s):\n        return s\n    def doStuff(self, rdd):\n        return rdd.map(self.func)\n```\n\nExample:\n```python\nclass MyClass(object):\n    def __init__(self):\n        self.field = \"Hello\"\n    def doStuff(self, rdd):\n        return rdd.map(lambda s: self.field + s)\n```\n\nExample:\n```python\ndef doStuff(self, rdd):\n    field = self.field\n    return rdd.map(lambda s: field + s)\n```\n\nExample:\n```scala\nobject MyFunctions {\n  def func1(s: String): String = { ... }\n}\n\nmyRdd.map(MyFunctions.func1)\n```\n\nExample:\n```scala\nclass MyClass {\n  def func1(s: String): String = { ... }\n  def doStuff(rdd: RDD[String]): RDD[String] = { rdd.map(func1) }\n}\n```\n\nExample:\n```scala\nclass MyClass {\n  val field = \"Hello\"\n  def doStuff(rdd: RDD[String]): RDD[String] = { rdd.map(x => field + x) }\n}\n```\n\nExample:\n```scala\ndef doStuff(rdd: RDD[String]): RDD[String] = {\n  val field_ = this.field\n  rdd.map(x => field_ + x)\n}\n```\n\nExample:\n```java\nJavaRDD<String> lines = sc.textFile(\"data.txt\");\nJavaRDD<Integer> lineLengths = lines.map(new Function<String, Integer>() {\n  public Integer call(String s) { return s.length(); }\n});\nint totalLength = lineLengths.reduce(new Function2<Integer, Integer, Integer>() {\n  public Integer call(Integer a, Integer b) { return a + b; }\n});\n```\n\nExample:\n```java\nclass GetLength implements Function<String, Integer> {\n  public Integer call(String s) { return s.length(); }\n}\nclass Sum implements Function2<Integer, Integer, Integer> {\n  public Integer call(Integer a, Integer b) { return a + b; }\n}\n\nJavaRDD<String> lines = sc.textFile(\"data.txt\");\nJavaRDD<Integer> lineLengths = lines.map(new GetLength());\nint totalLength = lineLengths.reduce(new Sum());\n```\n\nExample:\n```python\ncounter = 0\nrdd = sc.parallelize(data)\n\n# Wrong: Don't do this!!\ndef increment_counter(x):\n    global counter\n    counter += x\nrdd.foreach(increment_counter)\n\nprint(\"Counter value: \", counter)\n```\n\nExample:\n```scala\nvar counter = 0\nvar rdd = sc.parallelize(data)\n\n// Wrong: Don't do this!!\nrdd.foreach(x => counter += x)\n\nprintln(\"Counter value: \" + counter)\n```\n\nExample:\n```java\nint counter = 0;\nJavaRDD<Integer> rdd = sc.parallelize(data);\n\n// Wrong: Don't do this!!\nrdd.foreach(x -> counter += x);\n\nprintln(\"Counter value: \" + counter);\n```\n\nExample:\n```python\nlines = sc.textFile(\"data.txt\")\npairs = lines.map(lambda s: (s, 1))\ncounts = pairs.reduceByKey(lambda a, b: a + b)\n```\n\nExample:\n```scala\nval lines = sc.textFile(\"data.txt\")\nval pairs = lines.map(s => (s, 1))\nval counts = pairs.reduceByKey((a, b) => a + b)\n```\n\nExample:\n```scala\nJavaRDD<String> lines = sc.textFile(\"data.txt\");\nJavaPairRDD<String, Integer> pairs = lines.mapToPair(s -> new Tuple2(s, 1));\nJavaPairRDD<String, Integer> counts = pairs.reduceByKey((a, b) -> a + b);\n```\n\nExample:\n```python\n>>> broadcastVar = sc.broadcast([1, 2, 3])\n<pyspark.core.broadcast.Broadcast object at 0x102789f10>\n\n>>> broadcastVar.value\n[1, 2, 3]\n```\n\nExample:\n```scala\nscala> val broadcastVar = sc.broadcast(Array(1, 2, 3))\nbroadcastVar: org.apache.spark.broadcast.Broadcast[Array[Int]] = Broadcast(0)\n\nscala> broadcastVar.value\nres0: Array[Int] = Array(1, 2, 3)\n```\n\nExample:\n```java\nBroadcast<int[]> broadcastVar = sc.broadcast(new int[] {1, 2, 3});\n\nbroadcastVar.value();\n// returns [1, 2, 3]\n```\n\nExample:\n```python\n>>> accum = sc.accumulator(0)\n>>> accum\nAccumulator<id=0, value=0>\n\n>>> sc.parallelize([1, 2, 3, 4]).foreach(lambda x: accum.add(x))\n...\n10/09/29 18:41:08 INFO SparkContext: Tasks finished in 0.317106 s\n\n>>> accum.value\n10\n```\n\nExample:\n```python\nclass VectorAccumulatorParam(AccumulatorParam):\n    def zero(self, initialValue):\n        return Vector.zeros(initialValue.size)\n\n    def addInPlace(self, v1, v2):\n        v1 += v2\n        return v1\n\n# Then, create an Accumulator of this type:\nvecAccum = sc.accumulator(Vector(...), VectorAccumulatorParam())\n```\n\nExample:\n```scala\nscala> val accum = sc.longAccumulator(\"My Accumulator\")\naccum: org.apache.spark.util.LongAccumulator = LongAccumulator(id: 0, name: Some(My Accumulator), value: 0)\n\nscala> sc.parallelize(Array(1, 2, 3, 4)).foreach(x => accum.add(x))\n...\n10/09/29 18:41:08 INFO SparkContext: Tasks finished in 0.317106 s\n\nscala> accum.value\nres2: Long = 10\n```\n\nExample:\n```scala\nclass VectorAccumulatorV2 extends AccumulatorV2[MyVector, MyVector] {\n\n  private val myVector: MyVector = MyVector.createZeroVector\n\n  def reset(): Unit = {\n    myVector.reset()\n  }\n\n  def add(v: MyVector): Unit = {\n    myVector.add(v)\n  }\n  ...\n}\n\n// Then, create an Accumulator of this type:\nval myVectorAcc = new VectorAccumulatorV2\n// Then, register it into spark context:\nsc.register(myVectorAcc, \"MyVectorAcc1\")\n```\n\nExample:\n```java\nLongAccumulator accum = jsc.sc().longAccumulator();\n\nsc.parallelize(Arrays.asList(1, 2, 3, 4)).foreach(x -> accum.add(x));\n// ...\n// 10/09/29 18:41:08 INFO SparkContext: Tasks finished in 0.317106 s\n\naccum.value();\n// returns 10\n```\n\nExample:\n```java\nclass VectorAccumulatorV2 implements AccumulatorV2<MyVector, MyVector> {\n\n  private MyVector myVector = MyVector.createZeroVector();\n\n  public void reset() {\n    myVector.reset();\n  }\n\n  public void add(MyVector v) {\n    myVector.add(v);\n  }\n  ...\n}\n\n// Then, create an Accumulator of this type:\nVectorAccumulatorV2 myVectorAcc = new VectorAccumulatorV2();\n// Then, register it into spark context:\njsc.sc().register(myVectorAcc, \"MyVectorAcc1\");\n```\n\nExample:\n```python\naccum = sc.accumulator(0)\ndef g(x):\n    accum.add(x)\n    return f(x)\ndata.map(g)\n# Here, accum is still 0 because no actions have caused the `map` to be computed.\n```\n\nExample:\n```scala\nval accum = sc.longAccumulator\ndata.map { x => accum.add(x); x }\n// Here, accum is still 0 because no actions have caused the map operation to be computed.\n```\n\nExample:\n```java\nLongAccumulator accum = jsc.sc().longAccumulator();\ndata.map(x -> { accum.add(x); return f(x); });\n// Here, accum is still 0 because no actions have caused the `map` to be computed.\n```\n\nExample:\n```text\n./bin/run-example SparkPi\n```\n\nExample:\n```text\n./bin/spark-submit examples/src/main/python/pi.py\n```\n\nExample:\n```text\n./bin/spark-submit examples/src/main/r/dataframe.R\n```\n\n<|endofdoc|>","metadata":{"transformedAt":"2026-08-18T15:21:43.044Z","totalSectionsIncluded":1,"totalCodeBlocksIncluded":61,"totalLines":502,"estimatedTokens":13732}}20{"id":"doc-tuning_spark_4_2_0_documentation-ae7f780c","source":"documentation","title":"Tuning - Spark 4.2.0 Documentation","url":"https://spark.apache.org/docs/latest/tuning.html","text":"Tuning Spark Data Serialization Memory Tuning Memory Management Overview Determining Memory Consumption Tuning Data Structures Serialized RDD Storage Garbage Collection Tuning Other Considerations Level of Parallelism Parallel Listing on Input Paths Memory Usage of Reduce Tasks Broadcasting Large Variables Data Locality Summary Because of the in-memory nature of most Spark computations, Spark programs can be bottlenecked by any resource in the , network bandwidth, or memory. Most often, if the data fits in memory, the bottleneck is network bandwidth, but sometimes, you also need to do some tuning, such as storing RDDs in serialized form, to decrease memory usage. This guide will cover two main serialization, which is crucial for good network performance and can also reduce memory use, and memory tuning. We also sketch several smaller topics. Data Serialization Serialization plays an important role in the performance of any distributed application. Formats that are slow to serialize objects into, or consume a large number of bytes, will greatly slow down the computation. Often, this will be the first thing you should tune to optimize a Spark application. Spark aims to strike a balance between convenience (allowing you to work with any Java type in your operations) and performance. It provides two serialization default, Spark serializes objects using Java’s ObjectOutputStream framework, and can work with any class you create that implements java.io.Serializable. You can also control the performance of your serialization more closely by extending java.io.Externalizable. Java serialization is flexible but often quite slow, and leads to large serialized formats for many classes. Kryo can also use the Kryo library (version 4) to serialize objects more quickly. Kryo is significantly faster and more compact than Java serialization (often as much as 10x), but does not support all Serializable types and requires you to register the classes you’ll use in the program in advance for best performance. You can switch to using Kryo by initializing your job with a SparkConf and calling conf.set(\"spark.serializer\", \"org.apache.spark.serializer.KryoSerializer\"). This setting configures the serializer used for not only shuffling data between worker nodes but also when serializing RDDs to disk. The only reason Kryo is not the default is because of the custom registration requirement, but we recommend trying it in any network-intensive application. Since Spark 2.0.0, we internally use Kryo serializer when shuffling RDDs with simple types, arrays of simple types, or string type. Spark automatically includes Kryo serializers for the many commonly-used core Scala classes covered in the AllScalaRegistrar from the Twitter chill library. To register your own custom classes with Kryo, use the registerKryoClasses method. val conf = new SparkConf().setMaster(...).setAppName(...) conf.registerKryoClasses(Array(classOf[MyClass1], classOf[MyClass2])) val sc = new SparkContext(conf) The Kryo documentation describes more advanced registration options, such as adding custom serialization code. If your objects are large, you may also need to increase the spark.kryoserializer.buffer config. This value needs to be large enough to hold the largest object you will serialize. Finally, if you don’t register your custom classes, Kryo will still work, but it will have to store the full class name with each object, which is wasteful. Memory Tuning There are three considerations in tuning memory amount of memory used by your objects (you may want your entire dataset to fit in memory), the cost of accessing those objects, and the overhead of garbage collection (if you have high turnover in terms of objects). By default, Java objects are fast to access, but can easily consume a factor of 2-5x more space than the “raw” data inside their fields. This is due to several distinct Java object has an “object header”, which is about 16 bytes and contains information such as a pointer to its class. For an object with very little data in it (say one Int field), this can be bigger than the data. Java Strings have about 40 bytes of overhead over the raw string data (since they store it in an array of Chars and keep extra data such as the length), and store each character as two bytes due to String’s internal usage of UTF-16 encoding. Thus a 10-character string can easily consume 60 bytes. Common collection classes, such as HashMap and LinkedList, use linked data structures, where there is a “wrapper” object for each entry (e.g. Map.Entry). This object not only has a header, but also pointers (typically 8 bytes each) to the next object in the list. Collections of primitive types often store them as “boxed” objects such as java.lang.Integer. This section will start with an overview of memory management in Spark, then discuss specific strategies the user can take to make more efficient use of memory in his/her application. In particular, we will describe how to determine the memory usage of your objects, and how to improve it – either by changing your data structures, or by storing data in a serialized format. We will then cover tuning Spark’s cache size and the Java garbage collector. Memory Management Overview Memory usage in Spark largely falls under one of two and storage. Execution memory refers to that used for computation in shuffles, joins, sorts and aggregations, while storage memory refers to that used for caching and propagating internal data across the cluster. In Spark, execution and storage share a unified region (M). When no execution memory is used, storage can acquire all the available memory and vice versa. Execution may evict storage if necessary, but only until total storage memory usage falls under a certain threshold (R). In other words, R describes a subregion within M where cached blocks are never evicted. Storage may not evict execution due to complexities in implementation. This design ensures several desirable properties. First, applications that do not use caching can use the entire space for execution, obviating unnecessary disk spills. Second, applications that do use caching can reserve a minimum storage space (R) where their data blocks are immune to being evicted. Lastly, this approach provides reasonable out-of-the-box performance for a variety of workloads without requiring user expertise of how memory is divided internally. Although there are two relevant configurations, the typical user should not need to adjust them as the default values are applicable to most expresses the size of M as a fraction of the (JVM heap space - 300MiB) (default 0.6). The rest of the space (40%) is reserved for user data structures, internal metadata in Spark, and safeguarding against OOM errors in the case of sparse and unusually large records. spark.memory.storageFraction expresses the size of R as a fraction of M (default 0.5). R is the storage space within M where cached blocks immune to being evicted by execution. The value of spark.memory.fraction should be set in order to fit this amount of heap space comfortably within the JVM’s old or “tenured” generation. See the discussion of advanced GC tuning below for details. Determining Memory Consumption The best way to size the amount of memory consumption a dataset will require is to create an RDD, put it into cache, and look at the “Storage” page in the web UI. The page will tell you how much memory the RDD is occupying. To estimate the memory consumption of a particular object, use SizeEstimator’s estimate method. This is useful for experimenting with different data layouts to trim memory usage, as well as determining the amount of space a broadcast variable will occupy on each executor heap. Tuning Data Structures The first way to reduce memory consumption is to avoid the Java features that add overhead, such as pointer-based data structures and wrapper objects. There are several ways to do your data structures to prefer arrays of objects, and primitive types, instead of the standard Java or Scala collection classes (e.g. HashMap). The fastutil library provides convenient collection classes for primitive types that are compatible with the Java standard library. Avoid nested structures with a lot of small objects and pointers when possible. Consider using numeric IDs or enumeration objects instead of strings for keys. If you have less than 32 GiB of RAM, set the JVM flag -XX:+UseCompressedOops to make pointers be four bytes instead of eight. You can add these options in spark-env.sh. Serialized RDD Storage When your objects are still too large to efficiently store despite this tuning, a much simpler way to reduce memory usage is to store them in serialized form, using the serialized StorageLevels in the RDD persistence API, such as MEMORY_ONLY_SER. Spark will then store each RDD partition as one large byte array. The only downside of storing data in serialized form is slower access times, due to having to deserialize each object on the fly. We highly recommend using Kryo if you want to cache data in serialized form, as it leads to much smaller sizes than Java serialization (and certainly than raw Java objects). Garbage Collection Tuning JVM garbage collection can be a problem when you have large “churn” in terms of the RDDs stored by your program. (It is usually not a problem in programs that just read an RDD once and then run many operations on it.) When Java needs to evict old objects to make room for new ones, it will need to trace through all your Java objects and find the unused ones. The main point to remember here is that the cost of garbage collection is proportional to the number of Java objects, so using data structures with fewer objects (e.g. an array of Ints instead of a LinkedList) greatly lowers this cost. An even better method is to persist objects in serialized form, as described there will be only one object (a byte array) per RDD partition. Before trying other techniques, the first thing to try if GC is a problem is to use serialized caching. GC can also be a problem due to interference between your tasks’ working memory (the amount of space needed to run the task) and the RDDs cached on your nodes. We will discuss how to control the space allocated to the RDD cache to mitigate this. Measuring the Impact of GC The first step in GC tuning is to collect statistics on how frequently garbage collection occurs and the amount of time spent GC. This can be done by adding -XX:+PrintGCDetails -XX:+PrintGCTimeStamps to the Java options. (See the configuration guide for info on passing Java options to Spark jobs.) Next time your Spark job is run, you will see messages printed in the worker’s logs each time a garbage collection occurs. Note these logs will be on your cluster’s worker nodes (in the stdout files in their work directories), not on your driver program. Advanced GC Tuning To further tune garbage collection, we first need to understand some basic information about memory management in the Heap space is divided into two regions Young and Old. The Young generation is meant to hold short-lived objects while the Old generation is intended for objects with longer lifetimes. The Young generation is further divided into three regions [Eden, Survivor1, Survivor2]. A simplified description of the garbage collection Eden is full, a minor GC is run on Eden and objects that are alive from Eden and Survivor1 are copied to Survivor2. The Survivor regions are swapped. If an object is old enough or Survivor2 is full, it is moved to Old. Finally, when Old is close to full, a full GC is invoked. The goal of GC tuning in Spark is to ensure that only long-lived RDDs are stored in the Old generation and that the Young generation is sufficiently sized to store short-lived objects. This will help avoid full GCs to collect temporary objects created during task execution. Some steps which may be useful if there are too many garbage collections by collecting GC stats. If a full GC is invoked multiple times before a task completes, it means that there isn’t enough memory available for executing tasks. If there are too many minor collections but not many major GCs, allocating more memory for Eden would help. You can set the size of the Eden to be an over-estimate of how much memory each task will need. If the size of Eden is determined to be E, then you can set the size of the Young generation using the option -Xmn=4/3*E. (The scaling up by 4/3 is to account for space used by survivor regions as well.) In the GC stats that are printed, if the OldGen is close to being full, reduce the amount of memory used for caching by lowering spark.memory.fraction; it is better to cache fewer objects than to slow down task execution. Alternatively, consider decreasing the size of the Young generation. This means lowering -Xmn if you’ve set it as above. If not, try changing the value of the JVM’s NewRatio parameter. Many JVMs default this to 2, meaning that the Old generation occupies 2/3 of the heap. It should be large enough such that this fraction exceeds spark.memory.fraction. Since 4.0.0, Spark uses JDK 17 by default, which also makes the G1GC garbage collector the default. Note that with large executor heap sizes, it may be important to increase the G1 region size with As an example, if your task is reading data from HDFS, the amount of memory used by the task can be estimated using the size of the data block read from HDFS. Note that the size of a decompressed block is often 2 or 3 times the size of the block. So if we wish to have 3 or 4 tasks’ worth of working space, and the HDFS block size is 128 MiB, we can estimate the size of Eden to be 4*3*128MiB. Monitor how the frequency and time taken by garbage collection changes with the new settings. Our experience suggests that the effect of GC tuning depends on your application and the amount of memory available. There are many more tuning options described online, but at a high level, managing how frequently full GC takes place can help in reducing the overhead. GC tuning flags for executors can be specified by setting spark.executor.defaultJavaOptions or spark.executor.extraJavaOptions in a job’s configuration. Other Considerations Level of Parallelism Clusters will not be fully utilized unless you set the level of parallelism for each operation high enough. Spark automatically sets the number of “map” tasks to run on each file according to its size (though you can control it through optional parameters to SparkContext.textFile, etc), and for distributed “reduce” operations, such as groupByKey and reduceByKey, it uses the largest parent RDD’s number of partitions. You can pass the level of parallelism as a second argument (see the spark.PairRDDFunctions documentation), or set the config property spark.default.parallelism to change the default. In general, we recommend 2-3 tasks per CPU core in your cluster. Parallel Listing on Input Paths Sometimes you may also need to increase directory listing parallelism when job input has large number of directories, otherwise the process could take a very long time, especially when against object store like S3. If your job works on RDD with Hadoop input formats (e.g., via SparkContext.sequenceFile), the parallelism is controlled via spark.hadoop.mapreduce.input.fileinputformat.list-status.num-threads (currently default is 1). For Spark SQL with file-based data sources, you can tune spark.sql.sources.parallelPartitionDiscovery.threshold and spark.sql.sources.parallelPartitionDiscovery.parallelism to improve listing parallelism. Please refer to Spark SQL performance tuning guide for more details. Memory Usage of Reduce Tasks Sometimes, you will get an OutOfMemoryError not because your RDDs don’t fit in memory, but because the working set of one of your tasks, such as one of the reduce tasks in groupByKey, was too large. Spark’s shuffle operations (sortByKey, groupByKey, reduceByKey, join, etc) build a hash table within each task to perform the grouping, which can often be large. The simplest fix here is to increase the level of parallelism, so that each task’s input set is smaller. Spark can efficiently support tasks as short as 200 ms, because it reuses one executor JVM across many tasks and it has a low task launching cost, so you can safely increase the level of parallelism to more than the number of cores in your clusters. Broadcasting Large Variables Using the broadcast functionality available in SparkContext can greatly reduce the size of each serialized task, and the cost of launching a job over a cluster. If your tasks use any large object from the driver program inside of them (e.g. a static lookup table), consider turning it into a broadcast variable. Spark prints the serialized size of each task on the master, so you can look at that to decide whether your tasks are too large; in general, tasks larger than about 20 KiB are probably worth optimizing. Data Locality Data locality can have a major impact on the performance of Spark jobs. If data and the code that operates on it are together, then computation tends to be fast. But if code and data are separated, one must move to the other. Typically, it is faster to ship serialized code from place to place than a chunk of data because code size is much smaller than data. Spark builds its scheduling around this general principle of data locality. Data locality is how close data is to the code processing it. There are several levels of locality based on the data’s current location. In order from closest to data is in the same JVM as the running code. This is the best locality possible. NODE_LOCAL data is on the same node. Examples might be in HDFS on the same node, or in another executor on the same node. This is a little slower than PROCESS_LOCAL because the data has to travel between processes. NO_PREF data is accessed equally quickly from anywhere and has no locality preference. RACK_LOCAL data is on the same rack of servers. Data is on a different server on the same rack so needs to be sent over the network, typically through a single switch. ANY data is elsewhere on the network and not in the same rack. Spark prefers to schedule all tasks at the best locality level, but this is not always possible. In situations where there is no unprocessed data on any idle executor, Spark switches to lower locality levels. There are two ) wait until a busy CPU frees up to start a task on data on the same server, or b) immediately start a new task in a farther away place that requires moving data there. What Spark typically does is wait a bit in the hopes that a busy CPU frees up. Once that timeout expires, it starts moving the data from far away to the free CPU. The wait timeout for fallback between each level can be configured individually or all together in one parameter; see the spark.locality parameters on the configuration page for details. You should increase these settings if your tasks are long and see poor locality, but the default usually works well. Summary This has been a short guide to point out the main concerns you should know about when tuning a Spark application – most importantly, data serialization and memory tuning. For most programs, switching to Kryo serialization and persisting data in serialized form will solve most common performance issues. Feel free to ask on the Spark mailing list about other tuning best practices.\n\nExample:\n```scala\nval conf = new SparkConf().setMaster(...).setAppName(...)\nconf.registerKryoClasses(Array(classOf[MyClass1], classOf[MyClass2]))\nval sc = new SparkContext(conf)\n```\n\n<|endofdoc|>","metadata":{"transformedAt":"2026-08-18T15:21:43.047Z","totalSectionsIncluded":1,"totalCodeBlocksIncluded":1,"totalLines":10,"estimatedTokens":4908}}21{"id":"doc-spark_standalone_mode_spark_4_2_0_documentation-43844a57","source":"documentation","title":"Spark Standalone Mode - Spark 4.2.0 Documentation","url":"https://spark.apache.org/docs/latest/spark-standalone.html","text":"Spark Standalone Mode Security Installing Spark Standalone to a Cluster Starting a Cluster Manually Cluster Launch Scripts Resource Allocation and Configuration Overview Connecting an Application to the Cluster Client Properties Launching Spark Applications Spark Protocol REST API Resource Scheduling Executors Scheduling Stage Level Scheduling Overview Caveats Monitoring and Logging Running Alongside Hadoop Configuring Ports for Network Security High Availability Standby Masters with ZooKeeper Single-Node Recovery with Local File System In addition to running on the YARN cluster manager, Spark also provides a simple standalone deploy mode. You can launch a standalone cluster either manually, by starting a master and workers by hand, or use our provided launch scripts. It is also possible to run these daemons on a single machine for testing. Security Security features like authentication are not enabled by default. When deploying a cluster that is open to the internet or an untrusted network, it’s important to secure access to the cluster to prevent unauthorized applications from running on the cluster. Please see Spark Security and the specific security sections in this doc before running Spark. Installing Spark Standalone to a Cluster To install Spark Standalone mode, you simply place a compiled version of Spark on each node on the cluster. You can obtain pre-built versions of Spark with each release or build it yourself. Starting a Cluster Manually You can start a standalone master server by /sbin/start-master.sh Once started, the master will print out a spark://HOST:PORT URL for itself, which you can use to connect workers to it, or pass as the “master” argument to SparkContext. You can also find this URL on the master’s web UI, which is http://localhost:8080 by default. Similarly, you can start one or more workers and connect them to the master /sbin/start-worker.sh <master-spark-URL> Once you have started a worker, look at the master’s web UI (http://localhost:8080 by default). You should see the new node listed there, along with its number of CPUs and memory (minus one gigabyte left for the OS). Finally, the following configuration options can be passed to the master and -h HOST, --host HOST Hostname to listen on -p PORT, --port PORT Port for service to listen on (default: 7077 for master, random for worker) --webui-port PORT Port for web UI (default: 8080 for master, 8081 for worker) -c CORES, --cores CORES Total CPU cores to allow Spark applications to use on the machine (default: all available); only on worker -m MEM, --memory MEM Total amount of memory to allow Spark applications to use on the machine, in a format like 1000M or 2G (default: your machine's total RAM minus 1 GiB); only on worker -d DIR, --work-dir DIR Directory to use for scratch space and job output logs (default: SPARK_HOME/work); only on worker --properties-file FILE Path to a custom Spark properties file to load (default: conf/spark-defaults.conf) Cluster Launch Scripts To launch a Spark standalone cluster with the launch scripts, you should create a file called conf/workers in your Spark directory, which must contain the hostnames of all the machines where you intend to start Spark workers, one per line. If conf/workers does not exist, the launch scripts defaults to a single machine (localhost), which is useful for testing. Note, the master machine accesses each of the worker machines via ssh. By default, ssh is run in parallel and requires password-less (using a private key) access to be setup. If you do not have a password-less setup, you can set the environment variable SPARK_SSH_FOREGROUND and serially provide a password for each worker. Once you’ve set up this file, you can launch or stop your cluster with the following shell scripts, based on Hadoop’s deploy scripts, and available in SPARK_HOME/sbin: sbin/start-master.sh - Starts a master instance on the machine the script is executed on. sbin/start-workers.sh - Starts a worker instance on each machine specified in the conf/workers file. sbin/start-worker.sh - Starts a worker instance on the machine the script is executed on. sbin/start-connect-server.sh - Starts a Spark Connect server on the machine the script is executed on. sbin/start-all.sh - Starts both a master and a number of workers as described above. sbin/stop-master.sh - Stops the master that was started via the sbin/start-master.sh script. sbin/stop-worker.sh - Stops all worker instances on the machine the script is executed on. sbin/stop-workers.sh - Stops all worker instances on the machines specified in the conf/workers file. sbin/stop-connect-server.sh - Stops all Spark Connect server instances on the machine the script is executed on. sbin/stop-all.sh - Stops both the master and the workers as described above. Note that these scripts must be executed on the machine you want to run the Spark master on, not your local machine. You can optionally configure the cluster further by setting environment variables in conf/spark-env.sh. Create this file by starting with the conf/spark-env.sh.template, and copy it to all your worker machines for the settings to take effect. The following settings are VariableMeaning SPARK_MASTER_HOST Bind the master to a specific hostname or IP address, for example a public one. SPARK_MASTER_PORT Start the master on a different port (default: 7077). SPARK_MASTER_WEBUI_PORT Port for the master web UI (default: 8080). SPARK_MASTER_OPTS Configuration properties that apply only to the master in the form \"-Dx=y\" (default: none). See below for a list of possible options. SPARK_LOCAL_DIRS Directory to use for \"scratch\" space in Spark, including map output files and RDDs that get stored on disk. This should be on a fast, local disk in your system. It can also be a comma-separated list of multiple directories on different disks. SPARK_LOG_DIR Where log files are stored. (default: SPARK_HOME/logs). SPARK_LOG_MAX_FILES The maximum number of log files (default: 5). SPARK_PID_DIR Where pid files are stored. (default: /tmp). SPARK_WORKER_CORES Total number of cores to allow Spark applications to use on the machine (default: all available cores). SPARK_WORKER_MEMORY Total amount of memory to allow Spark applications to use on the machine, e.g. 1000m, 2g (default: total memory minus 1 GiB); note that each application's individual memory is configured using its spark.executor.memory property. SPARK_WORKER_PORT Start the Spark worker on a specific port (default: random). SPARK_WORKER_WEBUI_PORT Port for the worker web UI (default: 8081). SPARK_WORKER_DIR Directory to run applications in, which will include both logs and scratch space (default: SPARK_HOME/work). SPARK_WORKER_OPTS Configuration properties that apply only to the worker in the form \"-Dx=y\" (default: none). See below for a list of possible options. SPARK_DAEMON_MEMORY Memory to allocate to the Spark master and worker daemons themselves (default: 1g). SPARK_DAEMON_JAVA_OPTS JVM options for the Spark master and worker daemons themselves in the form \"-Dx=y\" (default: none). SPARK_DAEMON_CLASSPATH Classpath for the Spark master and worker daemons themselves (default: none). SPARK_PUBLIC_DNS The public DNS name of the Spark master and workers (default: none). launch scripts do not currently support Windows. To run a Spark cluster on Windows, start the master and workers by hand. SPARK_MASTER_OPTS supports the following system NameDefaultMeaningSince Version spark.master.ui.port 8080 Specifies the port number of the Master Web UI endpoint. 1.1.0 spark.master.ui.title (None) Specifies the title of the Master UI page. If unset, Spark Master at 'master url' is used by default. 4.0.0 spark.master.ui.decommission.allow.mode LOCAL Specifies the behavior of the Master Web UI's /workers/kill endpoint. Possible choices means allow this endpoint from IP's that are local to the machine running the Master, DENY means to completely disable this endpoint, ALLOW means to allow calling this endpoint from any IP. 3.1.0 spark.master.ui.historyServerUrl (None) The URL where Spark history server is running. Please note that this assumes that all Spark jobs share the same event log location where the history server accesses. 4.0.0 spark.master.rest.enabled true Whether to use the Master REST API endpoint or not. 1.3.0 spark.master.rest.host (None) Specifies the host of the Master REST API endpoint. 4.0.0 spark.master.rest.port 6066 Specifies the port number of the Master REST API endpoint. 1.3.0 spark.master.rest.filters (None) Comma separated list of filter class names to apply to the Master REST API. 4.0.0 spark.master.useAppNameAsAppId.enabled false (Experimental) If true, Spark master uses the user-provided appName for appId. 4.0.0 spark.deploy.retainedApplications 200 The maximum number of completed applications to display. Older applications will be dropped from the UI to maintain this limit. 0.8.0 spark.deploy.retainedDrivers 200 The maximum number of completed drivers to display. Older drivers will be dropped from the UI to maintain this limit. 1.1.0 spark.deploy.spreadOutDrivers true Whether the standalone cluster manager should spread drivers out across nodes or try to consolidate them onto as few nodes as possible. Spreading out is usually better for data locality in HDFS, but consolidating is more efficient for compute-intensive workloads. 4.0.0 spark.deploy.spreadOutApps true Whether the standalone cluster manager should spread applications out across nodes or try to consolidate them onto as few nodes as possible. Spreading out is usually better for data locality in HDFS, but consolidating is more efficient for compute-intensive workloads. 0.6.1 spark.deploy.defaultCores Int.MaxValue Default number of cores to give to applications in Spark's standalone mode if they don't set spark.cores.max. If not set, applications always get all available cores unless they configure spark.cores.max themselves. Set this lower on a shared cluster to prevent users from grabbing the whole cluster by default. 0.9.0 spark.deploy.maxExecutorRetries 10 Limit on the maximum number of back-to-back executor failures that can occur before the standalone cluster manager removes a faulty application. An application will never be removed if it has any running executors. If an application experiences more than spark.deploy.maxExecutorRetries failures in a row, no executors successfully start running in between those failures, and the application has no running executors then the standalone cluster manager will remove the application and mark it as failed. To disable this automatic removal, set spark.deploy.maxExecutorRetries to -1. 1.6.3 spark.deploy.maxDrivers Int.MaxValue The maximum number of running drivers. 4.0.0 spark.deploy.appNumberModulo (None) The modulo for app number. By default, the next of app-yyyyMMddHHmmss-9999 is app-yyyyMMddHHmmss-10000. If we have 10000 as modulo, it will be app-yyyyMMddHHmmss-0000. In most cases, the prefix app-yyyyMMddHHmmss is increased already during creating 10000 applications. 4.0.0 spark.deploy.driverIdPattern driver-%s-%04d The pattern for driver ID generation based on Java String.format method. The default value is driver-%s-%04d which represents the existing driver id string, e.g., driver-20231031224459-0019. Please be careful to generate unique IDs. 4.0.0 spark.deploy.appIdPattern app-%s-%04d The pattern for app ID generation based on Java String.format method. The default value is app-%s-%04d which represents the existing app id string, e.g., app-20231031224509-0008. Please be careful to generate unique IDs. 4.0.0 spark.worker.timeout 60 Number of seconds after which the standalone deploy master considers a worker lost if it receives no heartbeats. 0.6.2 spark.dead.worker.persistence 15 Number of iterations to keep the dead worker information in UI. By default, the dead worker is visible for (15 + 1) * spark.worker.timeout since its last heartbeat. 0.8.0 spark.worker.resource.{name}.amount (none) Amount of a particular resource to use on the worker. 3.0.0 spark.worker.resource.{name}.discoveryScript (none) Path to resource discovery script, which is used to find a particular resource while worker starting up. And the output of the script should be formatted like the ResourceInformation class. 3.0.0 spark.worker.resourcesFile (none) Path to resources file which is used to find various resources while worker starting up. The content of resources file should be formatted like [{\"id\":{\"componentName\": \"spark.worker\", \"resourceName\":\"gpu\"}, \"addresses\":[\"0\",\"1\",\"2\"]}]. If a particular resource is not found in the resources file, the discovery script would be used to find that resource. If the discovery script also does not find the resources, the worker will fail to start up. 3.0.0 SPARK_WORKER_OPTS supports the following system NameDefaultMeaningSince Version spark.worker.initialRegistrationRetries 6 The number of retries to reconnect in short intervals (between 5 and 15 seconds). 4.0.0 spark.worker.maxRegistrationRetries 16 The max number of retries to reconnect. After spark.worker.initialRegistrationRetries attempts, the interval is between 30 and 90 seconds. 4.0.0 spark.worker.cleanup.enabled true Enable periodic cleanup of worker / application directories. Note that this only affects standalone mode, as YARN works differently. Only the directories of stopped applications are cleaned up. This should be enabled if spark.shuffle.service.db.enabled is \"true\" 1.0.0 spark.worker.cleanup.interval 1800 (30 minutes) Controls the interval, in seconds, at which the worker cleans up old application work dirs on the local machine. 1.0.0 spark.worker.cleanup.appDataTtl 604800 (7 days, 7 * 24 * 3600) The number of seconds to retain application work directories on each worker. This is a Time To Live and should depend on the amount of available disk space you have. Application logs and jars are downloaded to each application work dir. Over time, the work dirs can quickly fill up disk space, especially if you run jobs very frequently. 1.0.0 spark.shuffle.service.db.enabled true Store External Shuffle service state on local disk so that when the external shuffle service is restarted, it will automatically reload info on current executors. This only affects standalone mode (yarn always has this behavior enabled). You should also enable spark.worker.cleanup.enabled, to ensure that the state eventually gets cleaned up. This config may be removed in the future. 3.0.0 spark.shuffle.service.db.backend ROCKSDB When spark.shuffle.service.db.enabled is true, user can use this to specify the kind of disk-based store used in shuffle service state store. This supports ROCKSDB and LEVELDB (deprecated) now and ROCKSDB as default value. The original data store in RocksDB/LevelDB will not be automatically convert to another kind of storage now. 3.4.0 spark.storage.cleanupFilesAfterExecutorExit true Enable cleanup non-shuffle files(such as temp. shuffle blocks, cached RDD/broadcast blocks, spill files, etc) of worker directories following executor exits. Note that this doesn't overlap with spark.worker.cleanup.enabled, as this enables cleanup of non-shuffle files in local directories of a dead executor, while spark.worker.cleanup.enabled enables cleanup of all files/subdirectories of a stopped and timeout application. This only affects Standalone mode, support of other cluster managers can be added in the future. 2.4.0 spark.worker.ui.compressedLogFileLengthCacheSize 100 For compressed log files, the uncompressed file can only be computed by uncompressing the files. Spark caches the uncompressed file size of compressed log files. This property controls the cache size. 2.0.2 spark.worker.idPattern worker-%s-%s-%d The pattern for worker ID generation based on Java String.format method. The default value is worker-%s-%s-%d which represents the existing worker id string, e.g., worker-20231109183042-[fe80::1%lo0]-39729. Please be careful to generate unique IDs 4.0.0 Resource Allocation and Configuration Overview Please make sure to have read the Custom Resource Scheduling and Configuration Overview section on the configuration page. This section only talks about the Spark Standalone specific aspects of resource scheduling. Spark Standalone has 2 parts, the first is configuring the resources for the Worker, the second is the resource allocation for a specific application. The user must configure the Workers to have a set of resources available so that it can assign them out to Executors. The spark.worker.resource.{resourceName}.amount is used to control the amount of each resource the worker has allocated. The user must also specify either spark.worker.resourcesFile or spark.worker.resource.{resourceName}.discoveryScript to specify how the Worker discovers the resources its assigned. See the descriptions above for each of those to see which method works best for your setup. The second part is running an application on Spark Standalone. The only special case from the standard Spark resource configs is when you are running the Driver in client mode. For a Driver in client mode, the user can specify the resources it uses via spark.driver.resourcesFile or spark.driver.resource.{resourceName}.discoveryScript. If the Driver is running on the same host as other Drivers, please make sure the resources file or discovery script only returns resources that do not conflict with other Drivers running on the same node. Note, the user does not need to specify a discovery script when submitting an application as the Worker will start each Executor with the resources it allocates to it. Connecting an Application to the Cluster To run an application on the Spark cluster, simply pass the spark://IP:PORT URL of the master as to the SparkContext constructor. To run an interactive Spark shell against the cluster, run the following /bin/spark-shell --master spark://IP:PORT You can also pass an option --total-executor-cores <numCores> to control the number of cores that spark-shell uses on the cluster. Client Properties Spark applications supports the following configuration properties specific to standalone NameDefault ValueMeaningSince Version spark.standalone.submit.waitAppCompletion false In standalone cluster mode, controls whether the client waits to exit until the application completes. If set to true, the client process will stay alive polling the driver's status. Otherwise, the client process will exit after submission. 3.1.0 Launching Spark Applications Spark Protocol The spark-submit script provides the most straightforward way to submit a compiled Spark application to the cluster. For standalone clusters, Spark currently supports two deploy modes. In client mode, the driver is launched in the same process as the client that submits the application. In cluster mode, however, the driver is launched from one of the Worker processes inside the cluster, and the client process exits as soon as it fulfills its responsibility of submitting the application without waiting for the application to finish. If your application is launched through Spark submit, then the application jar is automatically distributed to all worker nodes. For any additional jars that your application depends on, you should specify them through the --jars flag using comma as a delimiter (e.g. --jars jar1,jar2). To control the application’s configuration or execution environment, see Spark Configuration. Additionally, standalone cluster mode supports restarting your application automatically if it exited with non-zero exit code. To use this feature, you may pass in the --supervise flag to spark-submit when launching your application. Then, if you wish to kill an application that is failing repeatedly, you may do so /bin/spark-class org.apache.spark.deploy.Client kill <master url> <driver ID> You can find the driver ID through the standalone Master web UI at http://<master url>:8080. REST API If spark.master.rest.enabled is enabled, Spark master provides additional REST API via http://[host:port]/[version]/submissions/[action] where host is the master host, and port is the port number specified by spark.master.rest.port (default: 6066), and version is a protocol version, v1 as of today, and action is one of the following supported actions. CommandHTTP METHODDescriptionSince Version create POST Create a Spark driver via cluster mode. Since 4.0.0, Spark master supports server-side variable replacements for the values of Spark properties and environment variables. 1.3.0 kill POST Kill a single Spark driver. 1.3.0 killall POST Kill all running Spark drivers. 4.0.0 status GET Check the status of a Spark job. 1.3.0 clear POST Clear the completed drivers and applications. 4.0.0 The following is a curl CLI command example with the pi.py and REST API. $ curl -XPOST http://IP:PORT/v1/submissions/create \\ --header \"Content-Type:application/json;charset=UTF-8\" \\ --data '{ \"appResource\": \"\", \"sparkProperties\": { \"spark.master\": \"spark://master:7077\", \"spark.app.name\": \"Spark Pi\", \"spark.driver.memory\": \"1g\", \"spark.driver.cores\": \"1\", \"spark.jars\": \"\" }, \"clientSparkVersion\": \"\", \"mainClass\": \"org.apache.spark.deploy.SparkSubmit\", \"environmentVariables\": { }, \"action\": \"CreateSubmissionRequest\", \"appArgs\": [ \"/opt/spark/examples/src/main/python/pi.py\", \"10\" ] }' The following is the response from the REST API for the above create request. { \"action\" : \"CreateSubmissionResponse\", \"message\" : \"Driver successfully submitted as driver-20231124153531-0000\", \"serverSparkVersion\" : \"4.0.0\", \"submissionId\" : \"driver-20231124153531-0000\", \"success\" : true } When Spark master requires HTTP Authorization header via spark.master.rest.filters=org.apache.spark.ui.JWSFilter and spark.org.apache.spark.ui.JWSFilter.param.secretKey=BASE64URL-ENCODED-KEY configurations, curl CLI command can provide the required header like the following. $ curl -XPOST http://IP:PORT/v1/submissions/create \\ --header \"Authorization: Bearer USER-PROVIDED-WEB-TOEN-SIGNED-BY-THE-SAME-SHARED-KEY\" ... For sparkProperties and environmentVariables, users can use place holders for server-side environment variables like the following. ... \"sparkProperties\": { \"spark.hadoop.fs.s3a.endpoint\": \"{{AWS_ENDPOINT_URL}}\", \"spark.hadoop.fs.s3a.endpoint.region\": \"{{AWS_REGION}}\" }, \"environmentVariables\": { \"AWS_CA_BUNDLE\": \"{{AWS_CA_BUNDLE}}\" }, ... Resource Scheduling The standalone cluster mode currently only supports a simple FIFO scheduler across applications. However, to allow multiple concurrent users, you can control the maximum number of resources each application will use. By default, it will acquire all cores in the cluster, which only makes sense if you just run one application at a time. You can cap the number of cores by setting spark.cores.max in your SparkConf. For conf = new SparkConf() .setMaster(...) .setAppName(...) .set(\"spark.cores.max\", \"10\") val sc = new SparkContext(conf) In addition, you can configure spark.deploy.defaultCores on the cluster master process to change the default for applications that don’t set spark.cores.max to something less than infinite. Do this by adding the following to conf/spark-env.sh: export SPARK_MASTER_OPTS=\"-Dspark.deploy.defaultCores=<value>\" This is useful on shared clusters where users might not have configured a maximum number of cores individually. Executors Scheduling The number of cores assigned to each executor is configurable. When spark.executor.cores is explicitly set, multiple executors from the same application may be launched on the same worker if the worker has enough cores and memory. Otherwise, each executor grabs all the cores available on the worker by default, in which case only one executor per application may be launched on each worker during one single schedule iteration. Stage Level Scheduling Overview Stage level scheduling is supported on dynamic allocation is allows users to specify different task resource requirements at the stage level and will use the same executors requested at startup. When dynamic allocation is , when the Master allocates executors for one application, it will schedule based on the order of the ResourceProfile ids for multiple ResourceProfiles. The ResourceProfile with smaller id will be scheduled firstly. Normally this won’t matter as Spark finishes one stage before starting another one, the only case this might have an affect is in a job server type scenario, so its something to keep in mind. For scheduling, we will only take executor memory and executor cores from built-in executor resources and all other custom resources from a ResourceProfile, other built-in executor resources such as offHeap and memoryOverhead won’t take any effect. The base default profile will be created based on the spark configs when you submit an application. Executor memory and executor cores from the base default profile can be propagated to custom ResourceProfiles, but all other custom resources can not be propagated. Caveats As mentioned in Dynamic Resource Allocation, if cores for each executor is not explicitly specified with dynamic allocation enabled, spark will possibly acquire much more executors than expected. So you are recommended to explicitly set executor cores for each resource profile when using stage level scheduling. Monitoring and Logging Spark’s standalone mode offers a web-based user interface to monitor the cluster. The master and each worker has its own web UI that shows cluster and job statistics. By default, you can access the web UI for the master at port 8080. The port can be changed either in the configuration file or via command-line options. In addition, detailed log output for each job is also written to the work directory of each worker node (SPARK_HOME/work by default). You will see two files for each job, stdout and stderr, with all output it wrote to its console. Running Alongside Hadoop You can run Spark alongside your existing Hadoop cluster by just launching it as a separate service on the same machines. To access Hadoop data from Spark, just use an hdfs:// URL (typically hdfs://<namenode>:9000/path, but you can find the right URL on your Hadoop Namenode’s web UI). Alternatively, you can set up a separate cluster for Spark, and still have it access HDFS over the network; this will be slower than disk-local access, but may not be a concern if you are still running in the same local area network (e.g. you place a few Spark machines on each rack that you have Hadoop on). Configuring Ports for Network Security Generally speaking, a Spark cluster and its services are not deployed on the public internet. They are generally private services, and should only be accessible within the network of the organization that deploys Spark. Access to the hosts and ports used by Spark services should be limited to origin hosts that need to access the services. This is particularly important for clusters using the standalone resource manager, as they do not support fine-grained access control in a way that other resource managers do. For a complete list of ports to configure, see the security page. High Availability By default, standalone scheduling clusters are resilient to Worker failures (insofar as Spark itself is resilient to losing work by moving it to other workers). However, the scheduler uses a Master to make scheduling decisions, and this (by default) creates a single point of the Master crashes, no new applications can be created. In order to circumvent this, we have two high availability schemes, detailed below. Standby Masters with ZooKeeper Overview Utilizing ZooKeeper to provide leader election and some state storage, you can launch multiple Masters in your cluster connected to the same ZooKeeper instance. One will be elected “leader” and the others will remain in standby mode. If the current leader dies, another Master will be elected, recover the old Master’s state, and then resume scheduling. The entire recovery process (from the time the first leader goes down) should take between 1 and 2 minutes. Note that this delay only affects scheduling new applications – applications that were already running during Master failover are unaffected. Learn more about getting started with ZooKeeper here. Configuration In order to enable this recovery mode, you can set SPARK_DAEMON_JAVA_OPTS in spark-env by configuring spark.deploy.recoveryMode and related spark.deploy.zookeeper.* configurations. Possible you have multiple Masters in your cluster but fail to correctly configure the Masters to use ZooKeeper, the Masters will fail to discover each other and think they’re all leaders. This will not lead to a healthy cluster state (as all Masters will schedule independently). Details After you have a ZooKeeper cluster set up, enabling high availability is straightforward. Simply start multiple Master processes on different nodes with the same ZooKeeper configuration (ZooKeeper URL and directory). Masters can be added and removed at any time. In order to schedule new applications or add Workers to the cluster, they need to know the IP address of the current leader. This can be accomplished by simply passing in a list of Masters where you used to pass in a single one. For example, you might start your SparkContext pointing to spark://host1:port1,host2:port2. This would cause your SparkContext to try registering with both Masters – if host1 goes down, this configuration would still be correct as we’d find the new leader, host2. There’s an important distinction to be made between “registering with a Master” and normal operation. When starting up, an application or Worker needs to be able to find and register with the current lead Master. Once it successfully registers, though, it is “in the system” (i.e., stored in ZooKeeper). If failover occurs, the new leader will contact all previously registered applications and Workers to inform them of the change in leadership, so they need not even have known of the existence of the new Master at startup. Due to this property, new Masters can be created at any time, and the only thing you need to worry about is that new applications and Workers can find it to register with in case it becomes the leader. Once registered, you’re taken care of. Single-Node Recovery with Local File System Overview ZooKeeper is the best way to go for production-level high availability, but if you just want to be able to restart the Master if it goes down, FILESYSTEM mode can take care of it. When applications and Workers register, they have enough state written to the provided directory so that they can be recovered upon a restart of the Master process. Configuration In order to enable this recovery mode, you can set SPARK_DAEMON_JAVA_OPTS in spark-env using this propertyDefault ValueMeaningSince Version spark.deploy.recoveryMode NONE The recovery mode setting to recover submitted Spark jobs with cluster mode when it failed and relaunches. Set to FILESYSTEM to enable file-system-based single-node recovery mode, ROCKSDB to enable RocksDB-based single-node recovery mode, ZOOKEEPER to use Zookeeper-based recovery mode, and CUSTOM to provide a customer provider class via additional `spark.deploy.recoveryMode.factory` configuration. NONE is the default value which disables this recovery mode. 0.8.1 spark.deploy.recoveryDirectory \"\" The directory in which Spark will store recovery state, accessible from the Master's perspective. Note that the directory should be clearly manually if spark.deploy.recoveryMode or spark.deploy.recoveryCompressionCodec is changed. 0.8.1 spark.deploy.recoveryCompressionCodec (none) A compression codec for persistence engines. none (default), lz4, lzf, snappy, and zstd. Currently, only FILESYSTEM mode supports this configuration. 4.0.0 spark.deploy.recoveryTimeout (none) The timeout for recovery process. The default value is the same with spark.worker.timeout. 4.0.0 spark.deploy.recoveryMode.factory \"\" A class to implement StandaloneRecoveryModeFactory interface 1.2.0 spark.deploy.zookeeper.url None When spark.deploy.recoveryMode is set to ZOOKEEPER, this configuration is used to set the zookeeper URL to connect to. 0.8.1 spark.deploy.zookeeper.dir None When spark.deploy.recoveryMode is set to ZOOKEEPER, this configuration is used to set the zookeeper directory to store recovery state. 0.8.1 Details This solution can be used in tandem with a process monitor/manager like monit, or just to enable manual recovery via restart. While filesystem recovery seems straightforwardly better than not doing any recovery at all, this mode may be suboptimal for certain development or experimental purposes. In particular, killing a master via stop-master.sh does not clean up its recovery state, so whenever you start a new Master, it will enter recovery mode. This could increase the startup time by up to 1 minute if it needs to wait for all previously-registered Workers/clients to timeout. While it’s not officially supported, you could mount an NFS directory as the recovery directory. If the original Master node dies completely, you could then start a Master on a different node, which would correctly recover all previously registered Workers/applications (equivalent to ZooKeeper recovery). Future applications will have to be able to find the new Master, however, in order to register.\n\nExample:\n```text\n./sbin/start-master.sh\n```\n\nExample:\n```text\n./sbin/start-worker.sh <master-spark-URL>\n```\n\nExample:\n```text\n./bin/spark-shell --master spark://IP:PORT\n```\n\nExample:\n```text\n./bin/spark-class org.apache.spark.deploy.Client kill <master url> <driver ID>\n```\n\nExample:\n```text\n$ curl -XPOST http://IP:PORT/v1/submissions/create \\\n--header \"Content-Type:application/json;charset=UTF-8\" \\\n--data '{\n  \"appResource\": \"\",\n  \"sparkProperties\": {\n    \"spark.master\": \"spark://master:7077\",\n    \"spark.app.name\": \"Spark Pi\",\n    \"spark.driver.memory\": \"1g\",\n    \"spark.driver.cores\": \"1\",\n    \"spark.jars\": \"\"\n  },\n  \"clientSparkVersion\": \"\",\n  \"mainClass\": \"org.apache.spark.deploy.SparkSubmit\",\n  \"environmentVariables\": { },\n  \"action\": \"CreateSubmissionRequest\",\n  \"appArgs\": [ \"/opt/spark/examples/src/main/python/pi.py\", \"10\" ]\n}'\n```\n\nExample:\n```text\n{\n  \"action\" : \"CreateSubmissionResponse\",\n  \"message\" : \"Driver successfully submitted as driver-20231124153531-0000\",\n  \"serverSparkVersion\" : \"4.0.0\",\n  \"submissionId\" : \"driver-20231124153531-0000\",\n  \"success\" : true\n}\n```\n\nExample:\n```text\n$ curl -XPOST http://IP:PORT/v1/submissions/create \\\n--header \"Authorization: Bearer USER-PROVIDED-WEB-TOEN-SIGNED-BY-THE-SAME-SHARED-KEY\"\n...\n```\n\nExample:\n```text\n...\n  \"sparkProperties\": {\n    \"spark.hadoop.fs.s3a.endpoint\": \"{{AWS_ENDPOINT_URL}}\",\n    \"spark.hadoop.fs.s3a.endpoint.region\": \"{{AWS_REGION}}\"\n  },\n  \"environmentVariables\": {\n    \"AWS_CA_BUNDLE\": \"{{AWS_CA_BUNDLE}}\"\n  },\n...\n```\n\nExample:\n```scala\nval conf = new SparkConf()\n  .setMaster(...)\n  .setAppName(...)\n  .set(\"spark.cores.max\", \"10\")\nval sc = new SparkContext(conf)\n```\n\nExample:\n```bash\nexport SPARK_MASTER_OPTS=\"-Dspark.deploy.defaultCores=<value>\"\n```\n\n<|endofdoc|>","metadata":{"transformedAt":"2026-08-18T15:21:43.057Z","totalSectionsIncluded":1,"totalCodeBlocksIncluded":10,"totalLines":89,"estimatedTokens":8781}}22{"id":"doc-spark_connect_overview_spark_4_2_0_documentation-fd8fe5e6","source":"documentation","title":"Spark Connect Overview - Spark 4.2.0 Documentation","url":"https://spark.apache.org/docs/latest/spark-connect-overview.html","text":"Spark Connect Overview Building client-side Spark applications In Apache Spark 3.4, Spark Connect introduced a decoupled client-server architecture that allows remote connectivity to Spark clusters using the DataFrame API and unresolved logical plans as the protocol. The separation between client and server allows Spark and its open ecosystem to be leveraged from everywhere. It can be embedded in modern data applications, in IDEs, Notebooks and programming languages. To get started, see Connect. How Spark Connect works The Spark Connect client library is designed to simplify Spark application development. It is a thin API that can be embedded application servers, IDEs, notebooks, and programming languages. The Spark Connect API builds on Spark’s DataFrame API using unresolved logical plans as a language-agnostic protocol between the client and the Spark driver. The Spark Connect client translates DataFrame operations into unresolved logical query plans which are encoded using protocol buffers. These are sent to the server using the gRPC framework. The Spark Connect endpoint embedded on the Spark Server receives and translates unresolved logical plans into Spark’s logical plan operators. This is similar to parsing a SQL query, where attributes and relations are parsed and an initial parse plan is built. From there, the standard Spark execution process kicks in, ensuring that Spark Connect leverages all of Spark’s optimizations and enhancements. Results are streamed back to the client through gRPC as Apache Arrow-encoded row batches. How Spark Connect client applications differ from classic Spark applications One of the main design goals of Spark Connect is to enable a full separation and isolation of the client from the server. As a consequence, there are some changes that developers need to be aware of when using Spark client does not run in the same process as the Spark driver. This means that the client cannot directly access and interact with the driver JVM to manipulate the execution environment. In particular, in PySpark, the client does not use Py4J and thus the accessing the private fields holding the JVM implementation of DataFrame, Column, SparkSession, etc. is not possible (e.g. df._jdf). By design, the Spark Connect protocol uses Sparks logical plans as the abstraction to be able to declaratively describe the operations to be executed on the server. Consequently, the Spark Connect protocol does not support all the execution APIs of Spark, most importantly RDDs. Spark Connect provides a session-based client for its consumers. This means that the client does not have access to properties of the cluster that manipulate the environment for all connected clients. Most importantly, the client does not have access to the static Spark configuration or the SparkContext. Operational benefits of Spark Connect With this new architecture, Spark Connect mitigates several multi-tenant operational : Applications that use too much memory will now only impact their own environment as they can run in their own processes. Users can define their own dependencies on the client and don’t need to worry about potential conflicts with the Spark driver. Spark driver can now seamlessly be upgraded independently of applications, for example to benefit from performance improvements and security fixes. This means applications can be forward-compatible, as long as the server-side RPC definitions are designed to be backwards compatible. Debuggability and Connect enables interactive debugging during development directly from your favorite IDE. Similarly, applications can be monitored using the application’s framework native metrics and logging libraries. How to use Spark Connect Spark Connect is available and supports PySpark and Scala applications. We will walk through how to run an Apache Spark server with Spark Connect and connect to it from a client application using the Spark Connect client library. Download and start Spark server with Spark Connect First, download Spark from the Download Apache Spark page. Choose the latest release in the release drop down at the top of the page. Then choose your package type, typically “Pre-built for Apache Hadoop 3.5 and later”, and click the link to download. Now extract the Spark package you just downloaded on your computer, for -xvf spark-4.2.0-bin-hadoop3.tgz In a terminal window, go to the spark folder in the location where you extracted Spark before and run the start-connect-server.sh script to start Spark server with Spark Connect, like in this /sbin/start-connect-server.sh Make sure to use the same version of the package as the Spark version you downloaded previously. In this example, Spark 4.2.0 with Scala 2.13. Now Spark server is running and ready to accept Spark Connect sessions from client applications. In the next section we will walk through how to use Spark Connect when writing client applications. Use Spark Connect for interactive analysis When creating a Spark session, you can specify that you want to use Spark Connect and there are a few ways to do that outlined as follows. If you do not use one of the mechanisms outlined here, your Spark session will work just like before, without leveraging Spark Connect. Set SPARK_REMOTE environment variable If you set the SPARK_REMOTE environment variable on the client machine where your Spark client application is running and create a new Spark Session as in the following example, the session will be a Spark Connect session. With this approach, there is no code change needed to start using Spark Connect. In a terminal window, set the SPARK_REMOTE environment variable to point to the local Spark server you started previously on your SPARK_REMOTE=\"sc://localhost\" And start the Spark shell as /bin/pyspark The PySpark shell is now connected to Spark using Spark Connect as indicated in the welcome connected to the Spark Connect server at localhost Specify Spark Connect when creating Spark session You can also specify that you want to use Spark Connect explicitly when you create a Spark session. For example, you can launch the PySpark shell with Spark Connect as illustrated here. To launch the PySpark shell with Spark Connect, simply include the remote parameter and specify the location of your Spark server. We are using localhost in this example to connect to the local Spark server we started /bin/pyspark --remote \"sc://localhost\" And you will notice that the PySpark shell welcome message tells you that you have connected to Spark using Spark connected to the Spark Connect server at localhost You can also check the Spark session type. If it includes .connect. you are using Spark Connect as shown in this available as 'spark'. >>> type(spark) <class 'pyspark.sql.connect.session.SparkSession'> Now you can run PySpark code in the shell to see Spark Connect in action: >>> columns = [\"id\", \"name\"] >>> data = [(1,\"Sarah\"), (2,\"Maria\")] >>> df = spark.createDataFrame(data).toDF(*columns) >>> df.show() +---+-----+ | id| name| +---+-----+ | 1|Sarah| | 2|Maria| +---+-----+ For the Scala shell, we use an Ammonite-based REPL. Otherwise, very similar with PySpark shell. ./bin/spark-shell --remote \"sc://localhost\" A greeting message will appear when the REPL successfully to ____ __ / __/__ ___ _____/ /__ _\\ \\/ _ \\/ _ `/ __/ '_/ /___/ .__/\\_,_/_/ /_/\\_\\ version 4.2.0 /_/ Type in expressions to have them evaluated. Spark session available as 'spark'. By default, the REPL will attempt to connect to a local Spark Server. Run the following Scala code in the shell to see Spark Connect in action: @ spark.range(10).count = 10L Configure client-server connection By default, the REPL will attempt to connect to a local Spark Server on port 15002. The connection, however, may be configured in several ways as described in this configuration reference. Set SPARK_REMOTE environment variable The SPARK_REMOTE environment variable can be set on the client machine to customize the client-server connection that is initialized at REPL startup. export SPARK_REMOTE=\"sc://myhost.com:443/;token=ABCDEFG\" ./bin/spark-shell or SPARK_REMOTE=\"sc://myhost.com:443/;token=ABCDEFG\" spark-connect-repl Configure programmatically with a connection string The connection may also be programmatically created using SparkSession#builder as in this example: @ import org.apache.spark.sql.SparkSession @ val spark = SparkSession.builder.remote(\"sc://localhost:443/;token=ABCDEFG\").getOrCreate() Use Spark Connect in standalone applications First, install PySpark with pip install pyspark-client==4.2.0 or if building a packaged PySpark application/library, add it your setup.py file =[ 'pyspark-client==4.2.0' ] When writing your own code, include the remote function with a reference to your Spark server when you create a Spark session, as in this pyspark.sql import SparkSession spark = SparkSession.builder.remote(\"sc://localhost\").getOrCreate() For illustration purposes, we’ll create a simple Spark Connect application, SimpleApp.py: \"\"\"SimpleApp.py\"\"\" from pyspark.sql import SparkSession logFile = \"YOUR_SPARK_HOME/README.md\" # Should be some file on your system spark = SparkSession.builder.remote(\"sc://localhost\").appName(\"SimpleApp\").getOrCreate() logData = spark.read.text(logFile).cache() numAs = logData.filter(logData.value.contains('a')).count() numBs = logData.filter(logData.value.contains('b')).count() print(\"Lines with a: %i, lines with b: %i\" % (numAs, numBs)) spark.stop() This program just counts the number of lines containing ‘a’ and the number containing ‘b’ in a text file. Note that you’ll need to replace YOUR_SPARK_HOME with the location where Spark is installed. We can run this application with the regular Python interpreter as follows: # Use the Python interpreter to run your application $ python SimpleApp.py ... Lines with , lines with To use Spark Connect as part of a Scala application/project, we first need to include the right dependencies. Using the sbt build system as an example, we add the following dependencies to the build.sbt += \"org.apache.spark\" %% \"spark-connect-client-jvm\" % \"4.2.0\" When writing your own code, include the remote function with a reference to your Spark server when you create a Spark session, as in this org.apache.spark.sql.SparkSession val spark = SparkSession.builder().remote(\"sc://localhost\").getOrCreate() that reference User Defined Code such as UDFs, filter, map, etc require a ClassFinder to be registered to pickup and upload any required classfiles. Also, any JAR dependencies must be uploaded to the server using SparkSession#AddArtifact. org.apache.spark.sql.connect.client.REPLClassDirMonitor // Register a ClassFinder to monitor and upload the classfiles from the build output. val classFinder = new REPLClassDirMonitor(<ABSOLUTE_PATH_TO_BUILD_OUTPUT_DIR>) spark.registerClassFinder(classFinder) // Upload JAR dependencies spark.addArtifact(<ABSOLUTE_PATH_JAR_DEP>) Here, ABSOLUTE_PATH_TO_BUILD_OUTPUT_DIR is the output directory where the build system writes classfiles into and ABSOLUTE_PATH_JAR_DEP is the location of the JAR on the local file system. The REPLClassDirMonitor is a provided implementation of ClassFinder that monitors a specific directory but one may implement their own class extending ClassFinder for customized search and monitoring. For more information on application development with Spark Connect as well as extending Spark Connect with custom functionality, see Application Development with Spark Connect. Client application authentication While Spark Connect does not have built-in authentication, it is designed to work seamlessly with your existing authentication infrastructure. Its gRPC HTTP/2 interface allows for the use of authenticating proxies, which makes it possible to secure Spark Connect without having to implement authentication logic in Spark directly. What is supported Spark 3.4, Spark Connect supports most PySpark APIs, including DataFrame, Functions, and Column. However, some APIs such as SparkContext and RDD are not supported. You can check which APIs are currently supported in the API reference documentation. Supported APIs are labeled “Supports Spark Connect” so you can check whether the APIs you are using are available before migrating existing code to Spark Connect. Spark 3.5, Spark Connect supports most Scala APIs, including Dataset, functions, Column, Catalog and KeyValueGroupedDataset. User-Defined Functions (UDFs) are supported, by default for the shell and in standalone applications with additional set-up requirements. Majority of the Streaming API is supported, including DataStreamReader, DataStreamWriter, StreamingQuery and StreamingQueryListener. APIs such as SparkContext and RDD are unsupported in Spark Connect. Support for more APIs is planned for upcoming Spark releases.\n\nExample:\n```bash\ntar -xvf spark-4.2.0-bin-hadoop3.tgz\n```\n\nExample:\n```bash\n./sbin/start-connect-server.sh\n```\n\nExample:\n```bash\nexport SPARK_REMOTE=\"sc://localhost\"\n```\n\nExample:\n```bash\n./bin/pyspark\n```\n\nExample:\n```python\nClient connected to the Spark Connect server at localhost\n```\n\nExample:\n```bash\n./bin/pyspark --remote \"sc://localhost\"\n```\n\nExample:\n```python\nSparkSession available as 'spark'.\n>>> type(spark)\n<class 'pyspark.sql.connect.session.SparkSession'>\n```\n\nExample:\n```python\n>>> columns = [\"id\", \"name\"]\n>>> data = [(1,\"Sarah\"), (2,\"Maria\")]\n>>> df = spark.createDataFrame(data).toDF(*columns)\n>>> df.show()\n+---+-----+\n| id| name|\n+---+-----+\n|  1|Sarah|\n|  2|Maria|\n+---+-----+\n```\n\nExample:\n```bash\n./bin/spark-shell --remote \"sc://localhost\"\n```\n\nExample:\n```bash\nWelcome to\n      ____              __\n     / __/__  ___ _____/ /__\n    _\\ \\/ _ \\/ _ `/ __/  '_/\n   /___/ .__/\\_,_/_/ /_/\\_\\   version 4.2.0\n      /_/\n\nType in expressions to have them evaluated.\nSpark session available as 'spark'.\n```\n\nExample:\n```scala\n@ spark.range(10).count\nres0: Long = 10L\n```\n\nExample:\n```bash\nexport SPARK_REMOTE=\"sc://myhost.com:443/;token=ABCDEFG\"\n./bin/spark-shell\n```\n\nExample:\n```bash\nSPARK_REMOTE=\"sc://myhost.com:443/;token=ABCDEFG\" spark-connect-repl\n```\n\nExample:\n```scala\n@ import org.apache.spark.sql.SparkSession\n@ val spark = SparkSession.builder.remote(\"sc://localhost:443/;token=ABCDEFG\").getOrCreate()\n```\n\nExample:\n```python\ninstall_requires=[\n'pyspark-client==4.2.0'\n]\n```\n\nExample:\n```python\nfrom pyspark.sql import SparkSession\nspark = SparkSession.builder.remote(\"sc://localhost\").getOrCreate()\n```\n\nExample:\n```python\n\"\"\"SimpleApp.py\"\"\"\nfrom pyspark.sql import SparkSession\n\nlogFile = \"YOUR_SPARK_HOME/README.md\"  # Should be some file on your system\nspark = SparkSession.builder.remote(\"sc://localhost\").appName(\"SimpleApp\").getOrCreate()\nlogData = spark.read.text(logFile).cache()\n\nnumAs = logData.filter(logData.value.contains('a')).count()\nnumBs = logData.filter(logData.value.contains('b')).count()\n\nprint(\"Lines with a: %i, lines with b: %i\" % (numAs, numBs))\n\nspark.stop()\n```\n\nExample:\n```python\n# Use the Python interpreter to run your application\n$ python SimpleApp.py\n...\nLines with a: 72, lines with b: 39\n```\n\nExample:\n```sbt\nlibraryDependencies += \"org.apache.spark\" %% \"spark-connect-client-jvm\" % \"4.2.0\"\n```\n\nExample:\n```scala\nimport org.apache.spark.sql.SparkSession\nval spark = SparkSession.builder().remote(\"sc://localhost\").getOrCreate()\n```\n\nExample:\n```scala\nimport org.apache.spark.sql.connect.client.REPLClassDirMonitor\n// Register a ClassFinder to monitor and upload the classfiles from the build output.\nval classFinder = new REPLClassDirMonitor(<ABSOLUTE_PATH_TO_BUILD_OUTPUT_DIR>)\nspark.registerClassFinder(classFinder)\n\n// Upload JAR dependencies\nspark.addArtifact(<ABSOLUTE_PATH_JAR_DEP>)\n```\n\n<|endofdoc|>","metadata":{"transformedAt":"2026-08-18T15:21:43.061Z","totalSectionsIncluded":1,"totalCodeBlocksIncluded":21,"totalLines":155,"estimatedTokens":3922}}23{"id":"doc-building_spark_spark_4_2_0_documentation-8417d111","source":"documentation","title":"Building Spark - Spark 4.2.0 Documentation","url":"https://spark.apache.org/docs/latest/building-spark.html","text":"Building Spark Building Apache Spark Apache Maven Setting up Maven’s Memory Usage build/mvn Building a Runnable Distribution Specifying the Hadoop Version and Enabling YARN Building With Hive and JDBC Support Packaging without Hadoop Dependencies for YARN Building with Kubernetes support Building submodules individually Building with JVM Profile support Continuous Compilation Building with SBT Setting up SBT’s Memory Usage Speeding up Compilation Encrypted Filesystems IntelliJ IDEA or Eclipse Running Tests Testing with SBT Running Individual Tests PySpark pip installable PySpark Tests with Maven or SBT Running R Tests (deprecated) Running Docker-based Integration Test Suites Building and testing on an IPv6-only environment Building with a user-defined protoc Building Apache Spark Apache Maven The Maven-based build is the build of reference for Apache Spark. Building Spark using Maven requires Maven 3.9.15 and Java 17/21/25. Spark requires Scala 2.13; support for Scala 2.12 was removed in Spark 4.0.0. Setting up Maven’s Memory Usage You’ll need to configure Maven to use more memory than usual by setting MAVEN_OPTS=\"-Xss64m -Xmx4g -Xms4g =128m\" (The ReservedCodeCacheSize setting is optional but recommended.) If you don’t add these parameters to MAVEN_OPTS, you may see errors and warnings like the following: [INFO] Compiling 203 Scala sources and 9 Java sources to /Users/me/Development/spark/core/target/scala-2.13/classes... [ERROR] Java heap space -> [Help 1] You can fix these problems by setting the MAVEN_OPTS variable as discussed before. using build/mvn with no MAVEN_OPTS set, the script will automatically add the above options to the MAVEN_OPTS environment variable. The test phase of the Spark build will automatically add these options to MAVEN_OPTS, even when not using build/mvn. build/mvn Spark now comes packaged with a self-contained Maven installation to ease building and deployment of Spark from source located under the build/ directory. This script will automatically download and setup all necessary build requirements (Maven, Scala) locally within the build/ directory itself. It honors any mvn binary if present already, however, will pull down its own copy of Scala regardless to ensure proper version requirements are met. build/mvn execution acts as a pass through to the mvn call allowing easy transition from previous build methods. As an example, one can build a version of Spark as /build/mvn -DskipTests clean package Other build examples can be found below. Building a Runnable Distribution To create a Spark distribution like those distributed by the Spark Downloads page, and that is laid out so as to be runnable, use ./dev/make-distribution.sh in the project root directory. By default, it uses Maven as building tool, and can be configured with Maven profile settings and so on like the direct Maven build. /dev/make-distribution.sh --name custom-spark --pip --r --tgz -Psparkr -Phive -Phive-thriftserver -Pyarn -Pkubernetes This will build Spark distribution along with Python pip and R packages. To switch to SBT (experimental), use --sbt-enabled. /dev/make-distribution.sh --name custom-spark --pip --r --tgz --sbt-enabled -Psparkr -Phive -Phive-thriftserver -Pyarn -Pkubernetes For more information on usage, run ./dev/make-distribution.sh --help Specifying the Hadoop Version and Enabling YARN You can enable the yarn profile and specify the exact version of Hadoop to compile against through the hadoop.version property. /build/mvn -Pyarn -Dhadoop.version=3.5.0 -DskipTests clean package Building With Hive and JDBC Support To enable Hive integration for Spark SQL along with its JDBC server and CLI, add the -Phive and -Phive-thriftserver profiles to your existing build options. By default Spark will build with Hive 2.3.10. # With Hive 2.3.10 support ./build/mvn -Pyarn -Phive -Phive-thriftserver -DskipTests clean package Packaging without Hadoop Dependencies for YARN The assembly directory produced by mvn package will, by default, include all of Spark’s dependencies, including Hadoop and some of its ecosystem projects. On YARN deployments, this causes multiple versions of these to appear on executor version packaged in the Spark assembly and the version on each node, included with yarn.application.classpath. The hadoop-provided profile builds the assembly without including Hadoop-ecosystem projects, like ZooKeeper and Hadoop itself. Building with Kubernetes support ./build/mvn -Pkubernetes -DskipTests clean package Building submodules individually It’s possible to build Spark submodules using the mvn -pl option. For instance, you can build the Spark Streaming module /build/mvn clean install where spark-streaming_2.13 is the artifactId as defined in streaming/pom.xml file. Building with JVM Profile support ./build/mvn -Pjvm-profiler -DskipTests clean package jvm-profiler profile builds the assembly without including the dependency ap-loader, you can download it manually from maven central repo and use it together with spark-profiler_2.13. Continuous Compilation We use the scala-maven-plugin which supports incremental and continuous compilation. E.g. ./build/mvn should run continuous compilation (i.e. wait for changes). However, this has not been tested extensively. A couple of gotchas to only scans the paths src/main and src/test (see docs), so it will only work from within certain submodules that have that structure. you’ll typically need to run mvn install from the project root for compilation within specific submodules to work; this is because submodules that depend on other submodules do so via the spark-parent module). Thus, the full flow for running continuous-compilation of the core submodule may look more like: $ ./build/mvn install $ cd core $ ../build/mvn Building with SBT Maven is the official build tool recommended for packaging Spark, and is the build of reference. But SBT is supported for day-to-day development since it can provide much faster iterative compilation. More advanced developers may wish to use SBT. The SBT build is derived from the Maven POM files, and so the same Maven profiles and variables can be set to control the SBT build. For /build/sbt package To avoid the overhead of launching sbt each time you need to re-compile, you can launch sbt in interactive mode by running build/sbt, and then run all build commands at the command prompt. Setting up SBT’s Memory Usage Configure the JVM options for SBT in .jvmopts at the project root, for =1g For the meanings of these two options, please carefully read the Setting up Maven’s Memory Usage section. Speeding up Compilation Developers who compile Spark frequently may want to speed up compilation; e.g., by avoiding re-compilation of the assembly JAR (for developers who build with SBT). For more information about how to do this, refer to the Useful Developer Tools page. Encrypted Filesystems When building on an encrypted filesystem (if your home directory is encrypted, for example), then the Spark build might fail with a “Filename too long” error. As a workaround, add the following in the configuration args of the scala-maven-plugin in the project pom.xml: <arg>-Xmax-classfile-name</arg> <arg>128</arg> and in project/SparkBuild.scala in Compile ++= Seq(\"-Xmax-classfile-name\", \"128\"), to the sharedSettings val. See also this PR if you are unsure of where to add these lines. IntelliJ IDEA or Eclipse For help in setting up IntelliJ IDEA or Eclipse for Spark development, and troubleshooting, refer to the Useful Developer Tools page. Running Tests Tests are run by default via the ScalaTest Maven plugin. Note that tests should not be run as root or an admin user. The following is an example of a command to run the /build/mvn test Testing with SBT The following is an example of a command to run the /build/sbt test Running Individual Tests For information about how to run individual tests, refer to the Useful Developer Tools page. PySpark pip installable If you are building Spark for use in a Python environment and you wish to pip install it, you will first need to build the Spark JARs as described above. Then you can construct an sdist package suitable for setup.py and pip installable package. cd python; python packaging/classic/setup.py sdist to packaging requirements you can not directly pip install from the Python directory, rather you must first build the sdist package as described above. Alternatively, you can also run make-distribution.sh with the --pip option. PySpark Tests with Maven or SBT If you are building PySpark and wish to run the PySpark tests you will need to build Spark with Hive support. ./build/mvn -DskipTests clean package -Phive ./python/run-tests If you are building PySpark with SBT and wish to run the PySpark tests, you will need to build Spark with Hive support and also build the test /build/sbt -Phive clean package ./build/sbt ./python/run-tests The run-tests script also can be limited to a specific Python version or a specific module ./python/run-tests --python-executables=python --modules=pyspark-sql Running R Tests (deprecated) To run the SparkR tests you will need to install the knitr, rmarkdown, testthat, e1071 and survival packages -e \"install.packages(c('knitr', 'rmarkdown', 'testthat', 'e1071', 'survival'), repos='https://cloud.r-project.org/')\" You can run just the SparkR tests using the /R/run-tests.sh Running Docker-based Integration Test Suites In order to run Docker integration tests, you have to install the docker engine on your box. The instructions for installation can be found at the Docker site. Once installed, the docker service needs to be started, if not already running. On Linux, this can be done by sudo service docker start. ./build/mvn install -DskipTests ./build/mvn test -Pdocker-integration-tests or ./build/sbt -Pdocker-integration-tests docker-integration-tests/test Building and testing on an IPv6-only environment Use Apache Spark GitBox URL because GitHub doesn’t support IPv6 yet. https://gitbox.apache.org/repos/asf/spark.git To build and run tests on IPv6-only environment, the following configurations are required. export SPARK_LOCAL_HOSTNAME=\"your-IPv6-address\" # e.g. '[2600:1700:232e:3de0:...]' export DEFAULT_ARTIFACT_REPOSITORY=https://ipv6.repo1.maven.org/maven2/ export MAVEN_OPTS=\"-Djava.net.preferIPv6Addresses=true\" export SBT_OPTS=\"-Djava.net.preferIPv6Addresses=true\" export SERIAL_SBT_TESTS=1 Building with a user-defined protoc When the user cannot use the official protoc binary files to build the core module in the compilation environment, for example, compiling core module on CentOS 6 or CentOS 7 which the default glibc version is less than 2.14, we can try to compile and test by specifying the user-defined protoc binary files as SPARK_PROTOC_EXEC_PATH=/path-to-protoc-exe ./build/mvn -Puser-defined-protoc -DskipDefaultProtoc clean package or export SPARK_PROTOC_EXEC_PATH=/path-to-protoc-exe ./build/sbt -Puser-defined-protoc clean package The user-defined protoc binary files can be produced in the user’s compilation environment by source code compilation, for compilation steps, please refer to protobuf.\n\nExample:\n```text\nexport MAVEN_OPTS=\"-Xss64m -Xmx4g -Xms4g -XX:ReservedCodeCacheSize=128m\"\n```\n\nExample:\n```text\n[INFO] Compiling 203 Scala sources and 9 Java sources to /Users/me/Development/spark/core/target/scala-2.13/classes...\n[ERROR] Java heap space -> [Help 1]\n```\n\nExample:\n```text\n./build/mvn -DskipTests clean package\n```\n\nExample:\n```text\n./dev/make-distribution.sh --name custom-spark --pip --r --tgz -Psparkr -Phive -Phive-thriftserver -Pyarn -Pkubernetes\n```\n\nExample:\n```text\n./dev/make-distribution.sh --name custom-spark --pip --r --tgz --sbt-enabled -Psparkr -Phive -Phive-thriftserver -Pyarn -Pkubernetes\n```\n\nExample:\n```text\n./build/mvn -Pyarn -Dhadoop.version=3.5.0 -DskipTests clean package\n```\n\nExample:\n```text\n# With Hive 2.3.10 support\n./build/mvn -Pyarn -Phive -Phive-thriftserver -DskipTests clean package\n```\n\nExample:\n```text\n./build/mvn -Pkubernetes -DskipTests clean package\n```\n\nExample:\n```text\n./build/mvn -pl :spark-streaming_2.13 clean install\n```\n\nExample:\n```text\n./build/mvn -Pjvm-profiler -DskipTests clean package\n```\n\nExample:\n```text\n./build/mvn scala:cc\n```\n\nExample:\n```text\n$ ./build/mvn install\n$ cd core\n$ ../build/mvn scala:cc\n```\n\nExample:\n```text\n./build/sbt package\n```\n\nExample:\n```text\n-Xmx2g\n-XX:ReservedCodeCacheSize=1g\n```\n\nExample:\n```text\n<arg>-Xmax-classfile-name</arg>\n<arg>128</arg>\n```\n\nExample:\n```text\nscalacOptions in Compile ++= Seq(\"-Xmax-classfile-name\", \"128\"),\n```\n\nExample:\n```text\n./build/mvn test\n```\n\nExample:\n```text\n./build/sbt test\n```\n\nExample:\n```text\ncd python; python packaging/classic/setup.py sdist\n```\n\nExample:\n```text\n./build/mvn -DskipTests clean package -Phive\n./python/run-tests\n```\n\nExample:\n```text\n./build/sbt -Phive clean package\n./build/sbt test:compile\n./python/run-tests\n```\n\nExample:\n```text\n./python/run-tests --python-executables=python --modules=pyspark-sql\n```\n\nExample:\n```text\nRscript -e \"install.packages(c('knitr', 'rmarkdown', 'testthat', 'e1071', 'survival'), repos='https://cloud.r-project.org/')\"\n```\n\nExample:\n```text\n./R/run-tests.sh\n```\n\nExample:\n```text\n./build/mvn install -DskipTests\n./build/mvn test -Pdocker-integration-tests -pl :spark-docker-integration-tests_2.13\n```\n\nExample:\n```text\n./build/sbt -Pdocker-integration-tests docker-integration-tests/test\n```\n\nExample:\n```text\nhttps://gitbox.apache.org/repos/asf/spark.git\n```\n\nExample:\n```text\nexport SPARK_LOCAL_HOSTNAME=\"your-IPv6-address\" # e.g. '[2600:1700:232e:3de0:...]'\nexport DEFAULT_ARTIFACT_REPOSITORY=https://ipv6.repo1.maven.org/maven2/\nexport MAVEN_OPTS=\"-Djava.net.preferIPv6Addresses=true\"\nexport SBT_OPTS=\"-Djava.net.preferIPv6Addresses=true\"\nexport SERIAL_SBT_TESTS=1\n```\n\nExample:\n```text\nexport SPARK_PROTOC_EXEC_PATH=/path-to-protoc-exe\n./build/mvn -Puser-defined-protoc -DskipDefaultProtoc clean package\n```\n\nExample:\n```text\nexport SPARK_PROTOC_EXEC_PATH=/path-to-protoc-exe\n./build/sbt -Puser-defined-protoc clean package\n```\n\n<|endofdoc|>","metadata":{"transformedAt":"2026-08-18T15:21:43.065Z","totalSectionsIncluded":1,"totalCodeBlocksIncluded":30,"totalLines":169,"estimatedTokens":3519}}24{"id":"doc-submitting_applications_spark_4_2_0_documentatio-11bc62dd","source":"documentation","title":"Submitting Applications - Spark 4.2.0 Documentation","url":"https://spark.apache.org/docs/latest/submitting-applications.html","text":"Submitting Applications The spark-submit script in Spark’s bin directory is used to launch applications on a cluster. It can use all of Spark’s supported cluster managers through a uniform interface so you don’t have to configure your application especially for each one. Bundling Your Application’s Dependencies If your code depends on other projects, you will need to package them alongside your application in order to distribute the code to a Spark cluster. To do this, create an assembly jar (or “uber” jar) containing your code and its dependencies. Both sbt and Maven have assembly plugins. When creating assembly jars, list Spark and Hadoop as provided dependencies; these need not be bundled since they are provided by the cluster manager at runtime. Once you have an assembled jar you can call the bin/spark-submit script as shown here while passing your jar. For Python, you can use the --py-files argument of spark-submit to add .py, .zip or .egg files to be distributed with your application. If you depend on multiple Python files we recommend packaging them into a .zip or .egg. For third-party Python dependencies, see Python Package Management. Launching Applications with spark-submit Once a user application is bundled, it can be launched using the bin/spark-submit script. This script takes care of setting up the classpath with Spark and its dependencies, and can support different cluster managers and deploy modes that Spark /bin/spark-submit \\ --class <main-class> \\ --master <master-url> \\ --deploy-mode <deploy-mode> \\ --conf <key>=<value> \\ ... # other options <application-jar> \\ [application-arguments] Some of the commonly used options : The entry point for your application (e.g. org.apache.spark.examples.SparkPi) master URL for the cluster (e.g. spark://23.195.26.187:7077) to deploy your driver on the worker nodes (cluster) or locally as an external client (client) (default: client) † Spark configuration property in key=value format. For values that contain spaces wrap “key=value” in quotes (as shown). Multiple configurations should be passed as separate arguments. (e.g. --conf <key>=<value> --conf <key2>=<value2>) to a bundled jar including your application and all dependencies. The URL must be globally visible inside of your cluster, for instance, an hdfs:// path or a file:// path that is present on all nodes. passed to the main method of your main class, if any † A common deployment strategy is to submit your application from a gateway machine that is physically co-located with your worker machines (e.g. Master node in a standalone EC2 cluster). In this setup, client mode is appropriate. In client mode, the driver is launched directly within the spark-submit process which acts as a client to the cluster. The input and output of the application is attached to the console. Thus, this mode is especially suitable for applications that involve the REPL (e.g. Spark shell). Alternatively, if your application is submitted from a machine far from the worker machines (e.g. locally on your laptop), it is common to use cluster mode to minimize network latency between the drivers and the executors. Currently, the standalone mode does not support cluster mode for Python applications. For Python applications, simply pass a .py file in the place of <application-jar>, and add Python .zip, .egg or .py files to the search path with --py-files. There are a few options available that are specific to the cluster manager that is being used. For example, with a Spark standalone cluster with cluster deploy mode, you can also specify --supervise to make sure that the driver is automatically restarted if it fails with a non-zero exit code. To enumerate all such options available to spark-submit, run it with --help. Here are a few examples of common options: # Run application locally on 8 cores ./bin/spark-submit \\ --class org.apache.spark.examples.SparkPi \\ --master \"local[8]\" \\ /path/to/examples.jar \\ 100 # Run on a Spark standalone cluster in client deploy mode ./bin/spark-submit \\ --class org.apache.spark.examples.SparkPi \\ --master spark://207.184.161.138:7077 \\ --executor-memory 20G \\ --total-executor-cores 100 \\ /path/to/examples.jar \\ 1000 # Run on a Spark standalone cluster in cluster deploy mode with supervise ./bin/spark-submit \\ --class org.apache.spark.examples.SparkPi \\ --master spark://207.184.161.138:7077 \\ --deploy-mode cluster \\ --supervise \\ --executor-memory 20G \\ --total-executor-cores 100 \\ /path/to/examples.jar \\ 1000 # Run on a YARN cluster in cluster deploy mode export HADOOP_CONF_DIR=XXX ./bin/spark-submit \\ --class org.apache.spark.examples.SparkPi \\ --master yarn \\ --deploy-mode cluster \\ --executor-memory 20G \\ --num-executors 50 \\ /path/to/examples.jar \\ 1000 # Run a Python application on a Spark standalone cluster ./bin/spark-submit \\ --master spark://207.184.161.138:7077 \\ examples/src/main/python/pi.py \\ 1000 # Run on a Kubernetes cluster in cluster deploy mode ./bin/spark-submit \\ --class org.apache.spark.examples.SparkPi \\ --master k8s://xx.yy.zz.ww:443 \\ --deploy-mode cluster \\ --executor-memory 20G \\ --num-executors 50 \\ http://path/to/examples.jar \\ 1000 Master URLs The master URL passed to Spark can be in one of the following URLMeaning local Run Spark locally with one worker thread (i.e. no parallelism at all). local[K] Run Spark locally with K worker threads (ideally, set this to the number of cores on your machine). local[K,F] Run Spark locally with K worker threads and F maxFailures (see spark.task.maxFailures for an explanation of this variable). local[*] Run Spark locally with as many worker threads as logical cores on your machine. local[*,F] Run Spark locally with as many worker threads as logical cores on your machine and F maxFailures. local-cluster[N,C,M] Local-cluster mode is only for unit tests. It emulates a distributed cluster in a single JVM with N number of workers, C cores per worker and M MiB of memory per worker. spark://HOST:PORT Connect to the given Spark standalone cluster master. The port must be whichever one your master is configured to use, which is 7077 by default. spark://HOST1:PORT1,HOST2:PORT2 Connect to the given Spark standalone cluster with standby masters with Zookeeper. The list must have all the master hosts in the high availability cluster set up with Zookeeper. The port must be whichever each master is configured to use, which is 7077 by default. yarn Connect to a YARN cluster in client or cluster mode depending on the value of --deploy-mode. The cluster location will be found based on the HADOOP_CONF_DIR or YARN_CONF_DIR variable. k8s://HOST:PORT Connect to a Kubernetes cluster in client or cluster mode depending on the value of --deploy-mode. The HOST and PORT refer to the Kubernetes API Server. It connects using TLS by default. In order to force it to use an unsecured connection, you can use k8s://http://HOST:PORT. Loading Configuration from a File The spark-submit script can load default Spark configuration values from a properties file and pass them on to your application. The file can be specified via the --properties-file parameter. When this is not specified, by default Spark will read options from conf/spark-defaults.conf in the SPARK_HOME directory. An additional flag --load-spark-defaults can be used to tell Spark to load configurations from conf/spark-defaults.conf even when a property file is provided via --properties-file. This is useful, for instance, when users want to put system-wide default settings in the former while user/cluster specific settings in the latter. Loading default Spark configurations this way can obviate the need for certain flags to spark-submit. For instance, if the spark.master property is set, you can safely omit the --master flag from spark-submit. In general, configuration values explicitly set on a SparkConf take the highest precedence, then flags passed to spark-submit, then values in the defaults file. If you are ever unclear where configuration options are coming from, you can print out fine-grained debugging information by running spark-submit with the --verbose option. Advanced Dependency Management When using spark-submit, the application jar along with any jars included with the --jars option will be automatically transferred to the cluster. URLs supplied after --jars must be separated by commas. That list is included in the driver and executor classpaths. Directory expansion does not work with --jars. Spark uses the following URL scheme to allow different strategies for disseminating : - Absolute paths and file:/ URIs are served by the driver’s HTTP file server, and every executor pulls the file from the driver HTTP server. hdfs:, http:, https:, these pull down files and JARs from the URI as expected a URI starting with local:/ is expected to exist as a local file on each worker node. This means that no network IO will be incurred, and works well for large files/JARs that are pushed to each worker, or shared via NFS, GlusterFS, etc. Note that JARs and files are copied to the working directory for each SparkContext on the executor nodes. This can use up a significant amount of space over time and will need to be cleaned up. With YARN, cleanup is handled automatically, and with Spark standalone, automatic cleanup can be configured with the spark.worker.cleanup.appDataTtl property. Users may also include any other dependencies by supplying a comma-delimited list of Maven coordinates with --packages. All transitive dependencies will be handled when using this command. Additional repositories (or resolvers in SBT) can be added in a comma-delimited fashion with the flag --repositories. (Note that credentials for password-protected repositories can be supplied in some cases in the repository URI, such as in https://user:password@host/.... Be careful when supplying credentials this way.) These commands can be used with pyspark, spark-shell, and spark-submit to include Spark Packages. For Python, the equivalent --py-files option can be used to distribute .egg, .zip and .py libraries to executors. More Information Once you have deployed your application, the cluster mode overview describes the components involved in distributed execution, and how to monitor and debug applications.\n\nExample:\n```bash\n./bin/spark-submit \\\n  --class <main-class> \\\n  --master <master-url> \\\n  --deploy-mode <deploy-mode> \\\n  --conf <key>=<value> \\\n  ... # other options\n  <application-jar> \\\n  [application-arguments]\n```\n\nExample:\n```bash\n# Run application locally on 8 cores\n./bin/spark-submit \\\n  --class org.apache.spark.examples.SparkPi \\\n  --master \"local[8]\" \\\n  /path/to/examples.jar \\\n  100\n\n# Run on a Spark standalone cluster in client deploy mode\n./bin/spark-submit \\\n  --class org.apache.spark.examples.SparkPi \\\n  --master spark://207.184.161.138:7077 \\\n  --executor-memory 20G \\\n  --total-executor-cores 100 \\\n  /path/to/examples.jar \\\n  1000\n\n# Run on a Spark standalone cluster in cluster deploy mode with supervise\n./bin/spark-submit \\\n  --class org.apache.spark.examples.SparkPi \\\n  --master spark://207.184.161.138:7077 \\\n  --deploy-mode cluster \\\n  --supervise \\\n  --executor-memory 20G \\\n  --total-executor-cores 100 \\\n  /path/to/examples.jar \\\n  1000\n\n# Run on a YARN cluster in cluster deploy mode\nexport HADOOP_CONF_DIR=XXX\n./bin/spark-submit \\\n  --class org.apache.spark.examples.SparkPi \\\n  --master yarn \\\n  --deploy-mode cluster \\\n  --executor-memory 20G \\\n  --num-executors 50 \\\n  /path/to/examples.jar \\\n  1000\n\n# Run a Python application on a Spark standalone cluster\n./bin/spark-submit \\\n  --master spark://207.184.161.138:7077 \\\n  examples/src/main/python/pi.py \\\n  1000\n\n# Run on a Kubernetes cluster in cluster deploy mode\n./bin/spark-submit \\\n  --class org.apache.spark.examples.SparkPi \\\n  --master k8s://xx.yy.zz.ww:443 \\\n  --deploy-mode cluster \\\n  --executor-memory 20G \\\n  --num-executors 50 \\\n  http://path/to/examples.jar \\\n  1000\n```\n\n<|endofdoc|>","metadata":{"transformedAt":"2026-08-18T15:21:43.068Z","totalSectionsIncluded":1,"totalCodeBlocksIncluded":2,"totalLines":72,"estimatedTokens":2991}}25{"id":"doc-job_scheduling_spark_4_2_0_documentation-1b9cc9a4","source":"documentation","title":"Job Scheduling - Spark 4.2.0 Documentation","url":"https://spark.apache.org/docs/latest/job-scheduling.html","text":"Job Scheduling Overview Scheduling Across Applications Dynamic Resource Allocation Configuration and Setup Caveats Resource Allocation Policy Request Policy Remove Policy Graceful Decommission of Executors Scheduling Within an Application Fair Scheduler Pools Default Behavior of Pools Configuring Pool Properties Scheduling using JDBC Connections Concurrent Jobs in PySpark Overview Spark has several facilities for scheduling resources between computations. First, recall that, as described in the cluster mode overview, each Spark application (instance of SparkContext) runs an independent set of executor processes. The cluster managers that Spark runs on provide facilities for scheduling across applications. Second, within each Spark application, multiple “jobs” (Spark actions) may be running concurrently if they were submitted by different threads. This is common if your application is serving requests over the network. Spark includes a fair scheduler to schedule resources within each SparkContext. Scheduling Across Applications When running on a cluster, each Spark application gets an independent set of executor JVMs that only run tasks and store data for that application. If multiple users need to share your cluster, there are different options to manage allocation, depending on the cluster manager. The simplest option, available on all cluster managers, is static partitioning of resources. With this approach, each application is given a maximum amount of resources it can use and holds onto them for its whole duration. This is the approach used in Spark’s standalone and YARN modes, as well as the K8s mode. Resource allocation can be configured as follows, based on the cluster default, applications submitted to the standalone mode cluster will run in FIFO (first-in-first-out) order, and each application will try to use all available nodes. You can limit the number of nodes an application uses by setting the spark.cores.max configuration property in it, or change the default for applications that don’t set this setting through spark.deploy.defaultCores. Finally, in addition to controlling cores, each application’s spark.executor.memory setting controls its memory use. --num-executors option to the Spark YARN client controls how many executors it will allocate on the cluster (spark.executor.instances as configuration property), while --executor-memory (spark.executor.memory configuration property) and --executor-cores (spark.executor.cores configuration property) control the resources per executor. For more information, see the YARN Spark Properties. same as the situation with Yarn, please refer to the description of Yarn above. Furthermore, Spark on K8s offers higher priority versions of spark.kubernetes.executor.limit.cores and spark.kubernetes.executor.request.cores than spark.executor.cores. For more information, see the K8s Spark Properties. Note that none of the modes currently provide memory sharing across applications. If you would like to share data this way, we recommend running a single server application that can serve multiple requests by querying the same RDDs. Dynamic Resource Allocation Spark provides a mechanism to dynamically adjust the resources your application occupies based on the workload. This means that your application may give resources back to the cluster if they are no longer used and request them again later when there is demand. This feature is particularly useful if multiple applications share resources in your Spark cluster. This feature is disabled by default and available on all coarse-grained cluster managers, i.e. standalone mode, YARN mode and K8s mode. Configuration and Setup There are several ways for using this feature. Regardless of which approach you choose, your application must set spark.dynamicAllocation.enabled to true first, additionally, your application must set spark.shuffle.service.enabled to true after you set up an external shuffle service on each worker node in the same cluster, or your application must set spark.dynamicAllocation.shuffleTracking.enabled to true, or your application must set both spark.decommission.enabled and spark.storage.decommission.shuffleBlocks.enabled to true, or your application must configure spark.shuffle.sort.io.plugin.class to use a custom ShuffleDataIO who’s ShuffleDriverComponents supports reliable storage. The purpose of the external shuffle service or the shuffle tracking or the ShuffleDriverComponents supports reliable storage is to allow executors to be removed without deleting shuffle files written by them (more detail described below). While it is simple to enable shuffle tracking, the way to set up the external shuffle service varies across cluster standalone mode, simply start your workers with spark.shuffle.service.enabled set to true. In YARN mode, follow the instructions here. All other relevant configurations are optional and under the spark.dynamicAllocation.* and spark.shuffle.service.* namespaces. For more detail, see the configurations page. Caveats In standalone mode, without explicitly setting spark.executor.cores, each executor will get all the available cores of a worker. In this case, when dynamic allocation is enabled, spark will possibly acquire much more executors than expected. When you want to use dynamic allocation in standalone mode, you are recommended to explicitly set cores for each executor before the issue SPARK-30299 got fixed. In K8s mode, we can not use this feature by setting spark.shuffle.service.enabled to true due to Spark on K8s doesn’t yet support the external shuffle service. Resource Allocation Policy At a high level, Spark should relinquish executors when they are no longer used and acquire executors when they are needed. Since there is no definitive way to predict whether an executor that is about to be removed will run a task in the near future, or whether a new executor that is about to be added will actually be idle, we need a set of heuristics to determine when to remove and request executors. Request Policy A Spark application with dynamic allocation enabled requests additional executors when it has pending tasks waiting to be scheduled. This condition necessarily implies that the existing set of executors is insufficient to simultaneously saturate all tasks that have been submitted but not yet finished. Spark requests executors in rounds. The actual request is triggered when there have been pending tasks for spark.dynamicAllocation.schedulerBacklogTimeout seconds, and then triggered again every spark.dynamicAllocation.sustainedSchedulerBacklogTimeout seconds thereafter if the queue of pending tasks persists. Additionally, the number of executors requested in each round increases exponentially from the previous round. For instance, an application will add 1 executor in the first round, and then 2, 4, 8 and so on executors in the subsequent rounds. The motivation for an exponential increase policy is twofold. First, an application should request executors cautiously in the beginning in case it turns out that only a few additional executors is sufficient. This echoes the justification for TCP slow start. Second, the application should be able to ramp up its resource usage in a timely manner in case it turns out that many executors are actually needed. Remove Policy The policy for removing executors is much simpler. A Spark application removes an executor when it has been idle for more than spark.dynamicAllocation.executorIdleTimeout seconds. Note that, under most circumstances, this condition is mutually exclusive with the request condition, in that an executor should not be idle if there are still pending tasks to be scheduled. Graceful Decommission of Executors Before dynamic allocation, if a Spark executor exits when the associated application has also exited then all state associated with the executor is no longer needed and can be safely discarded. With dynamic allocation, however, the application is still running when an executor is explicitly removed. If the application attempts to access state stored in or written by the executor, it will have to perform a recompute the state. Thus, Spark needs a mechanism to decommission an executor gracefully by preserving its state before removing it. This requirement is especially important for shuffles. During a shuffle, the Spark executor first writes its own map outputs locally to disk, and then acts as the server for those files when other executors attempt to fetch them. In the event of stragglers, which are tasks that run for much longer than their peers, dynamic allocation may remove an executor before the shuffle completes, in which case the shuffle files written by that executor must be recomputed unnecessarily. The solution for preserving shuffle files is to use an external shuffle service, also introduced in Spark 1.2. This service refers to a long-running process that runs on each node of your cluster independently of your Spark applications and their executors. If the service is enabled, Spark executors will fetch shuffle files from the service instead of from each other. This means any shuffle state written by an executor may continue to be served beyond the executor’s lifetime. In addition to writing shuffle files, executors also cache data either on disk or in memory. When an executor is removed, however, all cached data will no longer be accessible. To mitigate this, by default executors containing cached data are never removed. You can configure this behavior with spark.dynamicAllocation.cachedExecutorIdleTimeout. When set spark.shuffle.service.fetch.rdd.enabled to true, Spark can use ExternalShuffleService for fetching disk persisted RDD blocks. In case of dynamic allocation if this feature is enabled executors having only disk persisted blocks are considered idle after spark.dynamicAllocation.executorIdleTimeout and will be released accordingly. In future releases, the cached data may be preserved through an off-heap storage similar in spirit to how shuffle files are preserved through the external shuffle service. Scheduling Within an Application Inside a given Spark application (SparkContext instance), multiple parallel jobs can run simultaneously if they were submitted from separate threads. By “job”, in this section, we mean a Spark action (e.g. save, collect) and any tasks that need to run to evaluate that action. Spark’s scheduler is fully thread-safe and supports this use case to enable applications that serve multiple requests (e.g. queries for multiple users). By default, Spark’s scheduler runs jobs in FIFO fashion. Each job is divided into “stages” (e.g. map and reduce phases), and the first job gets priority on all available resources while its stages have tasks to launch, then the second job gets priority, etc. If the jobs at the head of the queue don’t need to use the whole cluster, later jobs can start to run right away, but if the jobs at the head of the queue are large, then later jobs may be delayed significantly. Starting in Spark 0.8, it is also possible to configure fair sharing between jobs. Under fair sharing, Spark assigns tasks between jobs in a “round robin” fashion, so that all jobs get a roughly equal share of cluster resources. This means that short jobs submitted while a long job is running can start receiving resources right away and still get good response times, without waiting for the long job to finish. This mode is best for multi-user settings. This feature is disabled by default and available on all coarse-grained cluster managers, i.e. standalone mode, YARN mode, K8s mode. To enable the fair scheduler, simply set the spark.scheduler.mode property to FAIR when configuring a conf = new SparkConf().setMaster(...).setAppName(...) conf.set(\"spark.scheduler.mode\", \"FAIR\") val sc = new SparkContext(conf) Fair Scheduler Pools The fair scheduler also supports grouping jobs into pools, and setting different scheduling options (e.g. weight) for each pool. This can be useful to create a “high-priority” pool for more important jobs, for example, or to group the jobs of each user together and give users equal shares regardless of how many concurrent jobs they have instead of giving jobs equal shares. This approach is modeled after the Hadoop Fair Scheduler. Without any intervention, newly submitted jobs go into a default pool, but jobs’ pools can be set by adding the spark.scheduler.pool “local property” to the SparkContext in the thread that’s submitting them. This is done as follows: // Assuming sc is your SparkContext variable sc.setLocalProperty(\"spark.scheduler.pool\", \"pool1\") After setting this local property, all jobs submitted within this thread (by calls in this thread to RDD.save, count, collect, etc) will use this pool name. The setting is per-thread to make it easy to have a thread run multiple jobs on behalf of the same user. If you’d like to clear the pool that a thread is associated with, simply (\"spark.scheduler.pool\", null) Default Behavior of Pools By default, each pool gets an equal share of the cluster (also equal in share to each job in the default pool), but inside each pool, jobs run in FIFO order. For example, if you create one pool per user, this means that each user will get an equal share of the cluster, and that each user’s queries will run in order instead of later queries taking resources from that user’s earlier ones. Configuring Pool Properties Specific pools’ properties can also be modified through a configuration file. Each pool supports three : This can be FIFO or FAIR, to control whether jobs within the pool queue up behind each other (the default) or share the pool’s resources fairly. controls the pool’s share of the cluster relative to other pools. By default, all pools have a weight of 1. If you give a specific pool a weight of 2, for example, it will get 2x more resources as other active pools. Setting a high weight such as 1000 also makes it possible to implement priority between pools—in essence, the weight-1000 pool will always get to launch tasks first whenever it has jobs active. from an overall weight, each pool can be given a minimum shares (as a number of CPU cores) that the administrator would like it to have. The fair scheduler always attempts to meet all active pools’ minimum shares before redistributing extra resources according to the weights. The minShare property can, therefore, be another way to ensure that a pool can always get up to a certain number of resources (e.g. 10 cores) quickly without giving it a high priority for the rest of the cluster. By default, each pool’s minShare is 0. The pool properties can be set by creating an XML file, similar to conf/fairscheduler.xml.template, and either putting a file named fairscheduler.xml on the classpath, or setting spark.scheduler.allocation.file property in your SparkConf. The file path respects the hadoop configuration and can either be a local file path or HDFS file path. // scheduler file at local conf.set(\"spark.scheduler.allocation.file\", \"file:///path/to/file\") // scheduler file at hdfs conf.set(\"spark.scheduler.allocation.file\", \"hdfs:///path/to/file\") The format of the XML file is simply a <pool> element for each pool, with different elements within it for the various settings. For example: <?xml version=\"1.0\"?> <allocations> <pool name=\"production\"> <schedulingMode>FAIR</schedulingMode> <weight>1</weight> <minShare>2</minShare> </pool> <pool name=\"test\"> <schedulingMode>FIFO</schedulingMode> <weight>2</weight> <minShare>3</minShare> </pool> </allocations> A full example is also available in conf/fairscheduler.xml.template. Note that any pools not configured in the XML file will simply get default values for all settings (scheduling mode FIFO, weight 1, and minShare 0). Scheduling using JDBC Connections To set a Fair Scheduler pool for a JDBC client session, users can set the spark.sql.thriftserver.scheduler.pool spark.sql.thriftserver.scheduler.pool=accounting; Concurrent Jobs in PySpark PySpark, by default, does not support to synchronize PVM threads with JVM threads and launching multiple jobs in multiple PVM threads does not guarantee to launch each job in each corresponding JVM thread. Due to this limitation, it is unable to set a different job group via sc.setJobGroup in a separate PVM thread, which also disallows to cancel the job via sc.cancelJobGroup later. pyspark.InheritableThread is recommended to use together for a PVM thread to inherit the inheritable attributes such as local properties in a JVM thread.\n\nExample:\n```scala\nval conf = new SparkConf().setMaster(...).setAppName(...)\nconf.set(\"spark.scheduler.mode\", \"FAIR\")\nval sc = new SparkContext(conf)\n```\n\nExample:\n```scala\n// Assuming sc is your SparkContext variable\nsc.setLocalProperty(\"spark.scheduler.pool\", \"pool1\")\n```\n\nExample:\n```scala\nsc.setLocalProperty(\"spark.scheduler.pool\", null)\n```\n\nExample:\n```scala\n// scheduler file at local\nconf.set(\"spark.scheduler.allocation.file\", \"file:///path/to/file\")\n// scheduler file at hdfs\nconf.set(\"spark.scheduler.allocation.file\", \"hdfs:///path/to/file\")\n```\n\nExample:\n```xml\n<?xml version=\"1.0\"?>\n<allocations>\n  <pool name=\"production\">\n    <schedulingMode>FAIR</schedulingMode>\n    <weight>1</weight>\n    <minShare>2</minShare>\n  </pool>\n  <pool name=\"test\">\n    <schedulingMode>FIFO</schedulingMode>\n    <weight>2</weight>\n    <minShare>3</minShare>\n  </pool>\n</allocations>\n```\n\nExample:\n```sql\nSET spark.sql.thriftserver.scheduler.pool=accounting;\n```\n\n<|endofdoc|>","metadata":{"transformedAt":"2026-08-18T15:21:43.072Z","totalSectionsIncluded":1,"totalCodeBlocksIncluded":6,"totalLines":51,"estimatedTokens":4385}}26{"id":"doc-integration_with_cloud_infrastructures_spark_4_2-6ec8e5a1","source":"documentation","title":"Integration with Cloud Infrastructures - Spark 4.2.0 Documentation","url":"https://spark.apache.org/docs/latest/cloud-integration.html","text":"Integration with Cloud Infrastructures Introduction Object Stores are Not Real Filesystems Consistency Installation Authenticating Configuring Recommended settings for writing to object stores Parquet I/O Settings ORC I/O Settings Spark Streaming and Object Storage Committing work into cloud storage safely and fast. Hadoop S3A committers Amazon EMRFS S3-optimized committer Azure and Google cloud Intermediate Manifest Committer. IBM Cloud Object Cloud Committers and INSERT OVERWRITE TABLE Further Reading Introduction All major cloud providers offer persistent data storage in object stores. These are not classic “POSIX” file systems. In order to store hundreds of petabytes of data without any single points of failure, object stores replace the classic file system directory tree with a simpler model of object-name => data. To enable remote access, operations on objects are usually offered as (slow) HTTP REST operations. Spark can read and write data in object stores through filesystem connectors implemented in Hadoop or provided by the infrastructure suppliers themselves. These connectors make the object stores look almost like file systems, with directories and files and the classic operations on them such as list, delete and rename. Object Stores are Not Real Filesystems While the stores appear to be filesystems, underneath they are still object stores, and the difference is significant They cannot be used as a direct replacement for a cluster filesystem such as HDFS except where this is explicitly stated. Key differences means by which directories are emulated may make working with them slow. Rename operations may be very slow and, on failure, leave the store in an unknown state. Seeking within a file may require new HTTP calls, hurting performance. How does this affect Spark? Reading and writing data can be significantly slower than working with a normal filesystem. Some directory structures may be very inefficient to scan during query split calculation. The rename-based algorithm by which Spark normally commits work when saving an RDD, DataFrame or Dataset is potentially both slow and unreliable. For these reasons, it is not always safe to use an object store as a direct destination of queries, or as an intermediate store in a chain of queries. Consult the documentation of the object store and its connector to determine which uses are considered safe. Consistency As of 2021, the object stores of Amazon (S3), Google Cloud (GCS) and Microsoft (Azure Storage, ADLS Gen1, ADLS Gen2) are all consistent. This means that as soon as a file is written/updated it can be listed, viewed and opened by other processes -and the latest version will be retrieved. This was a known issue with AWS S3, especially with 404 caching of HEAD requests made before an object was created. Even of the store connectors provide any guarantees as to how their clients cope with objects which are overwritten while a stream is reading them. Do not assume that the old file can be safely read, nor that there is any bounded time period for changes to become visible -or indeed, that the clients will not simply fail if a file being read is overwritten. For this overwriting files where it is known/likely that other clients will be actively reading them. Other object stores are inconsistent This includes OpenStack Swift. Such stores are not always safe to use as a destination of work -consult each store’s specific documentation. Installation With the relevant libraries on the classpath and Spark configured with valid credentials, objects can be read or written by using their URLs as the path to data. For example sparkContext.textFile(\"s3a://landsat-pds/scene_list.gz\") will create an RDD of the file scene_list.gz stored in S3, using the s3a connector. To add the relevant libraries to an application’s classpath, include the hadoop-cloud module and its dependencies. In Maven, add the following to the pom.xml file, assuming spark.version is set to the chosen version of Spark: <dependencyManagement> ... <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-hadoop-cloud_2.13</artifactId> <version>${spark.version}</version> <scope>provided</scope> </dependency> ... </dependencyManagement> Commercial products based on Apache Spark generally directly set up the classpath for talking to cloud infrastructures, in which case this module may not be needed. Authenticating Spark jobs must authenticate with the object stores to access data within them. When Spark is running in a cloud infrastructure, the credentials are usually automatically set up. spark-submit is able to read the AWS_ENDPOINT_URL, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN environment variables and sets the associated authentication options for the s3n and s3a connectors to Amazon S3. In a Hadoop cluster, settings may be set in the core-site.xml file. Authentication details may be manually added to the Spark configuration in spark-defaults.conf Alternatively, they can be programmatically set in the SparkConf instance used to configure the application’s SparkContext. check authentication secrets into source code repositories, especially public ones Consult the Hadoop documentation for the relevant configuration and security options. Configuring Each cloud connector has its own set of configuration parameters, again, consult the relevant documentation. Recommended settings for writing to object stores For object stores whose consistency model means that rename-based commits are safe use the FileOutputCommitter v2 algorithm for performance; v1 for safety. spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version 2 This does less renaming at the end of a job than the “version 1” algorithm. As it still uses rename() to commit files, it is unsafe to use when the object store does not have consistent metadata/listings. The committer can also be set to ignore failures when cleaning up temporary files; this reduces the risk that a transient network problem is escalated into a job true The original v1 commit algorithm renames the output of successful tasks to a job attempt directory, and then renames all the files in that directory into the final destination during the job commit 1 The slow performance of mimicked renames on Amazon S3 makes this algorithm very, very slow. The recommended solution to this is switch to an S3 “Zero Rename” committer (see below). For reference, here are the performance and safety characteristics of different stores and connectors when renaming Connector Directory Rename Safety Rename Performance Amazon S3 s3a Unsafe O(data) Azure Storage wasb Safe O(files) Azure Datalake Gen 2 abfs Safe O(1) Google Cloud Storage gs Mixed O(files) As storing temporary files can run up charges; delete directories called \"_temporary\" on a regular basis. For AWS S3, set a limit on how long multipart uploads can remain outstanding. This avoids incurring bills from incompleted uploads. For Google cloud, directory rename is file-by-file. Consider using the v2 committer and only write code which generates idempotent output -including filenames, as it is no more unsafe than the v1 committer, and faster. Parquet I/O Settings For optimal performance when working with Parquet data use the following false spark.sql.parquet.mergeSchema false spark.sql.parquet.filterPushdown true spark.sql.hive.metastorePartitionPruning true These minimise the amount of data read during queries. ORC I/O Settings For best performance when working with ORC data, use these true spark.sql.orc.splits.include.file.footer true spark.sql.orc.cache.stripe.details.size 10000 spark.sql.hive.metastorePartitionPruning true Again, these minimise the amount of data read during queries. Spark Streaming and Object Storage Spark Streaming can monitor files added to object stores, by creating a FileInputDStream to monitor a path in the store through a call to StreamingContext.textFileStream(). The time to scan for new files is proportional to the number of files under the path, not the number of new files, so it can become a slow operation. The size of the window needs to be set to handle this. Files only appear in an object store once they are completely written; there is no need for a workflow of write-then-rename to ensure that files aren’t picked up while they are still being written. Applications can write straight to the monitored directory. In case of the default checkpoint file manager called FileContextBasedCheckpointFileManager streams should only be checkpointed to a store implementing a fast and atomic rename() operation. Otherwise the checkpointing may be slow and potentially unreliable. On AWS S3 with Hadoop 3.3.1 or later using the S3A connector the abortable stream based checkpoint file manager can be used (by setting the spark.sql.streaming.checkpointFileManagerClass configuration to org.apache.spark.internal.io.cloud.AbortableStreamBasedCheckpointFileManager) which eliminates the slow rename. In this case users must be extra careful to avoid the reuse of the checkpoint location among multiple queries running parallelly as that could lead to corruption of the checkpointing data. Committing work into cloud storage safely and fast. As covered earlier, commit-by-rename is dangerous on any object store which exhibits eventual consistency (example: S3), and often slower than classic filesystem renames. Some object store connectors provide custom committers to commit tasks and jobs without using rename. Hadoop S3A committers In versions of Spark built with Hadoop 3.1 or later, the hadoop-aws JAR contains committers safe to use for S3 storage accessed via the s3a connector. Instead of writing data to a temporary directory on the store for renaming, these committers write the files to the final destination, but do not issue the final POST command to make a large “multi-part” upload visible. Those operations are postponed until the job commit itself. As a result, task and job commit are much faster, and task failures do not affect the result. To switch to the S3A committers, use a version of Spark was built with Hadoop 3.1 or later, and switch the committers through the following options. spark.hadoop.fs.s3a.committer.name directory spark.sql.sources.commitProtocolClass org.apache.spark.internal.io.cloud.PathOutputCommitProtocol spark.sql.parquet.output.committer.class org.apache.spark.internal.io.cloud.BindingParquetOutputCommitter It has been tested with the most common formats supported by Spark. mydataframe.write.format(\"parquet\").save(\"s3a://bucket/destination\") More details on these committers can be found in the latest Hadoop documentation with S3A committer detail covered in Committing work to S3 with the S3A Committers. upon the committer used, in-progress statistics may be under-reported with Hadoop versions before 3.3.1. Amazon EMRFS S3-optimized committer Amazon EMR has its own S3-aware committers for parquet data. For instructions on use, see the EMRFS S3-optimized committer For implementation and performance details, see [“Improve Apache Spark write performance on Apache Parquet formats with the EMRFS S3-optimized committer”](https://aws.amazon.com/blogs/big-data/improve-apache-spark-write-performance-on-apache-parquet-formats-with-the-emrfs-s3-optimized-committer/ Azure and Google cloud Intermediate Manifest Committer. Versions of the hadoop-mapreduce-core JAR shipped after September 2022 (3.3.5 and later) contain a committer optimized for performance and resilience on Azure ADLS Generation 2 and Google Cloud Storage. This committer, the “manifest committer” uses a manifest file to propagate directory listing information from the task committers to the job committer. These manifests can be written atomically, without relying on atomic directory rename, something GCS lacks. The job committer reads these manifests and will rename files from the task output directories directly into the destination directory, in parallel, with optional rate limiting to avoid throttling IO. This delivers performance and scalability on the object stores. It is not critical for job correctness to use this with Azure storage; the classic FileOutputCommitter is safe there -however this new committer scales better for large jobs with deep and wide directory trees. Because Google GCS does not support atomic directory renaming, the manifest committer should be used where available. This committer does support “dynamic partition overwrite” (see below). For details on availability and use of this committer, consult the hadoop documentation for the Hadoop release used. It is not available on Hadoop 3.3.4 or earlier. IBM Cloud Object IBM provide the Stocator output committer for IBM Cloud Object Storage and OpenStack Swift. Source, documentation and releasea can be found at Stocator - Storage Connector for Apache Spark. Cloud Committers and INSERT OVERWRITE TABLE Spark has a feature called “dynamic partition overwrite”; a table can be updated and only those partitions into which new data is added will have their contents replaced. This is used in SQL statements of the form INSERT OVERWRITE TABLE, and when Datasets are written in mode “overwrite” eventDataset.write .mode(\"overwrite\") .partitionBy(\"year\", \"month\") .format(\"parquet\") .save(tablePath) This feature uses file renaming and has specific requirements of both the committer and the committer’s working directory must be in the destination filesystem. The target filesystem must support file rename efficiently. These conditions are not met by the S3A committers and AWS S3 storage. Committers for other cloud stores may support this feature, and declare to spark that they are compatible. If dynamic partition overwrite is required when writing data through a hadoop committer, Spark will always permit this when the original FileOutputCommitter is used. For other committers, after their instantiation, Spark will probe for their declaration of compatibility, and permit the operation if state that they are compatible. If the committer is not compatible, the operation will fail with the error message PathOutputCommitter does not support dynamicPartitionOverwrite Unless there is a compatible committer for the target filesystem, the sole solution is to use a cloud-friendly format for data storage. Further Reading Here is the documentation on the standard connectors both from Apache and the cloud providers. Azure Blob Storage. Azure Blob Filesystem (ABFS) and Azure Datalake Gen 2. Azure Data Lake Gen 1. Amazon S3 Strong Consistency Hadoop-AWS module (Hadoop 3.x). Amazon EMR File System (EMRFS). From Amazon. Using the EMRFS S3-optimized Committer Google Cloud Storage Connector for Spark and Hadoop. From Google. The Azure Blob Filesystem driver (ABFS) IBM Cloud Object Storage connector for Apache , IBM Object Storage. From IBM. Using JindoFS SDK to access Alibaba Cloud OSS. The Cloud Committer problem and hive-compatible solutions Committing work to S3 with the S3A Committers Improve Apache Spark write performance on Apache Parquet formats with the EMRFS S3-optimized committer The Manifest Committer for Azure and Google Cloud Storage A Zero-rename committer. High Performance Object Store Connector for Spark\n\nExample:\n```xml\n<dependencyManagement>\n  ...\n  <dependency>\n    <groupId>org.apache.spark</groupId>\n    <artifactId>spark-hadoop-cloud_2.13</artifactId>\n    <version>${spark.version}</version>\n    <scope>provided</scope>\n  </dependency>\n  ...\n</dependencyManagement>\n```\n\nExample:\n```text\nspark.hadoop.mapreduce.fileoutputcommitter.algorithm.version 2\n```\n\nExample:\n```text\nspark.hadoop.mapreduce.fileoutputcommitter.cleanup-failures.ignored true\n```\n\nExample:\n```text\nspark.hadoop.mapreduce.fileoutputcommitter.algorithm.version 1\n```\n\nExample:\n```text\nspark.hadoop.parquet.enable.summary-metadata false\nspark.sql.parquet.mergeSchema false\nspark.sql.parquet.filterPushdown true\nspark.sql.hive.metastorePartitionPruning true\n```\n\nExample:\n```text\nspark.sql.orc.filterPushdown true\nspark.sql.orc.splits.include.file.footer true\nspark.sql.orc.cache.stripe.details.size 10000\nspark.sql.hive.metastorePartitionPruning true\n```\n\nExample:\n```text\nspark.hadoop.fs.s3a.committer.name directory\nspark.sql.sources.commitProtocolClass org.apache.spark.internal.io.cloud.PathOutputCommitProtocol\nspark.sql.parquet.output.committer.class org.apache.spark.internal.io.cloud.BindingParquetOutputCommitter\n```\n\nExample:\n```text\nmydataframe.write.format(\"parquet\").save(\"s3a://bucket/destination\")\n```\n\nExample:\n```scala\neventDataset.write\n  .mode(\"overwrite\")\n  .partitionBy(\"year\", \"month\")\n  .format(\"parquet\")\n  .save(tablePath)\n```\n\n<|endofdoc|>","metadata":{"transformedAt":"2026-08-18T15:21:43.086Z","totalSectionsIncluded":1,"totalCodeBlocksIncluded":9,"totalLines":69,"estimatedTokens":4167}}27{"id":"doc-monitoring_and_instrumentation_spark_4_2_0_docum-c6ea5ba8","source":"documentation","title":"Monitoring and Instrumentation - Spark 4.2.0 Documentation","url":"https://spark.apache.org/docs/latest/monitoring.html","text":"Monitoring and Instrumentation Web Interfaces Viewing After the Fact Environment Variables Applying compaction on rolling event log files Spark History Server Configuration Options REST API Executor Task Metrics Executor Metrics API Versioning Policy Metrics List of available metrics providers Component instance = Driver Component instance = Executor Source = JVM Source Component instance = applicationMaster Component instance = master Component instance = ApplicationSource Component instance = worker Component instance = shuffleService Advanced Instrumentation There are several ways to monitor Spark UIs, metrics, and external instrumentation. Web Interfaces Every SparkContext launches a Web UI, by default on port 4040, that displays useful information about the application. This list of scheduler stages and tasks A summary of RDD sizes and memory usage Environmental information. Information about the running executors You can access this interface by simply opening http://<driver-node>:4040 in a web browser. If multiple SparkContexts are running on the same host, they will bind to successive ports beginning with 4040 (4041, 4042, etc). Note that this information is only available for the duration of the application by default. To view the web UI after the fact, set spark.eventLog.enabled to true before starting the application. This configures Spark to log Spark events that encode the information displayed in the UI to persisted storage. Viewing After the Fact It is still possible to construct the UI of an application through Spark’s history server, provided that the application’s event logs exist. You can start the history server by /sbin/start-history-server.sh This creates a web interface at http://<server-url>:18080 by default, listing incomplete and completed applications and attempts. When using the file-system provider class (see spark.history.provider below), the base logging directory must be supplied in the spark.history.fs.logDirectory configuration option, and should contain sub-directories that each represents an application’s event logs. The spark jobs themselves must be configured to log events, and to log them to the same shared, writable directory. For example, if the server was configured with a log directory of hdfs://namenode/shared/spark-logs, then the client-side options would true spark.eventLog.dir hdfs://namenode/shared/spark-logs The history server can be configured as Variables Environment VariableMeaning SPARK_DAEMON_MEMORY Memory to allocate to the history server (default: 1g). SPARK_DAEMON_JAVA_OPTS JVM options for the history server (default: none). SPARK_DAEMON_CLASSPATH Classpath for the history server (default: none). SPARK_PUBLIC_DNS The public address for the history server. If this is not set, links to application history may use the internal address of the server, resulting in broken links (default: none). SPARK_HISTORY_OPTS spark.history.* configuration options for the history server (default: none). Applying compaction on rolling event log files A long-running application (e.g. streaming) can bring a huge single event log file which may cost a lot to maintain and also requires a bunch of resource to replay per each update in Spark History Server. Enabling spark.eventLog.rolling.enabled and spark.eventLog.rolling.maxFileSize would let you have rolling event log files instead of single huge event log file which may help some scenarios on its own, but it still doesn’t help you reducing the overall size of logs. Spark History Server can apply compaction on the rolling event log files to reduce the overall size of logs, via setting the configuration spark.history.fs.eventLog.rolling.maxFilesToRetain on the Spark History Server. Details will be described below, but please note in prior that compaction is LOSSY operation. Compaction will discard some events which will be no longer seen on UI - you may want to check which events will be discarded before enabling the option. When the compaction happens, the History Server lists all the available event log files for the application, and considers the event log files having less index than the file with smallest index which will be retained as target of compaction. For example, if the application A has 5 event log files and spark.history.fs.eventLog.rolling.maxFilesToRetain is set to 2, then first 3 log files will be selected to be compacted. Once it selects the target, it analyzes them to figure out which events can be excluded, and rewrites them into one compact file with discarding events which are decided to exclude. The compaction tries to exclude the events which point to the outdated data. As of now, below describes the candidates of events to be for the job which is finished, and related stage/tasks events Events for the executor which is terminated Events for the SQL execution which is finished, and related job/stage/tasks events Once rewriting is done, original log files will be deleted, via best-effort manner. The History Server may not be able to delete the original log files, but it will not affect the operation of the History Server. Please note that Spark History Server may not compact the old event log files if figures out not a lot of space would be reduced during compaction. For streaming query we normally expect compaction will run as each micro-batch will trigger one or more jobs which will be finished shortly, but compaction won’t run in many cases for batch query. Please also note that this is a new feature introduced in Spark 3.0, and may not be completely stable. Under some circumstances, the compaction may exclude more events than you expect, leading some UI issues on History Server for the application. Use it with caution. Spark History Server Configuration Options Security options for the Spark History Server are covered more detail in the Security page. Property Name Default Meaning Since Version spark.history.provider org.apache.spark.deploy.history.FsHistoryProvider Name of the class implementing the application history backend. Currently there is only one implementation, provided by Spark, which looks for application logs stored in the file system. 1.1.0 spark.history.fs.logDirectory file:/tmp/spark-events For the filesystem history provider, the URL to the directory containing application event logs to load. This can be a local file:// path, an HDFS path hdfs://namenode/shared/spark-logs or that of an alternative filesystem supported by the Hadoop APIs. Multiple directories can be specified as a comma-separated list (e.g., hdfs:///logs/prod,s3a://bucket/logs/staging). Directories can be on the same or different filesystems. The directories should be disjoint (not nested within each other). If event log files with the same name exist in different directories, each file is indexed separately based on its source directory. When multiple directories are configured, all existing spark.history.fs.* settings apply globally across all directories (there are no per-directory configurations). 1.1.0 spark.history.fs.logDirectory.names (none) Optional comma-separated list of display names for the log directories specified in spark.history.fs.logDirectory. Names correspond to directories by position. If not set, the full path is shown in the UI. Empty entries fall back to the full path. Duplicate display names are rejected at startup. 4.2.0 spark.history.fs.update.interval 10s The period at which the filesystem history provider checks for new or updated logs in the log directory. A shorter interval detects new applications faster, at the expense of more server load re-reading updated applications. As soon as an update has completed, listings of the completed and incomplete applications will reflect the changes. When multiple log directories are configured, one scan cycle covers all directories sequentially. 1.4.0 spark.history.fs.update.scanDisabledPathPatterns (none) Comma-separated list of regular expressions matched against log directory paths. Directories whose full path matches any pattern will not be scanned periodically (e.g., s3a://.*,gs://.* disables scanning for all S3 and GCS directories). Applications in these directories rely on on-demand loading instead of scanning and will not appear in the listing until accessed by appId. When accessed, accurate metadata is populated immediately. Logs that are never accessed are not subject to the cleaner; use external lifecycle management (e.g., S3 Lifecycle Policies, GCS Object Lifecycle Management) for those. 4.2.0 spark.history.retainedApplications 50 The number of applications to retain UI data for in the cache. If this cap is exceeded, then the oldest applications will be removed from the cache. If an application is not in the cache, it will have to be loaded from disk if it is accessed from the UI. 1.0.0 spark.history.ui.maxApplications Int.MaxValue The number of applications to display on the history summary page. Application UIs are still available by accessing their URLs directly even if they are not displayed on the history summary page. 2.0.1 spark.history.ui.port 18080 The port to which the web interface of the history server binds. 1.0.0 spark.history.kerberos.enabled false Indicates whether the history server should use kerberos to login. This is required if the history server is accessing HDFS files on a secure Hadoop cluster. 1.0.1 spark.history.kerberos.principal (none) When spark.history.kerberos.enabled=true, specifies kerberos principal name for the History Server. 1.0.1 spark.history.kerberos.keytab (none) When spark.history.kerberos.enabled=true, specifies location of the kerberos keytab file for the History Server. 1.0.1 spark.history.fs.cleaner.enabled false Specifies whether the History Server should periodically clean up event logs from storage. 1.4.0 spark.history.fs.cleaner.interval 1d When spark.history.fs.cleaner.enabled=true, specifies how often the filesystem job history cleaner checks for files to delete. Files are deleted if at least one of two conditions holds. First, they're deleted if they're older than spark.history.fs.cleaner.maxAge. They are also deleted if the number of files is more than spark.history.fs.cleaner.maxNum, Spark tries to clean up the completed attempts from the applications based on the order of their oldest attempt time. When multiple log directories are configured, one cleaner cycle covers all directories. 1.4.0 spark.history.fs.cleaner.maxAge 7d When spark.history.fs.cleaner.enabled=true, job history files older than this will be deleted when the filesystem history cleaner runs. When multiple log directories are configured, this age threshold applies to files across all directories. 1.4.0 spark.history.fs.cleaner.maxNum Int.MaxValue When spark.history.fs.cleaner.enabled=true, specifies the maximum number of files in the event log directory. Spark tries to clean up the completed attempt logs to maintain the log directory under this limit. This should be smaller than the underlying file system limit like `dfs.namenode.fs-limits.max-directory-items` in HDFS. When multiple log directories are configured, this limit applies to the total number of files across all directories. The oldest completed attempts are deleted first regardless of which directory they belong to. 3.0.0 spark.history.fs.endEventReparseChunkSize 1m How many bytes to parse at the end of log files looking for the end event. This is used to speed up generation of application listings by skipping unnecessary parts of event log files. It can be disabled by setting this config to 0. 2.4.0 spark.history.fs.inProgressOptimization.enabled true Enable optimized handling of in-progress logs. This option may leave finished applications that fail to rename their event logs listed as in-progress. 2.4.0 spark.history.fs.driverlog.cleaner.enabled spark.history.fs.cleaner.enabled Specifies whether the History Server should periodically clean up driver logs from storage. 3.0.0 spark.history.fs.driverlog.cleaner.interval spark.history.fs.cleaner.interval When spark.history.fs.driverlog.cleaner.enabled=true, specifies how often the filesystem driver log cleaner checks for files to delete. Files are only deleted if they are older than spark.history.fs.driverlog.cleaner.maxAge 3.0.0 spark.history.fs.driverlog.cleaner.maxAge spark.history.fs.cleaner.maxAge When spark.history.fs.driverlog.cleaner.enabled=true, driver log files older than this will be deleted when the driver log cleaner runs. 3.0.0 spark.history.fs.numReplayThreads 25% of available cores Number of threads that will be used by history server to process event logs. When multiple log directories are configured, the thread pool is shared across all directories. 2.0.0 spark.history.fs.numCompactThreads 25% of available cores Number of threads that will be used by history server to compact event logs. When multiple log directories are configured, the thread pool is shared across all directories. 4.1.0 spark.history.store.maxDiskUsage 10g Maximum disk usage for the local directory where the cache application history information are stored. 2.3.0 spark.history.store.path (none) Local directory where to cache application history data. If set, the history server will store application data on disk instead of keeping it in memory. The data written to disk will be re-used in the event of a history server restart. 2.3.0 spark.history.store.serializer JSON Serializer for writing/reading in-memory UI objects to/from disk-based KV Store; JSON or PROTOBUF. JSON serializer is the only choice before Spark 3.4.0, thus it is the default value. PROTOBUF serializer is fast and compact, compared to the JSON serializer. 3.4.0 spark.history.custom.executor.log.url (none) Specifies custom spark executor log URL for supporting external log service instead of using cluster managers' application log URLs in the history server. Spark will support some path variables via patterns which can vary on cluster manager. Please check the documentation for your cluster manager to see which patterns are supported, if any. This configuration has no effect on a live application, it only affects the history server. For now, only YARN mode supports this configuration 3.0.0 spark.history.custom.executor.log.url.applyIncompleteApplication true Specifies whether to apply custom spark executor log URL to incomplete applications as well. If executor logs for running applications should be provided as origin log URLs, set this to `false`. Please note that incomplete applications may include applications which didn't shutdown gracefully. Even this is set to `true`, this configuration has no effect on a live application, it only affects the history server. 3.0.0 spark.history.fs.eventLog.rolling.maxFilesToRetain Int.MaxValue The maximum number of event log files which will be retained as non-compacted. By default, all event log files will be retained. The lowest value is 1 for technical reason. Please read the section of \"Applying compaction of old event log files\" for more details. When multiple log directories are configured, this setting applies independently to each directory. 3.0.0 spark.history.fs.eventLog.rolling.onDemandLoadEnabled true Whether to look up rolling event log locations on demand manner before listing files. 4.1.0 spark.history.store.hybridStore.enabled false Whether to use HybridStore as the store when parsing event logs. HybridStore will first write data to an in-memory store and having a background thread that dumps data to a disk store after the writing to in-memory store is completed. 3.1.0 spark.history.store.hybridStore.maxMemoryUsage 2g Maximum memory space that can be used to create HybridStore. The HybridStore co-uses the heap memory, so the heap memory should be increased through the memory option for SHS if the HybridStore is enabled. 3.1.0 spark.history.store.hybridStore.diskBackend ROCKSDB Specifies a disk-based store used in hybrid store; ROCKSDB or LEVELDB (deprecated). 3.3.0 spark.history.fs.update.batchSize Int.MaxValue Specifies the batch size for updating new eventlog files. This controls each scan process to be completed within a reasonable time, and such prevent the initial scan from running too long and blocking new eventlog files to be scanned in time in large environments. When multiple log directories are configured, this batch size applies independently to each directory's scan. 3.4.0 Note that in all of these UIs, the tables are sortable by clicking their headers, making it easy to identify slow tasks, data skew, etc. Note The history server displays both completed and incomplete Spark jobs. If an application makes multiple attempts after failures, the failed attempts will be displayed, as well as any ongoing incomplete attempt or the final successful attempt. Incomplete applications are only updated intermittently. The time between updates is defined by the interval between checks for changed files (spark.history.fs.update.interval). On larger clusters, the update interval may be set to large values. The way to view a running application is actually to view its own web UI. Applications which exited without registering themselves as completed will be listed as incomplete —even though they are no longer running. This can happen if an application crashes. One way to signal the completion of a Spark job is to stop the Spark Context explicitly (sc.stop()), or in Python using the with SparkContext() as to handle the Spark Context setup and tear down. REST API In addition to viewing the metrics in the UI, they are also available as JSON. This gives developers an easy way to create new visualizations and monitoring tools for Spark. The JSON is available for both running applications, and in the history server. The endpoints are mounted at /api/v1. For example, for the history server, they would typically be accessible at http://<server-url>:18080/api/v1, and for a running application, at http://localhost:4040/api/v1. In the API, an application is referenced by its application ID, [app-id]. When running on YARN, each application may have multiple attempts, but there are attempt IDs only for applications in cluster mode, not applications in client mode. Applications in YARN cluster mode can be identified by their [attempt-id]. In the API listed below, when running in YARN cluster mode, [app-id] will actually be [base-app-id]/[attempt-id], where [base-app-id] is the YARN application ID. EndpointMeaning /applications A list of all applications. ?status=[completed|running] list only applications in the chosen state. ?minDate=[date] earliest start date/time to list. ?maxDate=[date] latest start date/time to list. ?minEndDate=[date] earliest end date/time to list. ?maxEndDate=[date] latest end date/time to list. ?limit=[limit] limits the number of applications listed. Examples: ?minDate=2015-02-10 ?minDate=2015-02-03T16:42:40.000GMT ?maxDate=2015-02-11T20:41:30.000GMT ?minEndDate=2015-02-12 ?minEndDate=2015-02-12T09:15:10.000GMT ?maxEndDate=2015-02-14T16:30:45.000GMT ?limit=10 /applications/[app-id]/jobs A list of all jobs for a given application. ?status=[running|succeeded|failed|unknown] list only jobs in the specific state. /applications/[app-id]/jobs/[job-id] Details for the given job. /applications/[app-id]/stages A list of all stages for a given application. ?status=[active|complete|pending|failed] list only stages in the given state. ?details=true lists all stages with the task data. ?taskStatus=[RUNNING|SUCCESS|FAILED|KILLED|PENDING] lists only those tasks with the specified task status. Query parameter taskStatus takes effect only when details=true. This also supports multiple taskStatus such as ?details=true&taskStatus=SUCCESS&taskStatus=FAILED which will return all tasks matching any of specified task status. ?withSummaries=true lists stages with task metrics distribution and executor metrics distribution. ?quantiles=0.0,0.25,0.5,0.75,1.0 summarize the metrics with the given quantiles. Query parameter quantiles takes effect only when withSummaries=true. Default value is 0.0,0.25,0.5,0.75,1.0. /applications/[app-id]/stages/[stage-id] A list of all attempts for the given stage. ?details=true lists all attempts with the task data for the given stage. ?taskStatus=[RUNNING|SUCCESS|FAILED|KILLED|PENDING] lists only those tasks with the specified task status. Query parameter taskStatus takes effect only when details=true. This also supports multiple taskStatus such as ?details=true&taskStatus=SUCCESS&taskStatus=FAILED which will return all tasks matching any of specified task status. ?withSummaries=true lists task metrics distribution and executor metrics distribution of each attempt. ?quantiles=0.0,0.25,0.5,0.75,1.0 summarize the metrics with the given quantiles. Query parameter quantiles takes effect only when withSummaries=true. Default value is 0.0,0.25,0.5,0.75,1.0. Example: ?details=true ?details=true&taskStatus=RUNNING ?withSummaries=true ?details=true&withSummaries=true&quantiles=0.01,0.5,0.99 /applications/[app-id]/stages/[stage-id]/[stage-attempt-id] Details for the given stage attempt. ?details=true lists all task data for the given stage attempt. ?taskStatus=[RUNNING|SUCCESS|FAILED|KILLED|PENDING] lists only those tasks with the specified task status. Query parameter taskStatus takes effect only when details=true. This also supports multiple taskStatus such as ?details=true&taskStatus=SUCCESS&taskStatus=FAILED which will return all tasks matching any of specified task status. ?withSummaries=true lists task metrics distribution and executor metrics distribution for the given stage attempt. ?quantiles=0.0,0.25,0.5,0.75,1.0 summarize the metrics with the given quantiles. Query parameter quantiles takes effect only when withSummaries=true. Default value is 0.0,0.25,0.5,0.75,1.0. Example: ?details=true ?details=true&taskStatus=RUNNING ?withSummaries=true ?details=true&withSummaries=true&quantiles=0.01,0.5,0.99 /applications/[app-id]/stages/[stage-id]/[stage-attempt-id]/taskSummary Summary metrics of all tasks in the given stage attempt. ?quantiles summarize the metrics with the given quantiles. Example: ?quantiles=0.01,0.5,0.99 /applications/[app-id]/stages/[stage-id]/[stage-attempt-id]/taskList A list of all tasks for the given stage attempt. ?offset=[offset]&length=[len] list tasks in the given range. ?sortBy=[runtime|-runtime] sort the tasks. ?status=[running|success|killed|failed|unknown] list only tasks in the state. Example: ?offset=10&length=50&sortBy=runtime&status=running /applications/[app-id]/executors A list of all active executors for the given application. /applications/[app-id]/executors/[executor-id]/threads Stack traces of all the threads running within the given active executor. Not available via the history server. /applications/[app-id]/allexecutors A list of all(active and dead) executors for the given application. /applications/[app-id]/storage/rdd A list of stored RDDs for the given application. /applications/[app-id]/storage/rdd/[rdd-id] Details for the storage status of a given RDD. /applications/[base-app-id]/logs Download the event logs for all attempts of the given application as files within a zip file. /applications/[base-app-id]/[attempt-id]/logs Download the event logs for a specific application attempt as a zip file. /applications/[app-id]/streaming/statistics Statistics for the streaming context. /applications/[app-id]/streaming/receivers A list of all streaming receivers. /applications/[app-id]/streaming/receivers/[stream-id] Details of the given receiver. /applications/[app-id]/streaming/batches A list of all retained batches. /applications/[app-id]/streaming/batches/[batch-id] Details of the given batch. /applications/[app-id]/streaming/batches/[batch-id]/operations A list of all output operations of the given batch. /applications/[app-id]/streaming/batches/[batch-id]/operations/[outputOp-id] Details of the given operation and given batch. /applications/[app-id]/sql A list of all queries for a given application. ?details=[true (default) | false] lists/hides details of Spark plan nodes. ?planDescription=[true (default) | false] enables/disables Physical planDescription on demand when Physical Plan size is high. ?offset=[offset]&length=[len] lists queries in the given range. /applications/[app-id]/sql/[execution-id] Details for the given query. ?details=[true (default) | false] lists/hides metric details in addition to given query details. ?planDescription=[true (default) | false] enables/disables Physical planDescription on demand for the given query when Physical Plan size is high. /applications/[app-id]/environment Environment details of the given application. /version Get the current spark version. The number of jobs and stages which can be retrieved is constrained by the same retention mechanism of the standalone Spark UI; \"spark.ui.retainedJobs\" defines the threshold value triggering garbage collection on jobs, and spark.ui.retainedStages that for stages. Note that the garbage collection takes place on is possible to retrieve more entries by increasing these values and restarting the history server. Executor Task Metrics The REST API exposes the values of the Task Metrics collected by Spark executors with the granularity of task execution. The metrics can be used for performance troubleshooting and workload characterization. A list of the available metrics, with a short Executor Task Metric name Short description executorRunTime Elapsed time the executor spent running this task. This includes time fetching shuffle data. The value is expressed in milliseconds. executorCpuTime CPU time the executor spent running this task. This includes time fetching shuffle data. The value is expressed in nanoseconds. executorDeserializeTime Elapsed time spent to deserialize this task. The value is expressed in milliseconds. executorDeserializeCpuTime CPU time taken on the executor to deserialize this task. The value is expressed in nanoseconds. resultSize The number of bytes this task transmitted back to the driver as the TaskResult. jvmGCTime Elapsed time the JVM spent in garbage collection while executing this task. The value is expressed in milliseconds. ConcurrentGCCount This metric returns the total number of collections that have occurred. It only applies when the Java Garbage collector is G1 Concurrent GC. ConcurrentGCTime This metric returns the approximate accumulated collection elapsed time in milliseconds. It only applies when the Java Garbage collector is G1 Concurrent GC. resultSerializationTime Elapsed time spent serializing the task result. The value is expressed in milliseconds. memoryBytesSpilled The number of in-memory bytes spilled by this task. diskBytesSpilled The number of on-disk bytes spilled by this task. peakExecutionMemory Peak memory used by internal data structures created during shuffles, aggregations and joins. The value of this accumulator should be approximately the sum of the peak sizes across all such data structures created in this task. For SQL jobs, this only tracks all unsafe operators and ExternalSort. inputMetrics.* Metrics related to reading data from org.apache.spark.rdd.HadoopRDD or from persisted data. . This value is then expanded appropriately by Spark and is used as the root namespace of the metrics system. Non-driver and executor metrics are never prefixed with spark.app.id, nor does the spark.metrics.namespace property have any such affect on such metrics. Spark’s metrics are decoupled into different instances corresponding to Spark components. Within each instance, you can configure a set of sinks to which metrics are reported. The following instances are currently : The Spark standalone master process. component within the master which reports on various applications. Spark standalone worker process. Spark executor. Spark driver process (the process in which your SparkContext is created). Spark shuffle service. Spark ApplicationMaster when running on YARN. Each instance can report to zero or more sinks. Sinks are contained in the org.apache.spark.metrics.sink : Logs metrics information to the console. metrics data to CSV files at regular intervals. metrics for viewing in a JMX console. a servlet within the existing Spark UI to serve metrics data as JSON data. PrometheusServlet: (Experimental) Adds a servlet within the existing Spark UI to serve metrics data in Prometheus format. metrics to a Graphite node. metrics to slf4j as log entries. metrics to a StatsD node. The Prometheus Servlet mirrors the JSON data exposed by the Metrics Servlet and the REST API, but in a time-series format. The following are the equivalent Prometheus Servlet endpoints. Component Port JSON End Point Prometheus End Point Master 8080 /metrics/master/json/ /metrics/master/prometheus/ Master 8080 /metrics/applications/json/ /metrics/applications/prometheus/ Worker 8081 /metrics/json/ /metrics/prometheus/ Driver 4040 /metrics/json/ /metrics/prometheus/ Driver 4040 /api/v1/applications/{id}/executors/ /metrics/executors/prometheus/ Spark also supports a Ganglia sink which is not included in the default build due to licensing : Sends metrics to a Ganglia node or multicast group. To install the GangliaSink you’ll need to perform a custom build of Spark. Note that by embedding this library you will include LGPL-licensed code in your Spark package. For sbt users, set the SPARK_GANGLIA_LGPL environment variable before building. For Maven users, enable the -Pspark-ganglia-lgpl profile. In addition to modifying the cluster’s Spark build user applications will need to link to the spark-ganglia-lgpl artifact. The syntax of the metrics configuration file and the parameters available for each sink are defined in an example configuration file, $SPARK_HOME/conf/metrics.properties.template. When using Spark configuration parameters instead of the metrics configuration file, the relevant parameter names are composed by the prefix spark.metrics.conf. followed by the configuration details, i.e. the parameters take the following [instance|*].sink.[sink_name].[parameter_name]. This example shows a list of Spark configuration parameters for a Graphite sink: \"spark.metrics.conf.*.sink.graphite.class\"=\"org.apache.spark.metrics.sink.GraphiteSink\" \"spark.metrics.conf.*.sink.graphite.host\"=\"graphiteEndPoint_hostName>\" \"spark.metrics.conf.*.sink.graphite.port\"=<graphite_listening_port> \"spark.metrics.conf.*.sink.graphite.period\"=10 \"spark.metrics.conf.*.sink.graphite.unit\"=seconds \"spark.metrics.conf.*.sink.graphite.prefix\"=\"optional_prefix\" \"spark.metrics.conf.*.sink.graphite.regex\"=\"optional_regex_to_send_matching_metrics\" Default values of the Spark metrics configuration are as follows: \"*.sink.servlet.class\" = \"org.apache.spark.metrics.sink.MetricsServlet\" \"*.sink.servlet.path\" = \"/metrics/json\" \"master.sink.servlet.path\" = \"/metrics/master/json\" \"applications.sink.servlet.path\" = \"/metrics/applications/json\" Additional sources can be configured using the metrics configuration file or the configuration parameter spark.metrics.conf.[component_name].source.jvm.class=[source_name]. At present the JVM source is the only available optional source. For example the following configuration parameter activates the JVM source: \"spark.metrics.conf.*.source.jvm.class\"=\"org.apache.spark.metrics.source.JvmSource\" List of available metrics providers Metrics used by Spark are of multiple , counter, histogram, meter and timer, see Dropwizard library documentation for details. The following list of components and metrics reports the name and some details about the available metrics, grouped per component instance and source namespace. The most common time of metrics used in Spark instrumentation are gauges and counters. Counters can be recognized as they have the .count suffix. Timers, meters and histograms are annotated in the list, the rest of the list elements are metrics of type gauge. The large majority of metrics are active as soon as their parent component instance is configured, some metrics require also to be enabled via an additional configuration parameter, the details are reported in the list. Component instance = Driver This is the component with the largest amount of instrumented metrics namespace=BlockManager disk.diskSpaceUsed_MB memory.maxMem_MB memory.maxOffHeapMem_MB memory.maxOnHeapMem_MB memory.memUsed_MB memory.offHeapMemUsed_MB memory.onHeapMemUsed_MB memory.remainingMem_MB memory.remainingOffHeapMem_MB memory.remainingOnHeapMem_MB namespace=HiveExternalCatalog metrics are conditional to a configuration (default is true) fileCacheHits.count filesDiscovered.count hiveClientCalls.count parallelListingJobCount.count partitionsFetched.count namespace=CodeGenerator metrics are conditional to a configuration (default is true) compilationTime (histogram) generatedClassSize (histogram) generatedMethodSize (histogram) sourceCodeSize (histogram) namespace=DAGScheduler job.activeJobs job.allJobs messageProcessingTime (timer) stage.failedStages stage.runningStages stage.waitingStages namespace=LiveListenerBus listenerProcessingTime.org.apache.spark.HeartbeatReceiver (timer) listenerProcessingTime.org.apache.spark.scheduler.EventLoggingListener (timer) listenerProcessingTime.org.apache.spark.status.AppStatusListener (timer) numEventsPosted.count queue.appStatus.listenerProcessingTime (timer) queue.appStatus.numDroppedEvents.count queue.appStatus.size queue.eventLog.listenerProcessingTime (timer) queue.eventLog.numDroppedEvents.count queue.eventLog.size queue.executorManagement.listenerProcessingTime (timer) namespace=appStatus (all metrics of type=counter) in Spark 3.0. Conditional to a configuration (default is true) stages.failedStages.count stages.skippedStages.count stages.completedStages.count tasks.blackListedExecutors.count // deprecated use excludedExecutors instead tasks.excludedExecutors.count tasks.completedTasks.count tasks.failedTasks.count tasks.killedTasks.count tasks.skippedTasks.count tasks.unblackListedExecutors.count // deprecated use unexcludedExecutors instead tasks.unexcludedExecutors.count jobs.succeededJobs jobs.failedJobs jobDuration namespace=AccumulatorSource sources to attach accumulators to metric system DoubleAccumulatorSource LongAccumulatorSource namespace=spark.streaming applies to Spark Structured Streaming only. Conditional to a configuration =true (default is false) eventTime-watermark inputRate-total latency processingRate-total states-rowsTotal states-usedBytes namespace=JVMCPU jvmCpuTime namespace=executor metrics are available in the driver in local mode only. A full list of available metrics in this namespace can be found in the corresponding entry for the Executor component instance. namespace=ExecutorMetrics metrics are conditional to a configuration (default is true) This source contains memory-related metrics. A full list of available metrics in this namespace can be found in the corresponding entry for the Executor component instance. namespace=ExecutorAllocationManager metrics are only emitted when using dynamic allocation. Conditional to a configuration parameter spark.dynamicAllocation.enabled (default is false) executors.numberExecutorsToAdd executors.numberExecutorsPendingToRemove executors.numberAllExecutors executors.numberTargetExecutors executors.numberMaxNeededExecutors executors.numberDecommissioningExecutors executors.numberExecutorsGracefullyDecommissioned.count executors.numberExecutorsDecommissionUnfinished.count executors.numberExecutorsExitedUnexpectedly.count executors.numberExecutorsKilledByDriver.count namespace=plugin.<Plugin Class Name> Optional namespace(s). Metrics in this namespace are defined by user-supplied code, and configured using the Spark plugin API. See “Advanced Instrumentation” below for how to load custom plugins into Spark. Component instance = Executor These metrics are exposed by Spark executors. namespace=executor (metrics are of type counter or gauge) (default: file,hdfs) determines the exposed file system metrics. bytesRead.count bytesWritten.count cpuTime.count deserializeCpuTime.count deserializeTime.count diskBytesSpilled.count filesystem.file.largeRead_ops filesystem.file.read_bytes filesystem.file.read_ops filesystem.file.write_bytes filesystem.file.write_ops filesystem.hdfs.largeRead_ops filesystem.hdfs.read_bytes filesystem.hdfs.read_ops filesystem.hdfs.write_bytes filesystem.hdfs.write_ops jvmGCTime.count memoryBytesSpilled.count recordsRead.count recordsWritten.count resultSerializationTime.count resultSize.count runTime.count shuffleBytesWritten.count shuffleFetchWaitTime.count shuffleLocalBlocksFetched.count shuffleLocalBytesRead.count shuffleRecordsRead.count shuffleRecordsWritten.count shuffleRemoteBlocksFetched.count shuffleRemoteBytesRead.count shuffleRemoteBytesReadToDisk.count shuffleTotalBytesRead.count shuffleWriteTime.count Metrics related to push-based shuffleMergedFetchFallbackCount shuffleMergedRemoteBlocksFetched shuffleMergedLocalBlocksFetched shuffleMergedRemoteChunksFetched shuffleMergedLocalChunksFetched shuffleMergedRemoteBytesRead shuffleMergedLocalBytesRead shuffleRemoteReqsDuration shuffleMergedRemoteReqsDuration succeededTasks.count threadpool.activeTasks threadpool.completeTasks threadpool.currentPool_size threadpool.maxPool_size threadpool.startedTasks namespace=ExecutorMetrics metrics are conditional to a configuration (default value is true) ExecutorMetrics are updated as part of heartbeat processes scheduled for the executors and for the driver at regular (default value is 10 seconds) An optional faster polling mechanism is available for executor memory metrics, it can be activated by setting a polling interval (in milliseconds) using the configuration parameter spark.executor.metrics.pollingInterval JVMHeapMemory JVMOffHeapMemory OnHeapExecutionMemory OnHeapStorageMemory OnHeapUnifiedMemory OffHeapExecutionMemory OffHeapStorageMemory OffHeapUnifiedMemory DirectPoolMemory MappedPoolMemory MinorGCCount MinorGCTime MajorGCCount MajorGCTime “ProcessTree*” metric ProcessTreeJVMRSSMemory ProcessTreePythonVMemory ProcessTreePythonRSSMemory ProcessTreeOtherVMemory ProcessTreeOtherRSSMemory note: “ProcessTree” metrics are collected only under certain conditions. The conditions are the logical AND of the following: /proc filesystem exists, spark.executor.processTreeMetrics.enabled=true. “ProcessTree” metrics report 0 when those conditions are not met. namespace=JVMCPU jvmCpuTime namespace=NettyBlockTransfer shuffle-client.usedDirectMemory shuffle-client.usedHeapMemory shuffle-server.usedDirectMemory shuffle-server.usedHeapMemory namespace=HiveExternalCatalog metrics are conditional to a configuration (default is true) fileCacheHits.count filesDiscovered.count hiveClientCalls.count parallelListingJobCount.count partitionsFetched.count namespace=CodeGenerator metrics are conditional to a configuration (default is true) compilationTime (histogram) generatedClassSize (histogram) generatedMethodSize (histogram) sourceCodeSize (histogram) namespace=plugin.<Plugin Class Name> Optional namespace(s). Metrics in this namespace are defined by user-supplied code, and configured using the Spark plugin API. See “Advanced Instrumentation” below for how to load custom plugins into Spark. Source = JVM Source this source by setting the relevant metrics.properties file entry or the configuration *.source.jvm.class=org.apache.spark.metrics.source.JvmSource These metrics are conditional to a configuration (default is true) This source is available for driver and executor instances and is also available for other instances. This source provides information on JVM metrics using the Dropwizard/Codahale Metric Sets for JVM instrumentation and in particular the metric sets BufferPoolMetricSet, GarbageCollectorMetricSet and MemoryUsageGaugeSet. Component instance = applicationMaster when running on YARN numContainersPendingAllocate numExecutorsFailed numExecutorsRunning numLocalityAwareTasks numReleasedContainers Component instance = master when running in Spark standalone as master workers aliveWorkers apps waitingApps Component instance = ApplicationSource when running in Spark standalone as master status runtime_ms cores Component instance = worker when running in Spark standalone as worker executors coresUsed memUsed_MB coresFree memFree_MB Component instance = shuffleService to the shuffle service blockTransferRate (meter) - rate of blocks being transferred blockTransferMessageRate (meter) - rate of block transfer messages, i.e. if batch fetches are enabled, this represents number of batches rather than number of blocks blockTransferRateBytes (meter) blockTransferAvgSize_1min (gauge - 1-minute moving average) numActiveConnections.count numRegisteredConnections.count numCaughtExceptions.count openBlockRequestLatencyMillis (timer) registerExecutorRequestLatencyMillis (timer) fetchMergedBlocksMetaLatencyMillis (timer) finalizeShuffleMergeLatencyMillis (timer) registeredExecutorsSize shuffle-server.usedDirectMemory shuffle-server.usedHeapMemory metrics below apply when the server side configuration spark.shuffle.push.server.mergedShuffleFileManagerImpl is set to org.apache.spark.network.shuffle.MergedShuffleFileManager for Push-Based Shuffle blockBytesWritten - size of the pushed block data written to file in bytes blockAppendCollisions - number of shuffle push blocks collided in shuffle services as another block for the same reduce partition were being written lateBlockPushes - number of shuffle push blocks that are received in shuffle service after the specific shuffle merge has been finalized deferredBlocks - number of the current deferred block parts buffered in memory deferredBlockBytes - size of the current deferred block parts buffered in memory staleBlockPushes - number of stale shuffle block push requests ignoredBlockBytes - size of the pushed block data that was transferred to ESS, but ignored. The pushed block data are considered as ignored it was received after the shuffle was finalized; 2. when a push request is for a duplicate block; 3. ESS was unable to write the block. Advanced Instrumentation Several external tools can be used to help profile the performance of Spark monitoring tools, such as Ganglia, can provide insight into overall cluster utilization and resource bottlenecks. For instance, a Ganglia dashboard can quickly reveal whether a particular workload is disk bound, network bound, or CPU bound. OS profiling tools such as dstat, iostat, and iotop can provide fine-grained profiling on individual nodes. JVM utilities such as jstack for providing stack traces, jmap for creating heap-dumps, jstat for reporting time-series statistics and jconsole for visually exploring various JVM properties are useful for those comfortable with JVM internals. Spark also provides a plugin API so that custom instrumentation code can be added to Spark applications. There are two configuration keys available for loading plugins into spark.plugins.defaultList Both take a comma-separated list of class names that implement the org.apache.spark.api.plugin.SparkPlugin interface. The two names exist so that it’s possible for one list to be placed in the Spark default config file, allowing users to easily add other plugins from the command line without overwriting the config file’s list. Duplicate plugins are ignored.\n\nExample:\n```text\n./sbin/start-history-server.sh\n```\n\nExample:\n```text\nspark.eventLog.enabled true\nspark.eventLog.dir hdfs://namenode/shared/spark-logs\n```\n\nExample:\n```text\n\"spark.metrics.conf.*.sink.graphite.class\"=\"org.apache.spark.metrics.sink.GraphiteSink\"\n\"spark.metrics.conf.*.sink.graphite.host\"=\"graphiteEndPoint_hostName>\"\n\"spark.metrics.conf.*.sink.graphite.port\"=<graphite_listening_port>\n\"spark.metrics.conf.*.sink.graphite.period\"=10\n\"spark.metrics.conf.*.sink.graphite.unit\"=seconds\n\"spark.metrics.conf.*.sink.graphite.prefix\"=\"optional_prefix\"\n\"spark.metrics.conf.*.sink.graphite.regex\"=\"optional_regex_to_send_matching_metrics\"\n```\n\nExample:\n```text\n\"*.sink.servlet.class\" = \"org.apache.spark.metrics.sink.MetricsServlet\"\n\"*.sink.servlet.path\" = \"/metrics/json\"\n\"master.sink.servlet.path\" = \"/metrics/master/json\"\n\"applications.sink.servlet.path\" = \"/metrics/applications/json\"\n```\n\n<|endofdoc|>","metadata":{"transformedAt":"2026-08-18T15:21:43.097Z","totalSectionsIncluded":1,"totalCodeBlocksIncluded":4,"totalLines":33,"estimatedTokens":11071}}28{"id":"doc-spark_sql_cli_spark_4_2_0_documentation-c7b0e29f","source":"documentation","title":"Spark SQL CLI - Spark 4.2.0 Documentation","url":"https://spark.apache.org/docs/latest/sql-distributed-sql-engine-spark-sql-cli.html","text":"Spark SQL Guide Getting Started Data Sources Data Source V2 Performance Tuning Distributed SQL Engine Running the Thrift JDBC/ODBC server Running the Spark SQL CLI PySpark Usage Guide for Pandas with Apache Arrow Migration Guide SQL Reference Error Conditions Spark SQL CLI Spark SQL Command Line Options The hiverc File Path interpretation Supported comment types Spark SQL CLI Interactive Shell Commands Examples The Spark SQL CLI is a convenient interactive command tool to run the Hive metastore service and execute SQL queries input from the command line. Note that the Spark SQL CLI cannot talk to the Thrift JDBC server. To start the Spark SQL CLI, run the following in the Spark /bin/spark-sql Configuration of Hive is done by placing your hive-site.xml, core-site.xml and hdfs-site.xml files in conf/. Spark SQL Command Line Options You may run ./bin/spark-sql --help for a complete list of all available options. CLI ,--define <key=value> Variable substitution to apply to Hive commands. e.g. -d A=B or --define A=B --database <databasename> Specify the database to use -e <quoted-query-string> SQL from command line -f <filename> SQL from files -H,--help Print help information --hiveconf <property=value> Use value for given property --hivevar <key=value> Variable substitution to apply to Hive commands. e.g. --hivevar A=B -i <filename> Initialization SQL file -S,--silent Silent mode in interactive shell -v,--verbose Verbose mode (echo executed SQL to the console) The hiverc File When invoked without the -i, the Spark SQL CLI will attempt to load $HIVE_HOME/bin/.hiverc and $HOME/.hiverc as initialization files. Path interpretation Spark SQL CLI supports running SQL from initialization script file(-i) or normal SQL file(-f), If path url don’t have a scheme component, the path will be handled as local file. For example: /path/to/spark-sql-cli.sql equals to file:///path/to/spark-sql-cli.sql. User also can use Hadoop supported filesystems such as s3://<mys3bucket>/path/to/spark-sql-cli.sql or hdfs://<namenode>:<port>/path/to/spark-sql-cli.sql. Supported comment types CommentExample simple comment -- This is a simple comment. SELECT 1; bracketed comment /* This is a bracketed comment. */ SELECT 1; nested bracketed comment /* This is a /* nested bracketed comment*/ .*/ SELECT 1; Spark SQL CLI Interactive Shell Commands When ./bin/spark-sql is run without either the -e or -f option, it enters interactive shell mode. Use ; (semicolon) to terminate commands. CLI use ; to terminate commands only when it’s at the end of line, and it’s not escaped by \\\\;. ; is the only way to terminate commands. If the user types SELECT 1 and presses enter, the console will just wait for input. If the user types multiple commands in one line like SELECT 1; SELECT 2;, the commands SELECT 1 and SELECT 2 will be executed separately. If ; appears within a SQL statement (not the end of the line), then it has no special This is a ; comment SELECT ';' as a; This is just a comment line followed by a SQL query which returns a string literal. /* This is a comment contains ; */ SELECT 1; However, if ‘;’ is the end of the line, it terminates the SQL statement. The example above will be terminated into /* This is a comment contains and */ SELECT 1, Spark will submit these two commands separated and throw parser error (unclosed bracketed comment and Syntax error at or near '*/'). CommandDescription quit or exit Exits the interactive shell. !<command> Executes a shell command from the Spark SQL CLI shell. dfs <HDFS dfs command> Executes a HDFS dfs command from the Spark SQL CLI shell. <query string> Executes a Spark SQL query and prints results to standard output. source <filepath> Executes a script file inside the CLI. Examples Example of running a query from the command /bin/spark-sql -e 'SELECT COL FROM TBL' Example of setting Hive configuration /bin/spark-sql -e 'SELECT COL FROM TBL' --hiveconf hive.exec.scratchdir=/home/my/hive_scratch Example of setting Hive configuration variables and using it in the SQL /bin/spark-sql -e 'SELECT ${hiveconf:aaa}' --hiveconf aaa=bbb --hiveconf hive.exec.scratchdir=/home/my/hive_scratch spark-sql> SELECT ${aaa}; bbb Example of setting Hive variables /bin/spark-sql --hivevar aaa=bbb --define ccc=ddd spark-sql> SELECT ${aaa}, ${ccc}; bbb ddd Example of dumping data out from a query into a file using silent /bin/spark-sql -S -e 'SELECT COL FROM TBL' > result.txt Example of running a script /bin/spark-sql -f /path/to/spark-sql-script.sql Example of running an initialization script before entering interactive /bin/spark-sql -i /path/to/spark-sql-init.sql Example of entering interactive /bin/spark-sql spark-sql> SELECT 1; 1 spark-sql> -- This is a simple comment. spark-sql> SELECT 1; 1 Example of entering interactive mode with escape ; in /bin/spark-sql spark-sql>/* This is a comment contains \\\\; > It won't be terminated by \\\\; */ > SELECT 1; 1\n\nExample:\n```text\n./bin/spark-sql\n```\n\nExample:\n```text\nCLI options:\n -d,--define <key=value>          Variable substitution to apply to Hive\n                                  commands. e.g. -d A=B or --define A=B\n    --database <databasename>     Specify the database to use\n -e <quoted-query-string>         SQL from command line\n -f <filename>                    SQL from files\n -H,--help                        Print help information\n    --hiveconf <property=value>   Use value for given property\n    --hivevar <key=value>         Variable substitution to apply to Hive\n                                  commands. e.g. --hivevar A=B\n -i <filename>                    Initialization SQL file\n -S,--silent                      Silent mode in interactive shell\n -v,--verbose                     Verbose mode (echo executed SQL to the\n                                  console)\n```\n\nExample:\n```text\n-- This is a ; comment\nSELECT ';' as a;\n```\n\nExample:\n```text\n/* This is a comment contains ;\n*/ SELECT 1;\n```\n\nExample:\n```text\n./bin/spark-sql -e 'SELECT COL FROM TBL'\n```\n\nExample:\n```text\n./bin/spark-sql -e 'SELECT COL FROM TBL' --hiveconf hive.exec.scratchdir=/home/my/hive_scratch\n```\n\nExample:\n```text\n./bin/spark-sql -e 'SELECT ${hiveconf:aaa}' --hiveconf aaa=bbb --hiveconf hive.exec.scratchdir=/home/my/hive_scratch\nspark-sql> SELECT ${aaa};\nbbb\n```\n\nExample:\n```text\n./bin/spark-sql --hivevar aaa=bbb --define ccc=ddd\nspark-sql> SELECT ${aaa}, ${ccc};\nbbb ddd\n```\n\nExample:\n```text\n./bin/spark-sql -S -e 'SELECT COL FROM TBL' > result.txt\n```\n\nExample:\n```text\n./bin/spark-sql -f /path/to/spark-sql-script.sql\n```\n\nExample:\n```text\n./bin/spark-sql -i /path/to/spark-sql-init.sql\n```\n\nExample:\n```text\n./bin/spark-sql\nspark-sql> SELECT 1;\n1\nspark-sql> -- This is a simple comment.\nspark-sql> SELECT 1;\n1\n```\n\nExample:\n```text\n./bin/spark-sql\nspark-sql>/* This is a comment contains \\\\;\n         > It won't be terminated by \\\\; */\n         > SELECT 1;\n1\n```\n\n<|endofdoc|>","metadata":{"transformedAt":"2026-08-18T15:21:43.130Z","totalSectionsIncluded":1,"totalCodeBlocksIncluded":13,"totalLines":96,"estimatedTokens":1725}}29{"id":"doc-accessing_openstack_swift_from_spark_spark_4_2_0-296706d0","source":"documentation","title":"Accessing OpenStack Swift from Spark - Spark 4.2.0 Documentation","url":"https://spark.apache.org/docs/latest/storage-openstack-swift.html","text":"Accessing OpenStack Swift from Spark Spark’s support for Hadoop InputFormat allows it to process data in OpenStack Swift using the same URI formats as in Hadoop. You can specify a path in Swift as input through a URI of the form swift://container.PROVIDER/path. You will also need to set your Swift security credentials, through core-site.xml or via SparkContext.hadoopConfiguration. The current Swift driver requires Swift to use the Keystone authentication method, or its Rackspace-specific predecessor. Configuring Swift for Better Data Locality Although not mandatory, it is recommended to configure the proxy server of Swift with list_endpoints to have better data locality. More information is available here. Dependencies The Spark application should include hadoop-openstack dependency, which can be done by including the hadoop-cloud module for the specific version of spark used. For example, for Maven support, add the following to the pom.xml file: <dependencyManagement> ... <dependency> <groupId>org.apache.spark</groupId> <artifactId>hadoop-cloud_2.13</artifactId> <version>${spark.version}</version> </dependency> ... </dependencyManagement> Configuration Parameters Create core-site.xml and place it inside Spark’s conf directory. The main category of parameters that should be configured is the authentication parameters required by Keystone. The following table contains a list of Keystone mandatory parameters. PROVIDER can be any (alphanumeric) name. Property NameMeaningRequired fs.swift.service.PROVIDER.auth.url Keystone Authentication URL Mandatory fs.swift.service.PROVIDER.auth.endpoint.prefix Keystone endpoints prefix Optional fs.swift.service.PROVIDER.tenant Tenant Mandatory fs.swift.service.PROVIDER.username Username Mandatory fs.swift.service.PROVIDER.password Password Mandatory fs.swift.service.PROVIDER.http.port HTTP port Mandatory fs.swift.service.PROVIDER.region Keystone region Mandatory fs.swift.service.PROVIDER.public Indicates whether to use the public (off cloud) or private (in cloud; no transfer fees) endpoints Mandatory For example, assume PROVIDER=SparkTest and Keystone contains user tester with password testing defined for tenant test. Then core-site.xml should include: <configuration> <property> <name>fs.swift.service.SparkTest.auth.url</name> <value>http://127.0.0.1:5000/v2.0/tokens</value> </property> <property> <name>fs.swift.service.SparkTest.auth.endpoint.prefix</name> <value>endpoints</value> </property> <name>fs.swift.service.SparkTest.http.port</name> <value>8080</value> </property> <property> <name>fs.swift.service.SparkTest.region</name> <value>RegionOne</value> </property> <property> <name>fs.swift.service.SparkTest.public</name> <value>true</value> </property> <property> <name>fs.swift.service.SparkTest.tenant</name> <value>test</value> </property> <property> <name>fs.swift.service.SparkTest.username</name> <value>tester</value> </property> <property> <name>fs.swift.service.SparkTest.password</name> <value>testing</value> </property> </configuration> Notice that fs.swift.service.PROVIDER.tenant, fs.swift.service.PROVIDER.username, fs.swift.service.PROVIDER.password contains sensitive information and keeping them in core-site.xml is not always a good approach. We suggest to keep those parameters in core-site.xml for testing purposes when running Spark via spark-shell. For job submissions they should be provided via sparkContext.hadoopConfiguration.\n\nExample:\n```xml\n<dependencyManagement>\n  ...\n  <dependency>\n    <groupId>org.apache.spark</groupId>\n    <artifactId>hadoop-cloud_2.13</artifactId>\n    <version>${spark.version}</version>\n  </dependency>\n  ...\n</dependencyManagement>\n```\n\nExample:\n```xml\n<configuration>\n  <property>\n    <name>fs.swift.service.SparkTest.auth.url</name>\n    <value>http://127.0.0.1:5000/v2.0/tokens</value>\n  </property>\n  <property>\n    <name>fs.swift.service.SparkTest.auth.endpoint.prefix</name>\n    <value>endpoints</value>\n  </property>\n    <name>fs.swift.service.SparkTest.http.port</name>\n    <value>8080</value>\n  </property>\n  <property>\n    <name>fs.swift.service.SparkTest.region</name>\n    <value>RegionOne</value>\n  </property>\n  <property>\n    <name>fs.swift.service.SparkTest.public</name>\n    <value>true</value>\n  </property>\n  <property>\n    <name>fs.swift.service.SparkTest.tenant</name>\n    <value>test</value>\n  </property>\n  <property>\n    <name>fs.swift.service.SparkTest.username</name>\n    <value>tester</value>\n  </property>\n  <property>\n    <name>fs.swift.service.SparkTest.password</name>\n    <value>testing</value>\n  </property>\n</configuration>\n```\n\n<|endofdoc|>","metadata":{"transformedAt":"2026-08-18T15:21:43.130Z","totalSectionsIncluded":1,"totalCodeBlocksIncluded":2,"totalLines":53,"estimatedTokens":1154}}30{"id":"doc-web_ui_spark_4_2_0_documentation-ad97c8e8","source":"documentation","title":"Web UI - Spark 4.2.0 Documentation","url":"https://spark.apache.org/docs/latest/web-ui.html","text":"Example:\n```scala\nscala> import org.apache.spark.storage.StorageLevel._\nimport org.apache.spark.storage.StorageLevel._\n\nscala> val rdd = sc.range(0, 100, 1, 5).setName(\"rdd\")\nrdd: org.apache.spark.rdd.RDD[Long] = rdd MapPartitionsRDD[1] at range at <console>:27\n\nscala> rdd.persist(MEMORY_ONLY_SER)\nres0: rdd.type = rdd MapPartitionsRDD[1] at range at <console>:27\n\nscala> rdd.count\nres1: Long = 100\n\nscala> val df = Seq((1, \"andy\"), (2, \"bob\"), (2, \"andy\")).toDF(\"count\", \"name\")\ndf: org.apache.spark.sql.DataFrame = [count: int, name: string]\n\nscala> df.persist(DISK_ONLY)\nres2: df.type = [count: int, name: string]\n\nscala> df.count\nres3: Long = 3\n```\n\nExample:\n```python\ndf = spark.createDataFrame([(1, \"andy\"), (2, \"bob\"), (2, \"andy\")], [\"count\", \"name\"])\ndf.count()\ndf.createOrReplaceTempView(\"df\")\nspark.sql(\"SELECT name, SUM(count) FROM df GROUP BY name\").show()\n```\n\n<|endofdoc|>","metadata":{"transformedAt":"2026-08-18T15:21:43.132Z","totalSectionsIncluded":0,"totalCodeBlocksIncluded":2,"totalLines":33,"estimatedTokens":222}}31{"id":"doc-running_spark_on_yarn_spark_4_2_0_documentation-deffebe9","source":"documentation","title":"Running Spark on YARN - Spark 4.2.0 Documentation","url":"https://spark.apache.org/docs/latest/running-on-yarn.html","text":"Running Spark on YARN Security Launching Spark on YARN Adding Other JARs Preparations Configuration Debugging your Application Spark Properties Available patterns for SHS custom executor log URL Resource Allocation and Configuration Overview Stage Level Scheduling Overview Important notes Kerberos YARN-specific Kerberos Configuration Troubleshooting Kerberos Configuring the External Shuffle Service Launching your application with Apache Oozie Using the Spark History Server to replace the Spark Web UI Running multiple versions of the Spark Shuffle Service Configuring different JDKs for Spark Applications Support for running on YARN (Hadoop NextGen) was added to Spark in version 0.6.0, and improved in subsequent releases. Security Security features like authentication are not enabled by default. When deploying a cluster that is open to the internet or an untrusted network, it’s important to secure access to the cluster to prevent unauthorized applications from running on the cluster. Please see Spark Security and the specific security sections in this doc before running Spark. Launching Spark on YARN Apache Hadoop supports Java 17 since 3.5.0, while Apache Spark requires at least Java 17 since 4.0.0. When running on a YARN cluster whose Hadoop version is older than 3.5.0, a different JDK should be configured for Spark applications. Please refer to Configuring different JDKs for Spark Applications for details. Ensure that HADOOP_CONF_DIR or YARN_CONF_DIR points to the directory which contains the (client side) configuration files for the Hadoop cluster. These configs are used to write to HDFS and connect to the YARN ResourceManager. The configuration contained in this directory will be distributed to the YARN cluster so that all containers used by the application use the same configuration. If the configuration references Java system properties or environment variables not managed by YARN, they should also be set in the Spark application’s configuration (driver, executors, and the AM when running in client mode). There are two deploy modes that can be used to launch Spark applications on YARN. In cluster mode, the Spark driver runs inside an application master process which is managed by YARN on the cluster, and the client can go away after initiating the application. In client mode, the driver runs in the client process, and the application master is only used for requesting resources from YARN. Unlike other cluster managers supported by Spark in which the master’s address is specified in the --master parameter, in YARN mode the ResourceManager’s address is picked up from the Hadoop configuration. Thus, the --master parameter is yarn. To launch a Spark application in cluster mode: $ ./bin/spark-submit --class path.to.your.Class --master yarn --deploy-mode cluster [options] <app jar> [app options] For example: $ ./bin/spark-submit --class org.apache.spark.examples.SparkPi \\ --master yarn \\ --deploy-mode cluster \\ --driver-memory 4g \\ --executor-memory 2g \\ --executor-cores 1 \\ --queue thequeue \\ examples/jars/spark-examples*.jar \\ 10 The above starts a YARN client program which starts the default Application Master. Then SparkPi will be run as a child thread of Application Master. The client will periodically poll the Application Master for status updates and display them in the console. The client will exit once your application has finished running. Refer to the Debugging your Application section below for how to see driver and executor logs. To launch a Spark application in client mode, do the same, but replace cluster with client. The following shows how you can run spark-shell in client mode: $ ./bin/spark-shell --master yarn --deploy-mode client Adding Other JARs In cluster mode, the driver runs on a different machine than the client, so SparkContext.addJar won’t work out of the box with files that are local to the client. To make files on the client available to SparkContext.addJar, include them with the --jars option in the launch command. $ ./bin/spark-submit --class my.main.Class \\ --master yarn \\ --deploy-mode cluster \\ --jars my-other-jar.jar,my-other-other-jar.jar \\ my-main-jar.jar \\ app_arg1 app_arg2 Preparations Running Spark on YARN requires a binary distribution of Spark which is built with YARN support. Binary distributions can be downloaded from the downloads page of the project website. There are two variants of Spark binary distributions you can download. One is pre-built with a certain version of Apache Hadoop; this Spark distribution contains built-in Hadoop runtime, so we call it with-hadoop Spark distribution. The other one is pre-built with user-provided Hadoop; since this Spark distribution doesn’t contain a built-in Hadoop runtime, it’s smaller, but users have to provide a Hadoop installation separately. We call this variant no-hadoop Spark distribution. For with-hadoop Spark distribution, since it contains a built-in Hadoop runtime already, by default, when a job is submitted to Hadoop Yarn cluster, to prevent jar conflict, it will not populate Yarn’s classpath into Spark. To override this behavior, you can set spark.yarn.populateHadoopClasspath=true. For no-hadoop Spark distribution, Spark will populate Yarn’s classpath by default in order to get Hadoop runtime. For with-hadoop Spark distribution, if your application depends on certain library that is only available in the cluster, you can try to populate the Yarn classpath by setting the property mentioned above. If you run into jar conflict issue by doing so, you will need to turn it off and include this library in your application jar. To build Spark yourself, refer to Building Spark. To make Spark runtime jars accessible from YARN side, you can specify spark.yarn.archive or spark.yarn.jars. For details please refer to Spark Properties. If neither spark.yarn.archive nor spark.yarn.jars is specified, Spark will create a zip file with all jars under $SPARK_HOME/jars and upload it to the distributed cache. Configuration Most of the configs are the same for Spark on YARN as for other deployment modes. See the configuration page for more information on those. These are configs that are specific to Spark on YARN. Debugging your Application In YARN terminology, executors and application masters run inside “containers”. YARN has two modes for handling container logs after an application has completed. If log aggregation is turned on (with the yarn.log-aggregation-enable config), container logs are copied to HDFS and deleted on the local machine. These logs can be viewed from anywhere on the cluster with the yarn logs command. yarn logs -applicationId <app ID> will print out the contents of all log files from all containers from the given application. You can also view the container log files directly in HDFS using the HDFS shell or API. The directory where they are located can be found by looking at your YARN configs (yarn.nodemanager.remote-app-log-dir and yarn.nodemanager.remote-app-log-dir-suffix). The logs are also available on the Spark Web UI under the Executors Tab. You need to have both the Spark history server and the MapReduce history server running and configure yarn.log.server.url in yarn-site.xml properly. The log URL on the Spark history server UI will redirect you to the MapReduce history server to show the aggregated logs. When log aggregation isn’t turned on, logs are retained locally on each machine under YARN_APP_LOGS_DIR, which is usually configured to /tmp/logs or $HADOOP_HOME/logs/userlogs depending on the Hadoop version and installation. Viewing logs for a container requires going to the host that contains them and looking in this directory. Subdirectories organize log files by application ID and container ID. The logs are also available on the Spark Web UI under the Executors Tab and doesn’t require running the MapReduce history server. To review per-container launch environment, increase yarn.nodemanager.delete.debug-delay-sec to a large value (e.g. 36000), and then access the application cache through yarn.nodemanager.local-dirs on the nodes on which containers are launched. This directory contains the launch script, JARs, and all environment variables used for launching each container. This process is useful for debugging classpath problems in particular. (Note that enabling this requires admin privileges on cluster settings and a restart of all node managers. Thus, this is not applicable to hosted clusters). To use a custom log4j2 configuration for the application master or executors, here are the a custom log4j2.properties using spark-submit, by adding it to the --files list of files to be uploaded with the application. add -Dlog4j.configurationFile=<location of configuration file> to spark.driver.extraJavaOptions (for the driver) or spark.executor.extraJavaOptions (for executors). Note that if using a file, the should be explicitly provided, and the file needs to exist locally on all the nodes. update the $SPARK_CONF_DIR/log4j2.properties file and it will be automatically uploaded along with the other configurations. Note that other 2 options has higher priority than this option if multiple options are specified. Note that for the first option, both executors and the application master will share the same log4j configuration, which may cause issues when they run on the same node (e.g. trying to write to the same log file). If you need a reference to the proper location to put log files in the YARN so that YARN can properly display and aggregate them, use spark.yarn.app.container.log.dir in your log4j2.properties. For example, appender.file_appender.fileName=${sys:spark.yarn.app.container.log.dir}/spark.log. For streaming applications, configuring RollingFileAppender and setting file location to YARN’s log directory will avoid disk overflow caused by large log files, and logs can be accessed using YARN’s log utility. To use a custom metrics.properties for the application master and executors, update the $SPARK_CONF_DIR/metrics.properties file. It will automatically be uploaded with other configurations, so you don’t need to specify it manually with --files. Spark Properties Property NameDefaultMeaningSince Version spark.yarn.am.memory 512m Amount of memory to use for the YARN Application Master in client mode, in the same format as JVM memory strings (e.g. 512m, 2g). In cluster mode, use spark.driver.memory instead. Use lower-case suffixes, e.g. k, m, g, t, and p, for kibi-, mebi-, gibi-, tebi-, and pebibytes, respectively. 1.3.0 spark.yarn.am.resource.{resource-type}.amount (none) Amount of resource to use for the YARN Application Master in client mode. In cluster mode, use spark.yarn.driver.resource.<resource-type>.amount instead. Please note that this feature can be used only with YARN 3.0+ For reference, see YARN Resource Model documentation request GPU resources from YARN, /gpu.amount 3.0.0 spark.yarn.applicationType SPARK Defines more specific application types, e.g. SPARK, SPARK-SQL, SPARK-STREAMING, SPARK-MLLIB and SPARK-GRAPH. Please be careful not to exceed 20 characters. 3.1.0 spark.yarn.driver.resource.{resource-type}.amount (none) Amount of resource to use for the YARN Application Master in cluster mode. Please note that this feature can be used only with YARN 3.0+ For reference, see YARN Resource Model documentation request GPU resources from YARN, /gpu.amount 3.0.0 spark.yarn.executor.resource.{resource-type}.amount (none) Amount of resource to use per executor process. Please note that this feature can be used only with YARN 3.0+ For reference, see YARN Resource Model documentation request GPU resources from YARN, /gpu.amount 3.0.0 spark.yarn.resourceGpuDeviceName yarn.io/gpu Specify the mapping of the Spark resource type of gpu to the YARN resource representing a GPU. By default YARN uses yarn.io/gpu but if YARN has been configured with a custom resource type, this allows remapping it. Applies when using the spark.{driver/executor}.resource.gpu.* configs. 3.2.1 spark.yarn.resourceFpgaDeviceName yarn.io/fpga Specify the mapping of the Spark resource type of fpga to the YARN resource representing a FPGA. By default YARN uses yarn.io/fpga but if YARN has been configured with a custom resource type, this allows remapping it. Applies when using the spark.{driver/executor}.resource.fpga.* configs. 3.2.1 spark.yarn.am.cores 1 Number of cores to use for the YARN Application Master in client mode. In cluster mode, use spark.driver.cores instead. 1.3.0 spark.yarn.am.waitTime 100s Only used in cluster mode. Time for the YARN Application Master to wait for the SparkContext to be initialized. 1.3.0 spark.yarn.submit.file.replication The default HDFS replication (usually 3) HDFS replication level for the files uploaded into HDFS for the application. These include things like the Spark jar, the app jar, and any distributed cache files/archives. 0.8.1 spark.yarn.stagingDir Current user's home directory in the filesystem Staging directory used while submitting applications. 2.0.0 spark.yarn.preserve.staging.files false Set to true to preserve the staged files (Spark jar, app jar, distributed cache files) at the end of the job rather than delete them. 1.1.0 spark.yarn.scheduler.heartbeat.interval-ms 3000 The interval in ms in which the Spark application master heartbeats into the YARN ResourceManager. The value is capped at half the value of YARN's configuration for the expiry interval, i.e. yarn.am.liveness-monitor.expiry-interval-ms. 0.8.1 spark.yarn.scheduler.initial-allocation.interval 200ms The initial interval in which the Spark application master eagerly heartbeats to the YARN ResourceManager when there are pending container allocation requests. It should be no larger than spark.yarn.scheduler.heartbeat.interval-ms. The allocation interval will doubled on successive eager heartbeats if pending containers still exist, until spark.yarn.scheduler.heartbeat.interval-ms is reached. 1.4.0 spark.yarn.historyServer.address (none) The address of the Spark history server, e.g. host.com:18080. The address should not contain a scheme (http://). Defaults to not being set since the history server is an optional service. This address is given to the YARN ResourceManager when the Spark application finishes to link the application from the ResourceManager UI to the Spark history server UI. For this property, YARN properties can be used as variables, and these are substituted by Spark at runtime. For example, if the Spark history server runs on the same node as the YARN ResourceManager, it can be set to ${hadoopconf-yarn.resourcemanager.hostname}:18080. 1.0.0 spark.yarn.dist.archives (none) Comma separated list of archives to be extracted into the working directory of each executor. 1.0.0 spark.yarn.dist.files (none) Comma-separated list of files to be placed in the working directory of each executor. 1.0.0 spark.yarn.dist.jars (none) Comma-separated list of jars to be placed in the working directory of each executor. 2.0.0 spark.yarn.dist.forceDownloadSchemes (none) Comma-separated list of schemes for which resources will be downloaded to the local disk prior to being added to YARN's distributed cache. For use in cases where the YARN service does not support schemes that are supported by Spark, like http, https and ftp, or jars required to be in the local YARN client's classpath. Wildcard '*' is denoted to download resources for all the schemes. 2.3.0 spark.executor.instances 2 The number of executors for static allocation. With spark.dynamicAllocation.enabled, the initial set of executors will be at least this large. 1.0.0 spark.yarn.am.memoryOverhead AM memory * 0.10, with minimum of 384 Same as spark.driver.memoryOverhead, but for the YARN Application Master in client mode. 1.3.0 spark.yarn.queue default The name of the YARN queue to which the application is submitted. 1.0.0 spark.yarn.jars (none) List of libraries containing Spark code to distribute to YARN containers. By default, Spark on YARN will use Spark jars installed locally, but the Spark jars can also be in a world-readable location on HDFS. This allows YARN to cache it on nodes so that it doesn't need to be distributed each time an application runs. To point to jars on HDFS, for example, set this configuration to hdfs:///some/path. Globs are allowed. 2.0.0 spark.yarn.archive (none) An archive containing needed Spark jars for distribution to the YARN cache. If set, this configuration replaces spark.yarn.jars and the archive is used in all the application's containers. The archive should contain jar files in its root directory. Like with the previous option, the archive can also be hosted on HDFS to speed up file distribution. 2.0.0 spark.yarn.appMasterEnv.[EnvironmentVariableName] (none) Add the environment variable specified by EnvironmentVariableName to the Application Master process launched on YARN. The user can specify multiple of these and to set multiple environment variables. In cluster mode this controls the environment of the Spark driver and in client mode it only controls the environment of the executor launcher. 1.1.0 spark.yarn.containerLauncherMaxThreads 25 The maximum number of threads to use in the YARN Application Master for launching executor containers. 1.2.0 spark.yarn.am.defaultJavaOptions (none) A string of default JVM options to prepend to spark.yarn.am.extraJavaOptions for the YARN Application Master in client mode. Note that it is illegal to set maximum heap size (-Xmx) settings with this option. Maximum heap size settings can be set with spark.yarn.am.memory. This is intended to be set by administrators. 4.2.0 spark.yarn.am.extraJavaOptions (none) A string of extra JVM options to pass to the YARN Application Master in client mode. In cluster mode, use spark.driver.extraJavaOptions instead. Note that it is illegal to set maximum heap size (-Xmx) settings with this option. Maximum heap size settings can be set with spark.yarn.am.memory. spark.yarn.am.defaultJavaOptions will be prepended to this configuration. 1.3.0 spark.yarn.am.extraLibraryPath (none) Set a special library path to use when launching the YARN Application Master in client mode. 1.4.0 spark.yarn.populateHadoopClasspath For with-hadoop Spark distribution, this is set to false; for no-hadoop distribution, this is set to true. Whether to populate Hadoop classpath from yarn.application.classpath and mapreduce.application.classpath Note that if this is set to false, it requires a with-Hadoop Spark distribution that bundles Hadoop runtime or user has to provide a Hadoop installation separately. 2.4.6 spark.yarn.maxAppAttempts yarn.resourcemanager.am.max-attempts in YARN The maximum number of attempts that will be made to submit the application. It should be no larger than the global number of max attempts in the YARN configuration. 1.3.0 spark.yarn.am.attemptFailuresValidityInterval (none) Defines the validity interval for AM failure tracking. If the AM has been running for at least the defined interval, the AM failure count will be reset. This feature is not enabled if not configured. 1.6.0 spark.yarn.am.clientModeTreatDisconnectAsFailed false Treat yarn-client unclean disconnects as failures. In yarn-client mode, normally the application will always finish with a final status of SUCCESS because in some cases, it is not possible to know if the Application was terminated intentionally by the user or if there was a real error. This config changes that behavior such that if the Application Master disconnects from the driver uncleanly (ie without the proper shutdown handshake) the application will terminate with a final status of FAILED. This will allow the caller to decide if it was truly a failure. Note that if this config is set and the user just terminate the client application badly it may show a status of FAILED when it wasn't really FAILED. 3.3.0 spark.yarn.am.clientModeExitOnError false In yarn-client mode, when this is true, if driver got application report with final status of KILLED or FAILED, driver will stop corresponding SparkContext and exit program with code 1. Note, if this is true and called from another application, it will terminate the parent application as well. 3.3.0 spark.yarn.am.tokenConfRegex (none) The value of this config is a regex expression used to grep a list of config entries from the job's configuration file (e.g., hdfs-site.xml) and send to RM, which uses them when renewing delegation tokens. A typical use case of this feature is to support delegation tokens in an environment where a YARN cluster needs to talk to multiple downstream HDFS clusters, where the YARN RM may not have configs (e.g., dfs.nameservices, dfs.ha.namenodes.*, dfs.namenode.rpc-address.*) to connect to these clusters. In this scenario, Spark users can specify the config value to be ^dfs.nameservices\\$|^dfs.namenode.rpc-address.*\\$|^dfs.ha.namenodes.*\\$ to parse these HDFS configs from the job's local configuration files. This config is very similar to mapreduce.job.send-token-conf. Please check YARN-5910 for more details. 3.3.0 spark.yarn.submit.waitAppCompletion true In YARN cluster mode, controls whether the client waits to exit until the application completes. If set to true, the client process will stay alive reporting the application's status. Otherwise, the client process will exit after submission. 1.4.0 spark.yarn.am.nodeLabelExpression (none) A YARN node label expression that restricts the set of nodes AM will be scheduled on. Only versions of YARN greater than or equal to 2.6 support node label expressions, so when running against earlier versions, this property will be ignored. 1.6.0 spark.yarn.executor.nodeLabelExpression (none) A YARN node label expression that restricts the set of nodes executors will be scheduled on. Only versions of YARN greater than or equal to 2.6 support node label expressions, so when running against earlier versions, this property will be ignored. 1.4.0 spark.yarn.tags (none) Comma-separated list of strings to pass through as YARN application tags appearing in YARN ApplicationReports, which can be used for filtering when querying YARN apps. 1.5.0 spark.yarn.priority (none) Application priority for YARN to define pending applications ordering policy, those with higher integer value have a better opportunity to be activated. Currently, YARN only supports application priority when using FIFO ordering policy. 3.0.0 spark.yarn.config.gatewayPath (none) A path that is valid on the gateway host (the host where a Spark application is started) but may differ for paths for the same resource in other nodes in the cluster. Coupled with spark.yarn.config.replacementPath, this is used to support clusters with heterogeneous configurations, so that Spark can correctly launch remote processes. The replacement path normally will contain a reference to some environment variable exported by YARN (and, thus, visible to Spark containers). For example, if the gateway node has Hadoop libraries installed on /disk1/hadoop, and the location of the Hadoop install is exported by YARN as the HADOOP_HOME environment variable, setting this value to /disk1/hadoop and the replacement path to $HADOOP_HOME will make sure that paths used to launch remote processes properly reference the local YARN configuration. 1.5.0 spark.yarn.config.replacementPath (none) See spark.yarn.config.gatewayPath. 1.5.0 spark.yarn.rolledLog.includePattern (none) Java Regex to filter the log files which match the defined include pattern and those log files will be aggregated in a rolling fashion. This will be used with YARN's rolling log aggregation, to enable this feature in YARN side yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds should be configured in yarn-site.xml. The Spark log4j appender needs be changed to use FileAppender or another appender that can handle the files being removed while it is running. Based on the file name configured in the log4j configuration (like spark.log), the user should set the regex (spark*) to include all the log files that need to be aggregated. 2.0.0 spark.yarn.rolledLog.excludePattern (none) Java Regex to filter the log files which match the defined exclude pattern and those log files will not be aggregated in a rolling fashion. If the log file name matches both the include and the exclude pattern, this file will be excluded eventually. 2.0.0 spark.yarn.executor.launch.excludeOnFailure.enabled false Flag to enable exclusion of nodes having YARN resource allocation problems. The error limit for excluding can be configured by spark.excludeOnFailure.application.maxFailedExecutorsPerNode. 2.4.0 spark.yarn.exclude.nodes (none) Comma-separated list of YARN node names which are excluded from resource allocation. 3.0.0 spark.yarn.metrics.namespace (none) The root namespace for AM metrics reporting. If it is not set then the YARN application ID is used. 2.4.0 spark.yarn.report.interval 1s Interval between reports of the current Spark job status in cluster mode. 0.9.0 spark.yarn.report.loggingFrequency 30 Maximum number of application reports processed until the next application status is logged. If there is a change of state, the application status will be logged regardless of the number of application reports processed. 3.5.0 spark.yarn.clientLaunchMonitorInterval 1s Interval between requests for status the client mode AM when starting the app. 2.3.0 spark.yarn.includeDriverLogsLink false In cluster mode, whether the client application report includes links to the driver container's logs. This requires polling the ResourceManager's REST API, so it places some additional load on the RM. 3.1.0 spark.yarn.unmanagedAM.enabled false In client mode, whether to launch the Application Master service as part of the client using unmanaged am. 3.0.0 spark.yarn.shuffle.server.recovery.disabled false Set to true for applications that have higher security requirements and prefer that their secret is not saved in the db. The shuffle data of such applications will not be recovered after the External Shuffle Service restarts. 3.5.0 Available patterns for SHS custom executor log URL PatternMeaning {{HTTP_SCHEME}} http:// or https:// according to YARN HTTP policy. (Configured via yarn.http.policy) {{NM_HOST}} The \"host\" of node where container was run. {{NM_PORT}} The \"port\" of node manager where container was run. {{NM_HTTP_PORT}} The \"port\" of node manager's http server where container was run. {{NM_HTTP_ADDRESS}} Http URI of the node on which the container is allocated. {{CLUSTER_ID}} The cluster ID of Resource Manager. (Configured via yarn.resourcemanager.cluster-id) {{CONTAINER_ID}} The ID of container. {{USER}} SPARK_USER on system environment. {{FILE_NAME}} stdout, stderr. For example, suppose you would like to point log url link to Job History Server directly instead of let NodeManager http server redirects it, you can configure spark.history.custom.executor.log.url as below: {{HTTP_SCHEME}}<JHS_HOST>:<JHS_PORT>/jobhistory/logs/{{NM_HOST}}:{{NM_PORT}}/{{CONTAINER_ID}}/{{CONTAINER_ID}}/{{USER}}/{{FILE_NAME}}?start=-4096 need to replace <JHS_HOST> and <JHS_PORT> with actual value. Resource Allocation and Configuration Overview Please make sure to have read the Custom Resource Scheduling and Configuration Overview section on the configuration page. This section only talks about the YARN specific aspects of resource scheduling. YARN needs to be configured to support any resources the user wants to use with Spark. Resource scheduling on YARN was added in YARN 3.1.0. See the YARN documentation for more information on configuring resources and properly setting up isolation. Ideally the resources are setup isolated so that an executor can only see the resources it was allocated. If you do not have isolation enabled, the user is responsible for creating a discovery script that ensures the resource is not shared between executors. YARN supports user defined resource types but has built in types for GPU (yarn.io/gpu) and FPGA (yarn.io/fpga). For that reason, if you are using either of those resources, Spark can translate your request for spark resources into YARN resources and you only have to specify the spark.{driver/executor}.resource. configs. Note, if you are using a custom resource type for GPUs or FPGAs with YARN you can change the Spark mapping using spark.yarn.resourceGpuDeviceName and spark.yarn.resourceFpgaDeviceName. If you are using a resource other than FPGA or GPU, the user is responsible for specifying the configs for both YARN (spark.yarn.{driver/executor}.resource.) and Spark (spark.{driver/executor}.resource.). For example, the user wants to request 2 GPUs for each executor. The user can just specify spark.executor.resource.gpu.amount=2 and Spark will handle requesting yarn.io/gpu resource type from YARN. If the user has a user defined YARN resource, lets call it acceleratorX then the user must specify spark.yarn.executor.resource.acceleratorX.amount=2 and spark.executor.resource.acceleratorX.amount=2. YARN does not tell Spark the addresses of the resources allocated to each container. For that reason, the user must specify a discovery script that gets run by the executor on startup to discover what resources are available to that executor. You can find an example scripts in examples/src/main/scripts/getGpusResources.sh. The script must have execute permissions set and the user should setup permissions to not allow malicious users to modify it. The script should write to STDOUT a JSON string in the format of the ResourceInformation class. This has the resource name and an array of resource addresses available to just that executor. Stage Level Scheduling Overview Stage level scheduling is supported on dynamic allocation is allows users to specify different task resource requirements at the stage level and will use the same executors requested at startup. When dynamic allocation is allows users to specify task and executor resource requirements at the stage level and will request the extra executors. One thing to note that is YARN specific is that each ResourceProfile requires a different container priority on YARN. The mapping is simply the ResourceProfile id becomes the priority, on YARN lower numbers are higher priority. This means that profiles created earlier will have a higher priority in YARN. Normally this won’t matter as Spark finishes one stage before starting another one, the only case this might have an affect is in a job server type scenario, so its something to keep in mind. Note there is a difference in the way custom resources are handled between the base default profile and custom ResourceProfiles. To allow for the user to request YARN containers with extra resources without Spark scheduling on them, the user can specify resources via the spark.yarn.executor.resource. config. Those configs are only used in the base default profile though and do not get propagated into any other custom ResourceProfiles. This is because there would be no way to remove them if you wanted a stage to not have them. This results in your default profile getting custom resources defined in spark.yarn.executor.resource. plus spark defined resources of GPU or FPGA. Spark converts GPU and FPGA resources into the YARN built in types yarn.io/gpu) and yarn.io/fpga, but does not know the mapping of any other resources. Any other Spark custom resources are not propagated to YARN for the default profile. So if you want Spark to schedule based off a custom resource and have it requested from YARN, you must specify it in both YARN (spark.yarn.{driver/executor}.resource.) and Spark (spark.{driver/executor}.resource.) configs. Leave the Spark config off if you only want YARN containers with the extra resources but Spark not to schedule using them. Now for custom ResourceProfiles, it doesn’t currently have a way to only specify YARN resources without Spark scheduling off of them. This means for custom ResourceProfiles we propagate all the resources defined in the ResourceProfile to YARN. We still convert GPU and FPGA to the YARN build in types as well. This requires that the name of any custom resources you specify match what they are defined as in YARN. Important notes Whether core requests are honored in scheduling decisions depends on which scheduler is in use and how it is configured. In cluster mode, the local directories used by the Spark executors and the Spark driver will be the local directories configured for YARN (Hadoop YARN config yarn.nodemanager.local-dirs). If the user specifies spark.local.dir, it will be ignored. In client mode, the Spark executors will use the local directories configured for YARN while the Spark driver will use those defined in spark.local.dir. This is because the Spark driver does not run on the YARN cluster in client mode, only the Spark executors do. The --files and --archives options support specifying file names with the # similar to Hadoop. For example, you can localtest.txt#appSees.txt and this will upload the file you have locally named localtest.txt into HDFS but this will be linked to by the name appSees.txt, and your application should use the name as appSees.txt to reference it when running on YARN. The --jars option allows the SparkContext.addJar function to work if you are using it with local files and running in cluster mode. It does not need to be used if you are using it with HDFS, HTTP, HTTPS, or FTP files. Kerberos Standard Kerberos support in Spark is covered in the Security page. In YARN mode, when accessing Hadoop file systems, aside from the default file system in the hadoop configuration, Spark will also automatically obtain delegation tokens for the service hosting the staging directory of the Spark application. YARN-specific Kerberos Configuration Property NameDefaultMeaningSince Version spark.kerberos.keytab (none) The full path to the file that contains the keytab for the principal specified above. This keytab will be copied to the node running the YARN Application Master via the YARN Distributed Cache, and will be used for renewing the login tickets and the delegation tokens periodically. Equivalent to the --keytab command line argument. (Works also with the \"local\" master.) 3.0.0 spark.kerberos.principal (none) Principal to be used to login to KDC, while running on secure clusters. Equivalent to the --principal command line argument. (Works also with the \"local\" master.) 3.0.0 spark.yarn.kerberos.relogin.period 1m How often to check whether the kerberos TGT should be renewed. This should be set to a value that is shorter than the TGT renewal period (or the TGT lifetime if TGT renewal is not enabled). The default value should be enough for most deployments. 2.3.0 spark.yarn.kerberos.renewal.excludeHadoopFileSystems (none) A comma-separated list of Hadoop filesystems for whose hosts will be excluded from delegation token renewal at resource scheduler. For example, spark.yarn.kerberos.renewal.excludeHadoopFileSystems=hdfs://nn1.com:8032, hdfs://nn2.com:8032. This is known to work under YARN for now, so YARN Resource Manager won't renew tokens for the application. Note that as resource scheduler does not renew token, so any application running longer than the original token expiration that tries to use that token will likely fail. 3.2.0 Troubleshooting Kerberos Debugging Hadoop/Kerberos problems can be “difficult”. One useful technique is to enable extra logging of Kerberos operations in Hadoop by setting the HADOOP_JAAS_DEBUG environment variable. export HADOOP_JAAS_DEBUG=true The JDK classes can be configured to enable extra logging of their Kerberos and SPNEGO/REST authentication via the system properties sun.security.krb5.debug and sun.security.spnego.debug=true -Dsun.security.krb5.debug=true -Dsun.security.spnego.debug=true All these options can be enabled in the Application true spark.yarn.am.extraJavaOptions -Dsun.security.krb5.debug=true -Dsun.security.spnego.debug=true Finally, if the log level for org.apache.spark.deploy.yarn.Client is set to DEBUG, the log will include a list of all tokens obtained, and their expiry details Configuring the External Shuffle Service To start the Spark Shuffle Service on each NodeManager in your YARN cluster, follow these Spark with the YARN profile. Skip this step if you are using a pre-packaged distribution. Locate the spark-<version>-yarn-shuffle.jar. This should be under $SPARK_HOME/common/network-yarn/target/scala-<version> if you are building Spark yourself, and under yarn if you are using a distribution. Add this jar to the classpath of all NodeManagers in your cluster. In the yarn-site.xml on each node, add spark_shuffle to yarn.nodemanager.aux-services, then set yarn.nodemanager.aux-services.spark_shuffle.class to org.apache.spark.network.yarn.YarnShuffleService. Increase NodeManager's heap size by setting YARN_HEAPSIZE (1000 by default) in etc/hadoop/yarn-env.sh to avoid garbage collection issues during shuffle. Restart all NodeManagers in your cluster. The following extra configuration options are available when the shuffle service is running on NameDefaultMeaningSince Version spark.yarn.shuffle.stopOnFailure false Whether to stop the NodeManager when there's a failure in the Spark Shuffle Service's initialization. This prevents application failures caused by running containers on NodeManagers where the Spark Shuffle Service is not running. 2.1.0 spark.yarn.shuffle.service.metrics.namespace sparkShuffleService The namespace to use when emitting shuffle service metrics into Hadoop metrics2 system of the NodeManager. 3.2.0 spark.yarn.shuffle.service.logs.namespace (not set) A namespace which will be appended to the class name when forming the logger name to use for emitting logs from the YARN shuffle service, like org.apache.spark.network.yarn.YarnShuffleService.logsNamespaceValue. Since some logging frameworks may expect the logger name to look like a class name, it's generally recommended to provide a value which would be a valid Java package or class name and not include spaces. 3.3.0 spark.shuffle.service.db.backend ROCKSDB When work-preserving restart is enabled in YARN, this is used to specify the disk-base store used in shuffle service state store, supports `ROCKSDB` and `LEVELDB` (deprecated) with `ROCKSDB` as default value. The original data store in `RocksDB/LevelDB` will not be automatically converted to another kind of storage now. The original data store will be retained and the new type data store will be created when switching storage types. 3.4.0 Please note that the instructions above assume that the default shuffle service name, spark_shuffle, has been used. It is possible to use any name here, but the values used in the YARN NodeManager configurations must match the value of spark.shuffle.service.name in the Spark application. The shuffle service will, by default, take all of its configurations from the Hadoop Configuration used by the NodeManager (e.g. yarn-site.xml). However, it is also possible to configure the shuffle service independently using a file named spark-shuffle-site.xml which should be placed onto the classpath of the shuffle service (which is, by default, shared with the classpath of the NodeManager). The shuffle service will treat this as a standard Hadoop Configuration resource and overlay it on top of the NodeManager’s configuration. Launching your application with Apache Oozie Apache Oozie can launch Spark applications as part of a workflow. In a secure cluster, the launched application will need the relevant tokens to access the cluster’s services. If Spark is launched with a keytab, this is automatic. However, if Spark is to be launched without a keytab, the responsibility for setting up security must be handed over to Oozie. The details of configuring Oozie for secure clusters and obtaining credentials for a job can be found on the Oozie web site in the “Authentication” section of the specific release’s documentation. For Spark applications, the Oozie workflow must be set up for Oozie to request all tokens which the application needs, YARN resource manager. The local Hadoop filesystem. Any remote Hadoop filesystems used as a source or destination of I/O. Hive —if used. HBase —if used. The YARN timeline server, if the application interacts with this. To avoid Spark attempting —and then failing— to obtain Hive, HBase and remote HDFS tokens, the Spark configuration must be set to disable token collection for the services. The Spark configuration must include the false spark.security.credentials.hbase.enabled false The configuration option spark.kerberos.access.hadoopFileSystems must be unset. Using the Spark History Server to replace the Spark Web UI It is possible to use the Spark History Server application page as the tracking URL for running applications when the application UI is disabled. This may be desirable on secure clusters, or to reduce the memory usage of the Spark driver. To set up tracking through the Spark History Server, do the the application side, set spark.yarn.historyServer.allowTracking=true in Spark’s configuration. This will tell Spark to use the history server’s URL as the tracking URL if the application’s UI is disabled. On the Spark History Server, add org.apache.spark.deploy.yarn.YarnProxyRedirectFilter to the list of filters in the spark.ui.filters configuration. Be aware that the history server information may not be up-to-date with the application’s state. Running multiple versions of the Spark Shuffle Service Please note that this section only applies when running on YARN versions >= 2.9.0. In some cases it may be desirable to run multiple instances of the Spark Shuffle Service which are using different versions of Spark. This can be helpful, for example, when running a YARN cluster with a mixed workload of applications running multiple Spark versions, since a given version of the shuffle service is not always compatible with other versions of Spark. YARN versions since 2.9.0 support the ability to run shuffle services within an isolated classloader (see YARN-4577), meaning multiple Spark versions can coexist within a single NodeManager. The yarn.nodemanager.aux-services.<service-name>.classpath and, starting from YARN 2.10.2/3.1.1/3.2.0, yarn.nodemanager.aux-services.<service-name>.remote-classpath options can be used to configure this. Note that YARN 3.3.0/3.3.1 have an issue which requires setting yarn.nodemanager.aux-services.<service-name>.system-classes as a workaround. See YARN-11053 for details. In addition to setting up separate classpaths, it’s necessary to ensure the two versions advertise to different ports. This can be achieved using the spark-shuffle-site.xml file described above. For example, you may have configuration = spark_shuffle_x,spark_shuffle_y yarn.nodemanager.aux-services.spark_shuffle_x.classpath = /path/to/spark-x-path/fat.jar:/path/to/spark-x-config yarn.nodemanager.aux-services.spark_shuffle_y.classpath = /path/to/spark-y-path/fat.jar:/path/to/spark-y-config Or yarn.nodemanager.aux-services = spark_shuffle_x,spark_shuffle_y yarn.nodemanager.aux-services.spark_shuffle_x.classpath = /path/to/spark-x-path/*:/path/to/spark-x-config yarn.nodemanager.aux-services.spark_shuffle_y.classpath = /path/to/spark-y-path/*:/path/to/spark-y-config The two spark-*-config directories each contain one file, spark-shuffle-site.xml. These are XML files in the Hadoop Configuration format which each contain a few configurations to adjust the port number and metrics name prefix used: <configuration> <property> <name>spark.shuffle.service.port</name> <value>7001</value> </property> <property> <name>spark.yarn.shuffle.service.metrics.namespace</name> <value>sparkShuffleServiceX</value> </property> </configuration> The values should both be different for the two different services. Then, in the configuration of the Spark applications, one should be configured = spark_shuffle_x spark.shuffle.service.port = 7001 and one should be configured = spark_shuffle_y spark.shuffle.service.port = <other value> Configuring different JDKs for Spark Applications In some cases it may be desirable to use a different JDK from YARN node manager to run Spark applications, this can be achieved by setting the JAVA_HOME environment variable for YARN containers and the spark-submit process. Note that, Spark assumes that all JVM processes runs in one application use the same version of JDK, otherwise, you may encounter JDK serialization issues. To configure a Spark application to use a JDK which has been pre-installed on all nodes at /opt/openjdk-17: $ export JAVA_HOME=/opt/openjdk-17 $ ./bin/spark-submit --class path.to.your.Class \\ --master yarn \\ --conf spark.yarn.appMasterEnv.JAVA_HOME=/opt/openjdk-17 \\ --conf spark.executorEnv.JAVA_HOME=/opt/openjdk-17 \\ <app jar> [app options] Optionally, the user may want to avoid installing a different JDK on the YARN cluster nodes, in such a case, it’s also possible to distribute the JDK using YARN’s Distributed Cache. For example, to use Java 21 to run a Spark application, prepare a JDK 21 tarball openjdk-21.tar.gz and untar it to /opt on the local node, then submit a Spark application: $ export JAVA_HOME=/opt/openjdk-21 $ ./bin/spark-submit --class path.to.your.Class \\ --master yarn \\ --archives path/to/openjdk-21.tar.gz \\ --conf spark.yarn.appMasterEnv.JAVA_HOME=./openjdk-21.tar.gz/openjdk-21 \\ --conf spark.executorEnv.JAVA_HOME=./openjdk-21.tar.gz/openjdk-21 \\ <app jar> [app options]\n\nExample:\n```text\n$ ./bin/spark-submit --class path.to.your.Class --master yarn --deploy-mode cluster [options] <app jar> [app options]\n```\n\nExample:\n```text\n$ ./bin/spark-submit --class org.apache.spark.examples.SparkPi \\\n    --master yarn \\\n    --deploy-mode cluster \\\n    --driver-memory 4g \\\n    --executor-memory 2g \\\n    --executor-cores 1 \\\n    --queue thequeue \\\n    examples/jars/spark-examples*.jar \\\n    10\n```\n\nExample:\n```text\n$ ./bin/spark-shell --master yarn --deploy-mode client\n```\n\nExample:\n```text\n$ ./bin/spark-submit --class my.main.Class \\\n    --master yarn \\\n    --deploy-mode cluster \\\n    --jars my-other-jar.jar,my-other-other-jar.jar \\\n    my-main-jar.jar \\\n    app_arg1 app_arg2\n```\n\nExample:\n```text\nyarn logs -applicationId <app ID>\n```\n\nExample:\n```text\nexport HADOOP_JAAS_DEBUG=true\n```\n\nExample:\n```text\n-Dsun.security.krb5.debug=true -Dsun.security.spnego.debug=true\n```\n\nExample:\n```text\nspark.yarn.appMasterEnv.HADOOP_JAAS_DEBUG true\nspark.yarn.am.extraJavaOptions -Dsun.security.krb5.debug=true -Dsun.security.spnego.debug=true\n```\n\nExample:\n```text\nspark.security.credentials.hive.enabled   false\nspark.security.credentials.hbase.enabled  false\n```\n\nExample:\n```text\nyarn.nodemanager.aux-services = spark_shuffle_x,spark_shuffle_y\n  yarn.nodemanager.aux-services.spark_shuffle_x.classpath = /path/to/spark-x-path/fat.jar:/path/to/spark-x-config\n  yarn.nodemanager.aux-services.spark_shuffle_y.classpath = /path/to/spark-y-path/fat.jar:/path/to/spark-y-config\n```\n\nExample:\n```text\nyarn.nodemanager.aux-services = spark_shuffle_x,spark_shuffle_y\n  yarn.nodemanager.aux-services.spark_shuffle_x.classpath = /path/to/spark-x-path/*:/path/to/spark-x-config\n  yarn.nodemanager.aux-services.spark_shuffle_y.classpath = /path/to/spark-y-path/*:/path/to/spark-y-config\n```\n\nExample:\n```text\n<configuration>\n  <property>\n    <name>spark.shuffle.service.port</name>\n    <value>7001</value>\n  </property>\n  <property>\n    <name>spark.yarn.shuffle.service.metrics.namespace</name>\n    <value>sparkShuffleServiceX</value>\n  </property>\n</configuration>\n```\n\nExample:\n```text\nspark.shuffle.service.name = spark_shuffle_x\n  spark.shuffle.service.port = 7001\n```\n\nExample:\n```text\nspark.shuffle.service.name = spark_shuffle_y\n  spark.shuffle.service.port = <other value>\n```\n\nExample:\n```text\n$ export JAVA_HOME=/opt/openjdk-17\n$ ./bin/spark-submit --class path.to.your.Class \\\n    --master yarn \\\n    --conf spark.yarn.appMasterEnv.JAVA_HOME=/opt/openjdk-17 \\\n    --conf spark.executorEnv.JAVA_HOME=/opt/openjdk-17 \\\n    <app jar> [app options]\n```\n\nExample:\n```text\n$ export JAVA_HOME=/opt/openjdk-21\n$ ./bin/spark-submit --class path.to.your.Class \\\n    --master yarn \\\n    --archives path/to/openjdk-21.tar.gz \\\n    --conf spark.yarn.appMasterEnv.JAVA_HOME=./openjdk-21.tar.gz/openjdk-21 \\\n    --conf spark.executorEnv.JAVA_HOME=./openjdk-21.tar.gz/openjdk-21 \\\n    <app jar> [app options]\n```\n\n<|endofdoc|>","metadata":{"transformedAt":"2026-08-18T15:21:43.136Z","totalSectionsIncluded":1,"totalCodeBlocksIncluded":16,"totalLines":124,"estimatedTokens":12112}}32{"id":"doc-running_spark_on_kubernetes_spark_4_2_0_document-0d2cc87e","source":"documentation","title":"Running Spark on Kubernetes - Spark 4.2.0 Documentation","url":"https://spark.apache.org/docs/latest/running-on-kubernetes.html","text":"Running Spark on Kubernetes Security User Identity Volume Mounts Prerequisites How it works Submitting Applications to Kubernetes Docker Images Cluster Mode Client Mode Client Mode Networking Client Mode Executor Pod Garbage Collection Authentication Parameters IPv4 and IPv6 Dependency Management Secret Management Pod Template Using Kubernetes Volumes PVC-oriented executor pod allocation Local Storage Using RAM for local storage Introspection and Debugging Accessing Logs Accessing Driver UI Debugging Kubernetes Features Configuration File Contexts Namespaces RBAC Spark Application Management Future Work Configuration Spark Properties Pod template properties Pod Metadata Pod Spec Container spec Resource Allocation and Configuration Overview Resource Level Scheduling Overview Priority Scheduling Customized Kubernetes Schedulers for Spark on Kubernetes Using Volcano as Customized Scheduler for Spark on Kubernetes Prerequisites Build Usage Volcano Feature Step Volcano PodGroup Template Using Apache YuniKorn as Customized Scheduler for Spark on Kubernetes Prerequisites Get started Stage Level Scheduling Overview Spark can run on clusters managed by Kubernetes. This feature makes use of native Kubernetes scheduler that has been added to Spark. Security Security features like authentication are not enabled by default. When deploying a cluster that is open to the internet or an untrusted network, it’s important to secure access to the cluster to prevent unauthorized applications from running on the cluster. Please see Spark Security and the specific security sections in this doc before running Spark. User Identity Images built from the project provided Dockerfiles contain a default USER directive with a default UID of 185. This means that the resulting images will be running the Spark processes as this UID inside the container. Security conscious deployments should consider providing custom images with USER directives specifying their desired unprivileged UID and GID. The resulting UID should include the root group in its supplementary groups in order to be able to run the Spark executables. Users building their own images with the provided docker-image-tool.sh script can use the -u <uid> option to specify the desired UID. Alternatively the Pod Template feature can be used to add a Security Context with a runAsUser to the pods that Spark submits. This can be used to override the USER directives in the images themselves. Please bear in mind that this requires cooperation from your users and as such may not be a suitable solution for shared environments. Cluster administrators should use the Pod Security Admission Controller if they wish to limit the users that pods may run as. Volume Mounts As described later in this document under Using Kubernetes Volumes Spark on K8S provides configuration options that allow for mounting certain volume types into the driver and executor pods. In particular it allows for hostPath volumes which as described in the Kubernetes documentation have known security vulnerabilities. Cluster administrators should use the Pod Security Admission Controller to limit the ability to mount hostPath volumes appropriately for their environments. Prerequisites A running Kubernetes cluster at version >= 1.34 with access configured to it using kubectl. If you do not already have a working Kubernetes cluster, you may set up a test cluster on your local machine using minikube. We recommend using the latest release of minikube with the DNS addon enabled. Be aware that the default minikube configuration is not enough for running Spark applications. We recommend 3 CPUs and 4g of memory to be able to start a simple Spark application with a single executor. Check kubernetes-client library’s version of your Spark environment, and its compatibility with your Kubernetes cluster’s version. You must have appropriate permissions to list, create, edit and delete pods in your cluster. You can verify that you can list these resources by running kubectl auth can-i <list|create|edit|delete> pods. The service account credentials used by the driver pods must be allowed to create pods, services and configmaps. You must have Kubernetes DNS configured in your cluster. How it works spark-submit can be directly used to submit a Spark application to a Kubernetes cluster. The submission mechanism works as creates a Spark driver running within a Kubernetes pod. The driver creates executors which are also running within Kubernetes pods and connects to them, and executes application code. When the application completes, the executor pods terminate and are cleaned up, but the driver pod persists logs and remains in “completed” state in the Kubernetes API until it’s eventually garbage collected or manually cleaned up. Note that in the completed state, the driver pod does not use any computational or memory resources. The driver and executor pod scheduling is handled by Kubernetes. Communication to the Kubernetes API is done via fabric8. It is possible to schedule the driver and executor pods on a subset of available nodes through a node selector using the configuration property for it. It will be possible to use more advanced scheduling hints like node/pod affinities in a future release. Submitting Applications to Kubernetes Docker Images Kubernetes requires users to supply images that can be deployed into containers within pods. The images are built to be run in a container runtime environment that Kubernetes supports. Docker is a container runtime environment that is frequently used with Kubernetes. Spark (starting with version 2.3) ships with a Dockerfile that can be used for this purpose, or customized to match an individual application’s needs. It can be found in the kubernetes/dockerfiles/ directory. Spark also ships with a bin/docker-image-tool.sh script that can be used to build and publish the Docker images to use with the Kubernetes backend. Example usage is: $ ./bin/docker-image-tool.sh -r <repo> -t my-tag build $ ./bin/docker-image-tool.sh -r <repo> -t my-tag push This will build using the projects provided default Dockerfiles. To see more options available for customising the behaviour of this tool, including providing custom Dockerfiles, please run with the -h flag. By default bin/docker-image-tool.sh builds docker image for running JVM jobs. You need to opt-in to build additional language binding docker images. Example usage is # To build additional PySpark docker image $ ./bin/docker-image-tool.sh -r <repo> -t my-tag -p ./kubernetes/dockerfiles/spark/bindings/python/Dockerfile build # To build additional SparkR docker image $ ./bin/docker-image-tool.sh -r <repo> -t my-tag -R ./kubernetes/dockerfiles/spark/bindings/R/Dockerfile build You can also use the Apache Spark Docker images (such as apache/spark:<version>) directly. Cluster Mode To launch Spark Pi in cluster mode, $ ./bin/spark-submit \\ --master k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port> \\ --deploy-mode cluster \\ --name spark-pi \\ --class org.apache.spark.examples.SparkPi \\ --conf spark.executor.instances=5 \\ --conf spark.kubernetes.container.image=<spark-image> \\ local:///path/to/examples.jar The Spark master, specified either via passing the --master command line argument to spark-submit or by setting spark.master in the application’s configuration, must be a URL with the format k8s://<api_server_host>:<k8s-apiserver-port>. The port must always be specified, even if it’s the HTTPS port 443. Prefixing the master string with k8s:// will cause the Spark application to launch on the Kubernetes cluster, with the API server being contacted at api_server_url. If no HTTP protocol is specified in the URL, it defaults to https. For example, setting the master to k8s://example.com:443 is equivalent to setting it to k8s://https://example.com:443, but to connect without TLS on a different port, the master would be set to k8s://http://example.com:8080. In Kubernetes mode, the Spark application name that is specified by spark.app.name or the --name argument to spark-submit is used by default to name the Kubernetes resources created like drivers and executors. So, application names must consist of lower case alphanumeric characters, -, and . and must start and end with an alphanumeric character. If you have a Kubernetes cluster setup, one way to discover the apiserver URL is by executing kubectl cluster-info. $ kubectl cluster-info Kubernetes master is running at http://127.0.0.1:6443 In the above example, the specific Kubernetes cluster can be used with spark-submit by specifying --master k8s://http://127.0.0.1:6443 as an argument to spark-submit. Additionally, it is also possible to use the authenticating proxy, kubectl proxy to communicate to the Kubernetes API. The local proxy can be started by: $ kubectl proxy If the local proxy is running at , --master k8s://http://127.0.0.1:8001 can be used as the argument to spark-submit. Finally, notice that in the above example we specify a jar with a specific URI with a scheme of local://. This URI is the location of the example jar that is already in the Docker image. Client Mode Starting with Spark 2.4.0, it is possible to run Spark applications on Kubernetes in client mode. When your application runs in client mode, the driver can run inside a pod or on a physical host. When running an application in client mode, it is recommended to account for the following Mode Networking Spark executors must be able to connect to the Spark driver over a hostname and a port that is routable from the Spark executors. The specific network configuration that will be required for Spark to work in client mode will vary per setup. If you run your driver inside a Kubernetes pod, you can use a headless service to allow your driver pod to be routable from the executors by a stable hostname. When deploying your headless service, ensure that the service’s label selector will only match the driver pod and no other pods; it is recommended to assign your driver pod a sufficiently unique label and to use that label in the label selector of the headless service. Specify the driver’s hostname via spark.driver.host and your spark driver’s port to spark.driver.port. Client Mode Executor Pod Garbage Collection If you run your Spark driver in a pod, it is highly recommended to set spark.kubernetes.driver.pod.name to the name of that pod. When this property is set, the Spark scheduler will deploy the executor pods with an OwnerReference, which in turn will ensure that once the driver pod is deleted from the cluster, all of the application’s executor pods will also be deleted. The driver will look for a pod with the given name in the namespace specified by spark.kubernetes.namespace, and an OwnerReference pointing to that pod will be added to each executor pod’s OwnerReferences list. Be careful to avoid setting the OwnerReference to a pod that is not actually that driver pod, or else the executors may be terminated prematurely when the wrong pod is deleted. If your application is not running inside a pod, or if spark.kubernetes.driver.pod.name is not set when your application is actually running in a pod, keep in mind that the executor pods may not be properly deleted from the cluster when the application exits. The Spark scheduler attempts to delete these pods, but if the network request to the API server fails for any reason, these pods will remain in the cluster. The executor processes should exit when they cannot reach the driver, so the executor pods should not consume compute resources (cpu and memory) in the cluster after your application exits. You may use spark.kubernetes.executor.podNamePrefix to fully control the executor pod names. When this property is set, it’s highly recommended to make it unique across all jobs in the same namespace. Authentication Parameters Use the exact prefix spark.kubernetes.authenticate for Kubernetes authentication parameters in client mode. IPv4 and IPv6 Starting with 3.4.0, Spark supports additionally IPv6-only environment via IPv4/IPv6 dual-stack network feature which enables the allocation of both IPv4 and IPv6 addresses to Pods and Services. According to the K8s cluster capability, spark.kubernetes.driver.service.ipFamilyPolicy and spark.kubernetes.driver.service.ipFamilies can be one of SingleStack, PreferDualStack, and RequireDualStack and one of IPv4, IPv6, IPv4,IPv6, and IPv6,IPv4 respectively. By default, Spark uses spark.kubernetes.driver.service.ipFamilyPolicy=SingleStack and spark.kubernetes.driver.service.ipFamilies=IPv4. To use only IPv6, you can submit your jobs with the following. ... --conf spark.kubernetes.driver.service.ipFamilies=IPv6 \\ In DualStack environment, you may need java.net.preferIPv6Addresses=true for JVM and SPARK_PREFER_IPV6=true for Python additionally to use IPv6. Dependency Management If your application’s dependencies are all hosted in remote locations like HDFS or HTTP servers, they may be referred to by their appropriate remote URIs. Also, application dependencies can be pre-mounted into custom-built Docker images. Those dependencies can be added to the classpath by referencing them with local:// URIs and/or setting the SPARK_EXTRA_CLASSPATH environment variable in your Dockerfiles. The local:// scheme is also required when referring to dependencies in custom-built Docker images in spark-submit. We support dependencies from the submission client’s local file system using the file:// scheme or without a scheme (using a full path), where the destination should be a Hadoop compatible filesystem. A typical example of this using S3 is via passing the following --packages org.apache.hadoop:hadoop-aws:3.4.1 --conf spark.kubernetes.file.upload.path=s3a://<s3-bucket>/path --conf spark.hadoop.fs.s3a.access.key=... --conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem --conf spark.hadoop.fs.s3a.fast.upload=true --conf spark.hadoop.fs.s3a.secret.key=.... --conf spark.driver.extraJavaOptions=-Divy.cache.dir=/tmp -Divy.home=/tmp file:///full/path/to/app.jar The app jar file will be uploaded to the S3 and then when the driver is launched it will be downloaded to the driver pod and will be added to its classpath. Spark will generate a subdir under the upload path with a random name to avoid conflicts with spark apps running in parallel. User could manage the subdirs created according to his needs. The client scheme is supported for the application jar, and dependencies specified by properties spark.jars, spark.files and spark.archives. client-side dependencies will be uploaded to the given path with a flat directory structure so file names must be unique otherwise files will be overwritten. Also make sure in the derived k8s image default ivy dir has the required access rights or modify the settings as above. The latter is also important if you use --packages in cluster mode. Secret Management Kubernetes Secrets can be used to provide credentials for a Spark application to access secured services. To mount a user-specified secret into the driver container, users can use the configuration property of the form spark.kubernetes.driver.secrets.[SecretName]=<mount path>. Similarly, the configuration property of the form spark.kubernetes.executor.secrets.[SecretName]=<mount path> can be used to mount a user-specified secret into the executor containers. Note that it is assumed that the secret to be mounted is in the same namespace as that of the driver and executor pods. For example, to mount a secret named spark-secret onto the path /etc/secrets in both the driver and executor containers, add the following options to the spark-submit spark.kubernetes.driver.secrets.spark-secret=/etc/secrets --conf spark.kubernetes.executor.secrets.spark-secret=/etc/secrets To use a secret through an environment variable use the following options to the spark-submit spark.kubernetes.driver.secretKeyRef.ENV_NAME=name:key --conf spark.kubernetes.executor.secretKeyRef.ENV_NAME=name:key Pod Template Kubernetes allows defining pods from template files. Spark users can similarly use template files to define the driver or executor pod configurations that Spark configurations do not support. To do so, specify the spark properties spark.kubernetes.driver.podTemplateFile and spark.kubernetes.executor.podTemplateFile to point to files accessible to the spark-submit process. --conf spark.kubernetes.driver.podTemplateFile=s3a://bucket/driver.yml --conf spark.kubernetes.executor.podTemplateFile=s3a://bucket/executor.yml To allow the driver pod access the executor pod template file, the file will be automatically mounted onto a volume in the driver pod when it’s created. Spark does not do any validation after unmarshalling these template files and relies on the Kubernetes API server for validation. It is important to note that Spark is opinionated about certain pod configurations so there are values in the pod template that will always be overwritten by Spark. Therefore, users of this feature should note that specifying the pod template file only lets Spark start with a template pod instead of an empty pod during the pod-building process. For details, see the full list of pod template values that will be overwritten by spark. Pod template files can also define multiple containers. In such cases, you can use the spark properties spark.kubernetes.driver.podTemplateContainerName and spark.kubernetes.executor.podTemplateContainerName to indicate which container should be used as a basis for the driver or executor. If not specified, or if the container name is not valid, Spark will assume that the first container in the list will be the driver or executor container. Using Kubernetes Volumes Users can mount the following types of Kubernetes volumes into the driver and executor : mounts a file or directory from the host node’s filesystem into a pod. initially empty volume created when a pod is assigned to a node. an existing NFS(Network File System) into a pod. a PersistentVolume into a pod. see the Security section of this document for security issues related to volume mounts. To mount a volume of any of the types above into the driver pod, use the following configuration spark.kubernetes.driver.volumes.[VolumeType].[VolumeName].mount.path=<mount path> --conf spark.kubernetes.driver.volumes.[VolumeType].[VolumeName].mount.readOnly=<true|false> --conf spark.kubernetes.driver.volumes.[VolumeType].[VolumeName].mount.subPath=<mount subPath> Specifically, VolumeType can be one of the following , emptyDir, nfs and persistentVolumeClaim. VolumeName is the name you want to use for the volume under the volumes field in the pod specification. Each supported type of volumes may have some specific configuration options, which can be specified using configuration properties of the following [VolumeType].[VolumeName].options.[OptionName]=<value> For example, the server and path of a nfs with volume name images can be specified using the following =example.com spark.kubernetes.driver.volumes.nfs.images.options.path=/data And, the claim name of a persistentVolumeClaim with volume name checkpointpvc can be specified using the following =check-point-pvc-claim The configuration properties for mounting volumes into the executor pods use prefix spark.kubernetes.executor. instead of spark.kubernetes.driver.. For example, you can mount a dynamically-created persistent volume claim per executor by using OnDemand as a claim name and storageClass and sizeLimit options like the following. This is useful in case of Dynamic Allocation. spark.kubernetes.executor.volumes.persistentVolumeClaim.data.options.claimName=OnDemand spark.kubernetes.executor.volumes.persistentVolumeClaim.data.options.storageClass=gp spark.kubernetes.executor.volumes.persistentVolumeClaim.data.options.sizeLimit=500Gi spark.kubernetes.executor.volumes.persistentVolumeClaim.data.mount.path=/data spark.kubernetes.executor.volumes.persistentVolumeClaim.data.mount.readOnly=false For a complete list of available options for each supported type of volumes, please refer to the Spark Properties section below. PVC-oriented executor pod allocation Since disks are one of the important resource types, Spark driver provides a fine-grained control via a set of configurations. For example, by default, on-demand PVCs are owned by executors and the lifecycle of PVCs are tightly coupled with its owner executors. However, on-demand PVCs can be owned by driver and reused by another executors during the Spark job’s lifetime with the following options. This reduces the overhead of PVC creation and deletion. spark.kubernetes.driver.ownPersistentVolumeClaim=true spark.kubernetes.driver.reusePersistentVolumeClaim=true In addition, since Spark 3.4, Spark driver is able to do PVC-oriented executor allocation which means Spark counts the total number of created PVCs which the job can have, and holds on a new executor creation if the driver owns the maximum number of PVCs. This helps the transition of the existing PVC from one executor to another executor. spark.kubernetes.driver.waitToReusePersistentVolumeClaim=true Local Storage Spark supports using volumes to spill data during shuffles and other operations. To use a volume as local storage, the volume’s name should starts with spark-local-dir-, for spark.kubernetes.driver.volumes.[VolumeType].spark-local-dir-[VolumeName].mount.path=<mount path> --conf spark.kubernetes.driver.volumes.[VolumeType].spark-local-dir-[VolumeName].mount.readOnly=false Specifically, you can use persistent volume claims if the jobs require large shuffle and sorting operations in executors. spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.options.claimName=OnDemand spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.options.storageClass=gp spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.options.sizeLimit=500Gi spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.mount.path=/data spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.mount.readOnly=false To enable shuffle data recovery feature via the built-in KubernetesLocalDiskShuffleDataIO plugin, we need to have the following. You may want to enable spark.kubernetes.driver.waitToReusePersistentVolumeClaim additionally. spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.mount.path=/data/spark-x/executor-x spark.shuffle.sort.io.plugin.class=org.apache.spark.shuffle.KubernetesLocalDiskShuffleDataIO If no volume is set as local storage, Spark uses temporary scratch space to spill data to disk during shuffles and other operations. When using Kubernetes as the resource manager the pods will be created with an emptyDir volume mounted for each directory listed in spark.local.dir or the environment variable SPARK_LOCAL_DIRS . If no directories are explicitly specified then a default directory is created and configured appropriately. emptyDir volumes use the ephemeral storage feature of Kubernetes and do not persist beyond the life of the pod. Using RAM for local storage emptyDir volumes use the nodes backing storage for ephemeral storage by default, this behaviour may not be appropriate for some compute environments. For example if you have diskless nodes with remote storage mounted over a network, having lots of executors doing IO to this remote storage may actually degrade performance. In this case it may be desirable to set spark.kubernetes.local.dirs.tmpfs=true in your configuration which will cause the emptyDir volumes to be configured as tmpfs i.e. RAM backed volumes. When configured like this Spark’s local storage usage will count towards your pods memory usage therefore you may wish to increase your memory requests by increasing the value of spark.{driver,executor}.memoryOverheadFactor as appropriate. Introspection and Debugging These are the different ways in which you can investigate a running/completed Spark application, monitor progress, and take actions. Accessing Logs Logs can be accessed using the Kubernetes API and the kubectl CLI. When a Spark application is running, it’s possible to stream logs from the application using: $ kubectl -n=<namespace> logs -f <driver-pod-name> The same logs can also be accessed through the Kubernetes dashboard if installed on the cluster. When there exists a log collection system, you can expose it at Spark Driver Executors tab UI. For example, spark.ui.custom.executor.log.url='https://log-server/log?appId=&execId=' You can add additional custom variables to this url template, populated with the values of existing executor environment variables like spark.executorEnv.SPARK_EXECUTOR_ATTRIBUTE_YOUR_VAR='$(EXISTING_EXECUTOR_ENV_VAR)' spark.ui.custom.executor.log.url='https://log-server/log?appId=&execId=&your_var=' Accessing Driver UI The UI associated with any application can be accessed locally using kubectl port-forward. $ kubectl port-forward <driver-pod-name> Then, the Spark driver UI can be accessed on http://localhost:4040. Since Apache Spark 4.0.0, Driver UI provides a way to see driver logs via a new configuration. spark.driver.log.localDir=/tmp Then, the Spark driver UI can be accessed on http://localhost:4040/logs/. Optionally, the layout of log is configured by the following. spark.driver.log.layout=\"%m%n%ex\" Debugging There may be several kinds of failures. If the Kubernetes API server rejects the request made from spark-submit, or the connection is refused for a different reason, the submission logic should indicate the error encountered. However, if there are errors during the running of the application, often, the best way to investigate may be through the Kubernetes CLI. To get some basic information about the scheduling decisions made around the driver pod, you can run: $ kubectl describe pod <spark-driver-pod> If the pod has encountered a runtime error, the status can be probed further using: $ kubectl logs <spark-driver-pod> Status and logs of failed executor pods can be checked in similar ways. Finally, deleting the driver pod will clean up the entire spark application, including all executors, associated service, etc. The driver pod can be thought of as the Kubernetes representation of the Spark application. Kubernetes Features Configuration File Your Kubernetes config file typically lives under .label.* Spark will add additional labels specified by the spark configuration. annotations Adds the annotations from spark.kubernetes.{driver,executor}.annotation.* Spark will add additional annotations specified by the spark configuration. Pod Spec Pod spec keyModified valueDescription imagePullSecrets Adds image pull secrets from spark.kubernetes.container.image.pullSecrets Additional pull secrets will be added from the spark configuration to both executor pods. nodeSelector Adds node selectors from spark.kubernetes.node.selector.* Additional node selectors will be added from the spark configuration to both executor pods. restartPolicy \"never\" Spark assumes that both drivers and executors never restart. serviceAccount Value of spark.kubernetes.authenticate.driver.serviceAccountName Spark will override serviceAccount with the value of the spark configuration for only driver pods, and only if the spark configuration is specified. Executor pods will remain unaffected. serviceAccountName Value of spark.kubernetes.authenticate.driver.serviceAccountName Spark will override serviceAccountName with the value of the spark configuration for only driver pods, and only if the spark configuration is specified. Executor pods will remain unaffected. volumes Adds volumes from spark.kubernetes.{driver,executor}.volumes.[VolumeType].[VolumeName].mount.path Spark will add volumes as specified by the spark conf, as well as additional volumes necessary for passing spark conf and pod template files. Container spec The following affect the driver and executor containers. All other containers in the pod spec will be unaffected. Container spec keyModified valueDescription env Adds env variables from spark.kubernetes.driverEnv.[EnvironmentVariableName] Spark will add driver env variables from spark.kubernetes.driverEnv.[EnvironmentVariableName], and executor env variables from spark.executorEnv.[EnvironmentVariableName]. image Value of spark.kubernetes.{driver,executor}.container.image The image will be defined by the spark configurations. imagePullPolicy Value of spark.kubernetes.container.image.pullPolicy Spark will override the pull policy for both driver and executors. name See description The container name will be assigned by spark (\"spark-kubernetes-driver\" for the driver container, and \"spark-kubernetes-executor\" for each executor container) if not defined by the pod template. If the container is defined by the template, the template's name will be used. resources See description The cpu limits are set by spark.kubernetes.{driver,executor}.limit.cores. The cpu is set by spark.{driver,executor}.cores. The memory request and limit are set by summing the values of spark.{driver,executor}.memory and spark.{driver,executor}.memoryOverhead. Other resource limits are set by spark.{driver,executor}.resources.{resourceName}.* configs. volumeMounts Add volumes from spark.kubernetes.driver.volumes.[VolumeType].[VolumeName].mount.{path,readOnly} Spark will add volumes as specified by the spark conf, as well as additional volumes necessary for passing spark conf and pod template files. Resource Allocation and Configuration Overview Please make sure to have read the Custom Resource Scheduling and Configuration Overview section on the configuration page. This section only talks about the Kubernetes specific aspects of resource scheduling. The user is responsible to properly configuring the Kubernetes cluster to have the resources available and ideally isolate each resource per container so that a resource is not shared between multiple containers. If the resource is not isolated the user is responsible for writing a discovery script so that the resource is not shared between containers. See the Kubernetes documentation for specifics on configuring Kubernetes with custom resources. Spark automatically handles translating the Spark configs spark.{driver/executor}.resource.{resourceType} into the kubernetes configs as long as the Kubernetes resource type follows the Kubernetes device plugin format of vendor-domain/resourcetype. The user must specify the vendor using the spark.{driver/executor}.resource.{resourceType}.vendor config. The user does not need to explicitly add anything if you are using Pod templates. For reference and an example, you can see the Kubernetes documentation for scheduling GPUs. Spark only supports setting the resource limits. Kubernetes does not tell Spark the addresses of the resources allocated to each container. For that reason, the user must specify a discovery script that gets run by the executor on startup to discover what resources are available to that executor. You can find an example scripts in examples/src/main/scripts/getGpusResources.sh. The script must have execute permissions set and the user should setup permissions to not allow malicious users to modify it. The script should write to STDOUT a JSON string in the format of the ResourceInformation class. This has the resource name and an array of resource addresses available to just that executor. Resource Level Scheduling Overview There are several resource level scheduling features supported by Spark on Kubernetes. Priority Scheduling Kubernetes supports Pod priority by default. Spark on Kubernetes allows defining the priority of jobs by Pod template. The user can specify the priorityClassName in driver or executor Pod template spec section. Below is an example to show how to specify : v1 : spec: # Specify the priority in here Customized Kubernetes Schedulers for Spark on Kubernetes Spark allows users to specify a custom Kubernetes schedulers. Specify a scheduler name. Users can specify a custom scheduler using spark.kubernetes.scheduler.name or spark.kubernetes.{driver/executor}.scheduler.name configuration. Specify scheduler related configurations. To configure the custom scheduler the user can use Pod templates, add labels (spark.kubernetes.{driver,executor}.label.*), annotations (spark.kubernetes.{driver/executor}.annotation.*) or scheduler specific configurations (such as spark.kubernetes.scheduler.volcano.podGroupTemplateFile). Specify scheduler feature step. Users may also consider to use spark.kubernetes.{driver/executor}.pod.featureSteps to support more complex requirements, including but not limited additional Kubernetes custom resources for driver/executor scheduling. Set scheduler hints according to configuration or existing Pod info dynamically. Using Volcano as Customized Scheduler for Spark on Kubernetes Prerequisites Spark on Kubernetes with Volcano as a custom scheduler is supported since Spark v3.3.0 and Volcano v1.7.0. Below is an example to install Volcano 1.14.2: kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/v1.14.2/installer/volcano-development.yaml Build To create a Spark distribution along with Volcano support like those distributed by the Spark Downloads page, also see more in “Building Spark”: ./dev/make-distribution.sh --name custom-spark --pip --r --tgz -Psparkr -Phive -Phive-thriftserver -Pkubernetes -Pvolcano Usage Spark on Kubernetes allows using Volcano as a custom scheduler. Users can use Volcano to support more advanced resource scheduling, resource reservation, priority scheduling, and more. To use Volcano as a custom scheduler the user needs to specify the following configuration options: # Specify volcano scheduler and PodGroup template --conf spark.kubernetes.scheduler.name=volcano --conf spark.kubernetes.scheduler.volcano.podGroupTemplateFile=/path/to/podgroup-template.yaml # Specify driver/executor VolcanoFeatureStep --conf spark.kubernetes.driver.pod.featureSteps=org.apache.spark.deploy.k8s.features.VolcanoFeatureStep --conf spark.kubernetes.executor.pod.featureSteps=org.apache.spark.deploy.k8s.features.VolcanoFeatureStep Volcano Feature Step Volcano feature steps help users to create a Volcano PodGroup and set driver/executor pod annotation to link with this PodGroup. Note that currently only driver/job level PodGroup is supported in Volcano Feature Step. Volcano PodGroup Template Volcano defines PodGroup spec using CRD yaml. Similar to Pod template, Spark users can use Volcano PodGroup Template to define the PodGroup spec configurations. Below is an example of PodGroup : scheduling.volcano.sh/v1beta1 spec: # Specify minMember to 1 to make a driver pod # Specify minResources to support resource reservation (the driver pod resource and executors pod resource should be considered) # It is useful for ensource the available resources meet the minimum requirements of the Spark job and avoiding the # situation where drivers are scheduled, and then they are unable to schedule sufficient executors to progress. : \"2\" memory: \"3Gi\" # Specify the priority, help users to specify job priority in the queue during scheduling. # Specify the queue, indicates the resource queue which the job should be submitted to You have two options to provide the PodGroup template in spark. If both are provided, the podGroupTemplateFile will takes precedence. Use spark.kubernetes.scheduler.volcano.podGroupTemplateFile to point to files accessible to the spark-submit process --conf spark.kubernetes.scheduler.volcano.podGroupTemplateFile=/path/to/podgroup Use spark.kubernetes.scheduler.volcano.podGroupTemplateJson to pass the template directly in JSON --conf spark.kubernetes.scheduler.volcano.podGroupTemplateJson={\"spec\": {\"minMember\": 1,\"minResources\": {\"cpu\": \"2\",\"memory\": \"3Gi\"},\"priorityClassName\": \"system-node-critical\",\"queue\": \"default\"}} Using Apache YuniKorn as Customized Scheduler for Spark on Kubernetes Apache YuniKorn is a resource scheduler for Kubernetes that provides advanced batch scheduling capabilities, such as job queuing, resource fairness, min/max queue capacity and flexible job ordering policies. For available Apache YuniKorn features, please refer to core features. Prerequisites Install Apache repo add yunikorn https://apache.github.io/yunikorn-release helm repo update helm install yunikorn yunikorn/yunikorn --namespace yunikorn --version 1.8.0 --create-namespace --set embedAdmissionController=false The above steps will install YuniKorn v1.8.0 on an existing Kubernetes cluster. Get started Submit Spark jobs with the following extra spark.kubernetes.scheduler.name=yunikorn --conf spark.kubernetes.driver.label.queue=root.default --conf spark.kubernetes.executor.label.queue=root.default --conf spark.kubernetes.driver.annotation.yunikorn.apache.org/app-id={{APP_ID}} --conf spark.kubernetes.executor.annotation.yunikorn.apache.org/app-id={{APP_ID}} Note that {{APP_ID}} is the built-in variable that will be substituted with Spark job ID automatically. With the above configuration, the job will be scheduled by YuniKorn scheduler instead of the default Kubernetes scheduler. Stage Level Scheduling Overview Stage level scheduling is supported on dynamic allocation is allows users to specify different task resource requirements at the stage level and will use the same executors requested at startup. When dynamic allocation is allows users to specify task and executor resource requirements at the stage level and will request the extra executors. This also requires spark.dynamicAllocation.shuffleTracking.enabled to be enabled since Kubernetes doesn’t support an external shuffle service at this time. The order in which containers for different profiles is requested from Kubernetes is not guaranteed. Note that since dynamic allocation on Kubernetes requires the shuffle tracking feature, this means that executors from previous stages that used a different ResourceProfile may not idle timeout due to having shuffle data on them. This could result in using more cluster resources and in the worst case if there are no remaining resources on the Kubernetes cluster then Spark could potentially hang. You may consider looking at config spark.dynamicAllocation.shuffleTracking.timeout to set a timeout, but that could result in data having to be recomputed if the shuffle data is really needed. Note, there is a difference in the way pod template resources are handled between the base default profile and custom ResourceProfiles. Any resources specified in the pod template file will only be used with the base default profile. If you create custom ResourceProfiles be sure to include all necessary resources there since the resources from the template file will not be propagated to custom ResourceProfiles.\n\nExample:\n```text\n$ ./bin/docker-image-tool.sh -r <repo> -t my-tag build\n$ ./bin/docker-image-tool.sh -r <repo> -t my-tag push\n```\n\nExample:\n```text\n# To build additional PySpark docker image\n$ ./bin/docker-image-tool.sh -r <repo> -t my-tag -p ./kubernetes/dockerfiles/spark/bindings/python/Dockerfile build\n\n# To build additional SparkR docker image\n$ ./bin/docker-image-tool.sh -r <repo> -t my-tag -R ./kubernetes/dockerfiles/spark/bindings/R/Dockerfile build\n```\n\nExample:\n```text\n$ ./bin/spark-submit \\\n    --master k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port> \\\n    --deploy-mode cluster \\\n    --name spark-pi \\\n    --class org.apache.spark.examples.SparkPi \\\n    --conf spark.executor.instances=5 \\\n    --conf spark.kubernetes.container.image=<spark-image> \\\n    local:///path/to/examples.jar\n```\n\nExample:\n```text\n$ kubectl cluster-info\nKubernetes master is running at http://127.0.0.1:6443\n```\n\nExample:\n```text\n$ kubectl proxy\n```\n\nExample:\n```text\n...\n    --conf spark.kubernetes.driver.service.ipFamilies=IPv6 \\\n```\n\nExample:\n```text\n...\n--packages org.apache.hadoop:hadoop-aws:3.4.1\n--conf spark.kubernetes.file.upload.path=s3a://<s3-bucket>/path\n--conf spark.hadoop.fs.s3a.access.key=...\n--conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem\n--conf spark.hadoop.fs.s3a.fast.upload=true\n--conf spark.hadoop.fs.s3a.secret.key=....\n--conf spark.driver.extraJavaOptions=-Divy.cache.dir=/tmp -Divy.home=/tmp\nfile:///full/path/to/app.jar\n```\n\nExample:\n```text\n--conf spark.kubernetes.driver.secrets.spark-secret=/etc/secrets\n--conf spark.kubernetes.executor.secrets.spark-secret=/etc/secrets\n```\n\nExample:\n```text\n--conf spark.kubernetes.driver.secretKeyRef.ENV_NAME=name:key\n--conf spark.kubernetes.executor.secretKeyRef.ENV_NAME=name:key\n```\n\nExample:\n```text\n--conf spark.kubernetes.driver.podTemplateFile=s3a://bucket/driver.yml\n--conf spark.kubernetes.executor.podTemplateFile=s3a://bucket/executor.yml\n```\n\nExample:\n```text\n--conf spark.kubernetes.driver.volumes.[VolumeType].[VolumeName].mount.path=<mount path>\n--conf spark.kubernetes.driver.volumes.[VolumeType].[VolumeName].mount.readOnly=<true|false>\n--conf spark.kubernetes.driver.volumes.[VolumeType].[VolumeName].mount.subPath=<mount subPath>\n```\n\nExample:\n```text\nspark.kubernetes.driver.volumes.[VolumeType].[VolumeName].options.[OptionName]=<value>\n```\n\nExample:\n```text\nspark.kubernetes.driver.volumes.nfs.images.options.server=example.com\nspark.kubernetes.driver.volumes.nfs.images.options.path=/data\n```\n\nExample:\n```text\nspark.kubernetes.driver.volumes.persistentVolumeClaim.checkpointpvc.options.claimName=check-point-pvc-claim\n```\n\nExample:\n```text\nspark.kubernetes.executor.volumes.persistentVolumeClaim.data.options.claimName=OnDemand\nspark.kubernetes.executor.volumes.persistentVolumeClaim.data.options.storageClass=gp\nspark.kubernetes.executor.volumes.persistentVolumeClaim.data.options.sizeLimit=500Gi\nspark.kubernetes.executor.volumes.persistentVolumeClaim.data.mount.path=/data\nspark.kubernetes.executor.volumes.persistentVolumeClaim.data.mount.readOnly=false\n```\n\nExample:\n```text\nspark.kubernetes.driver.ownPersistentVolumeClaim=true\nspark.kubernetes.driver.reusePersistentVolumeClaim=true\n```\n\nExample:\n```text\nspark.kubernetes.driver.waitToReusePersistentVolumeClaim=true\n```\n\nExample:\n```text\n--conf spark.kubernetes.driver.volumes.[VolumeType].spark-local-dir-[VolumeName].mount.path=<mount path>\n--conf spark.kubernetes.driver.volumes.[VolumeType].spark-local-dir-[VolumeName].mount.readOnly=false\n```\n\nExample:\n```text\nspark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.options.claimName=OnDemand\nspark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.options.storageClass=gp\nspark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.options.sizeLimit=500Gi\nspark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.mount.path=/data\nspark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.mount.readOnly=false\n```\n\nExample:\n```text\nspark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.mount.path=/data/spark-x/executor-x\nspark.shuffle.sort.io.plugin.class=org.apache.spark.shuffle.KubernetesLocalDiskShuffleDataIO\n```\n\nExample:\n```text\n$ kubectl -n=<namespace> logs -f <driver-pod-name>\n```\n\nExample:\n```text\nspark.ui.custom.executor.log.url='https://log-server/log?appId=&execId='\n```\n\nExample:\n```text\nspark.executorEnv.SPARK_EXECUTOR_ATTRIBUTE_YOUR_VAR='$(EXISTING_EXECUTOR_ENV_VAR)'\nspark.ui.custom.executor.log.url='https://log-server/log?appId=&execId=&your_var='\n```\n\nExample:\n```text\n$ kubectl port-forward <driver-pod-name> 4040:4040\n```\n\nExample:\n```text\nspark.driver.log.localDir=/tmp\n```\n\nExample:\n```text\nspark.driver.log.layout=\"%m%n%ex\"\n```\n\nExample:\n```text\n$ kubectl describe pod <spark-driver-pod>\n```\n\nExample:\n```text\n$ kubectl logs <spark-driver-pod>\n```\n\nExample:\n```text\n--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark\n```\n\nExample:\n```text\n$ kubectl create serviceaccount spark\n```\n\nExample:\n```text\n$ kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount=default:spark --namespace=default\n```\n\nExample:\n```text\n$ spark-submit --kill spark:spark-pi-1547948636094-driver --master k8s://https://192.168.2.8:8443\n```\n\nExample:\n```text\n$ spark-submit --status spark:spark-pi-1547948636094-driver --master  k8s://https://192.168.2.8:8443\n```\n\nExample:\n```text\n$ spark-submit --kill spark:spark-pi* --master  k8s://https://192.168.2.8:8443\n```\n\nExample:\n```text\napiVersion: v1\nKind: Pod\nmetadata:\n  labels:\n    template-label-key: driver-template-label-value\nspec:\n  # Specify the priority in here\n  priorityClassName: system-node-critical\n  containers:\n  - name: test-driver-container\n    image: will-be-overwritten\n```\n\nExample:\n```text\nkubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/v1.14.2/installer/volcano-development.yaml\n```\n\nExample:\n```text\n./dev/make-distribution.sh --name custom-spark --pip --r --tgz -Psparkr -Phive -Phive-thriftserver -Pkubernetes -Pvolcano\n```\n\nExample:\n```text\n# Specify volcano scheduler and PodGroup template\n--conf spark.kubernetes.scheduler.name=volcano\n--conf spark.kubernetes.scheduler.volcano.podGroupTemplateFile=/path/to/podgroup-template.yaml\n# Specify driver/executor VolcanoFeatureStep\n--conf spark.kubernetes.driver.pod.featureSteps=org.apache.spark.deploy.k8s.features.VolcanoFeatureStep\n--conf spark.kubernetes.executor.pod.featureSteps=org.apache.spark.deploy.k8s.features.VolcanoFeatureStep\n```\n\nExample:\n```text\napiVersion: scheduling.volcano.sh/v1beta1\nkind: PodGroup\nspec:\n  # Specify minMember to 1 to make a driver pod\n  minMember: 1\n  # Specify minResources to support resource reservation (the driver pod resource and executors pod resource should be considered)\n  # It is useful for ensource the available resources meet the minimum requirements of the Spark job and avoiding the\n  # situation where drivers are scheduled, and then they are unable to schedule sufficient executors to progress.\n  minResources:\n    cpu: \"2\"\n    memory: \"3Gi\"\n  # Specify the priority, help users to specify job priority in the queue during scheduling.\n  priorityClassName: system-node-critical\n  # Specify the queue, indicates the resource queue which the job should be submitted to\n  queue: default\n```\n\nExample:\n```text\n--conf spark.kubernetes.scheduler.volcano.podGroupTemplateFile=/path/to/podgroup\n```\n\nExample:\n```text\n--conf spark.kubernetes.scheduler.volcano.podGroupTemplateJson={\"spec\": {\"minMember\": 1,\"minResources\": {\"cpu\": \"2\",\"memory\": \"3Gi\"},\"priorityClassName\": \"system-node-critical\",\"queue\": \"default\"}}\n```\n\nExample:\n```text\nhelm repo add yunikorn https://apache.github.io/yunikorn-release\nhelm repo update\nhelm install yunikorn yunikorn/yunikorn --namespace yunikorn --version 1.8.0 --create-namespace --set embedAdmissionController=false\n```\n\nExample:\n```text\n--conf spark.kubernetes.scheduler.name=yunikorn\n--conf spark.kubernetes.driver.label.queue=root.default\n--conf spark.kubernetes.executor.label.queue=root.default\n--conf spark.kubernetes.driver.annotation.yunikorn.apache.org/app-id={{APP_ID}}\n--conf spark.kubernetes.executor.annotation.yunikorn.apache.org/app-id={{APP_ID}}\n```\n\n<|endofdoc|>","metadata":{"transformedAt":"2026-08-18T15:21:43.158Z","totalSectionsIncluded":1,"totalCodeBlocksIncluded":43,"totalLines":293,"estimatedTokens":11774}}33{"id":"doc-spark_streaming_spark_4_2_0_documentation-9d3976a2","source":"documentation","title":"Spark Streaming - Spark 4.2.0 Documentation","url":"https://spark.apache.org/docs/latest/streaming-programming-guide.html","text":"Example:\n```python\nfrom pyspark import SparkContext\nfrom pyspark.streaming import StreamingContext\n\n# Create a local StreamingContext with two working thread and batch interval of 1 second\nsc = SparkContext(\"local[2]\", \"NetworkWordCount\")\nssc = StreamingContext(sc, 1)\n```\n\nExample:\n```python\n# Create a DStream that will connect to hostname:port, like localhost:9999\nlines = ssc.socketTextStream(\"localhost\", 9999)\n```\n\nExample:\n```python\n# Split each line into words\nwords = lines.flatMap(lambda line: line.split(\" \"))\n```\n\nExample:\n```python\n# Count each word in each batch\npairs = words.map(lambda word: (word, 1))\nwordCounts = pairs.reduceByKey(lambda x, y: x + y)\n\n# Print the first ten elements of each RDD generated in this DStream to the console\nwordCounts.pprint()\n```\n\nExample:\n```python\nssc.start()             # Start the computation\nssc.awaitTermination()  # Wait for the computation to terminate\n```\n\nExample:\n```scala\nimport org.apache.spark._\nimport org.apache.spark.streaming._\nimport org.apache.spark.streaming.StreamingContext._ // not necessary since Spark 1.3\n\n// Create a local StreamingContext with two working thread and batch interval of 1 second.\n// The master requires 2 cores to prevent a starvation scenario.\n\nval conf = new SparkConf().setMaster(\"local[2]\").setAppName(\"NetworkWordCount\")\nval ssc = new StreamingContext(conf, Seconds(1))\n```\n\nExample:\n```scala\n// Create a DStream that will connect to hostname:port, like localhost:9999\nval lines = ssc.socketTextStream(\"localhost\", 9999)\n```\n\nExample:\n```scala\n// Split each line into words\nval words = lines.flatMap(_.split(\" \"))\n```\n\nExample:\n```scala\nimport org.apache.spark.streaming.StreamingContext._ // not necessary since Spark 1.3\n// Count each word in each batch\nval pairs = words.map(word => (word, 1))\nval wordCounts = pairs.reduceByKey(_ + _)\n\n// Print the first ten elements of each RDD generated in this DStream to the console\nwordCounts.print()\n```\n\nExample:\n```scala\nssc.start()             // Start the computation\nssc.awaitTermination()  // Wait for the computation to terminate\n```\n\nExample:\n```java\nimport org.apache.spark.*;\nimport org.apache.spark.api.java.function.*;\nimport org.apache.spark.streaming.*;\nimport org.apache.spark.streaming.api.java.*;\nimport scala.Tuple2;\n\n// Create a local StreamingContext with two working thread and batch interval of 1 second\nSparkConf conf = new SparkConf().setMaster(\"local[2]\").setAppName(\"NetworkWordCount\");\nJavaStreamingContext jssc = new JavaStreamingContext(conf, Durations.seconds(1));\n```\n\nExample:\n```java\n// Create a DStream that will connect to hostname:port, like localhost:9999\nJavaReceiverInputDStream<String> lines = jssc.socketTextStream(\"localhost\", 9999);\n```\n\nExample:\n```java\n// Split each line into words\nJavaDStream<String> words = lines.flatMap(x -> Arrays.asList(x.split(\" \")).iterator());\n```\n\nExample:\n```java\n// Count each word in each batch\nJavaPairDStream<String, Integer> pairs = words.mapToPair(s -> new Tuple2<>(s, 1));\nJavaPairDStream<String, Integer> wordCounts = pairs.reduceByKey((i1, i2) -> i1 + i2);\n\n// Print the first ten elements of each RDD generated in this DStream to the console\nwordCounts.print();\n```\n\nExample:\n```java\njssc.start();              // Start the computation\njssc.awaitTermination();   // Wait for the computation to terminate\n```\n\nExample:\n```bash\n$ nc -lk 9999\n```\n\nExample:\n```bash\n$ ./bin/spark-submit examples/src/main/python/streaming/network_wordcount.py localhost 9999\n```\n\nExample:\n```bash\n$ ./bin/run-example streaming.NetworkWordCount localhost 9999\n```\n\nExample:\n```bash\n$ ./bin/run-example streaming.JavaNetworkWordCount localhost 9999\n```\n\nExample:\n```bash\n# TERMINAL 1:\n# Running Netcat\n\n$ nc -lk 9999\n\nhello world\n\n\n\n...\n```\n\nExample:\n```bash\n# TERMINAL 2: RUNNING network_wordcount.py\n\n$ ./bin/spark-submit examples/src/main/python/streaming/network_wordcount.py localhost 9999\n...\n-------------------------------------------\nTime: 2014-10-14 15:25:21\n-------------------------------------------\n(hello,1)\n(world,1)\n...\n```\n\nExample:\n```bash\n# TERMINAL 2: RUNNING NetworkWordCount\n\n$ ./bin/run-example streaming.NetworkWordCount localhost 9999\n...\n-------------------------------------------\nTime: 1357008430000 ms\n-------------------------------------------\n(hello,1)\n(world,1)\n...\n```\n\nExample:\n```bash\n# TERMINAL 2: RUNNING JavaNetworkWordCount\n\n$ ./bin/run-example streaming.JavaNetworkWordCount localhost 9999\n...\n-------------------------------------------\nTime: 1357008430000 ms\n-------------------------------------------\n(hello,1)\n(world,1)\n...\n```\n\nExample:\n```text\n<dependency>\n    <groupId>org.apache.spark</groupId>\n    <artifactId>spark-streaming_2.13</artifactId>\n    <version>4.2.0</version>\n    <scope>provided</scope>\n</dependency>\n```\n\nExample:\n```text\nlibraryDependencies += \"org.apache.spark\" % \"spark-streaming_2.13\" % \"4.2.0\" % \"provided\"\n```\n\nExample:\n```python\nfrom pyspark import SparkContext\nfrom pyspark.streaming import StreamingContext\n\nsc = SparkContext(master, appName)\nssc = StreamingContext(sc, 1)\n```\n\nExample:\n```scala\nimport org.apache.spark._\nimport org.apache.spark.streaming._\n\nval conf = new SparkConf().setAppName(appName).setMaster(master)\nval ssc = new StreamingContext(conf, Seconds(1))\n```\n\nExample:\n```scala\nimport org.apache.spark.streaming._\n\nval sc = ...                // existing SparkContext\nval ssc = new StreamingContext(sc, Seconds(1))\n```\n\nExample:\n```java\nimport org.apache.spark.*;\nimport org.apache.spark.streaming.api.java.*;\n\nSparkConf conf = new SparkConf().setAppName(appName).setMaster(master);\nJavaStreamingContext ssc = new JavaStreamingContext(conf, new Duration(1000));\n```\n\nExample:\n```java\nimport org.apache.spark.streaming.api.java.*;\n\nJavaSparkContext sc = ...   //existing JavaSparkContext\nJavaStreamingContext ssc = new JavaStreamingContext(sc, Durations.seconds(1));\n```\n\nExample:\n```python\nstreamingContext.textFileStream(dataDirectory)\n```\n\nExample:\n```scala\nstreamingContext.fileStream[KeyClass, ValueClass, InputFormatClass](dataDirectory)\n```\n\nExample:\n```java\nstreamingContext.fileStream<KeyClass, ValueClass, InputFormatClass>(dataDirectory);\n```\n\nExample:\n```java\nstreamingContext.textFileStream(dataDirectory);\n```\n\nExample:\n```python\ndef updateFunction(newValues, runningCount):\n    if runningCount is None:\n        runningCount = 0\n    return sum(newValues, runningCount)  # add the new values with the previous running count to get the new count\n```\n\nExample:\n```python\nrunningCounts = pairs.updateStateByKey(updateFunction)\n```\n\nExample:\n```scala\ndef updateFunction(newValues: Seq[Int], runningCount: Option[Int]): Option[Int] = {\n    val newCount = ...  // add the new values with the previous running count to get the new count\n    Some(newCount)\n}\n```\n\nExample:\n```scala\nval runningCounts = pairs.updateStateByKey[Int](updateFunction _)\n```\n\nExample:\n```java\nFunction2<List<Integer>, Optional<Integer>, Optional<Integer>> updateFunction =\n  (values, state) -> {\n    Integer newSum = ...  // add the new values with the previous running count to get the new count\n    return Optional.of(newSum);\n  };\n```\n\nExample:\n```java\nJavaPairDStream<String, Integer> runningCounts = pairs.updateStateByKey(updateFunction);\n```\n\nExample:\n```python\nspamInfoRDD = sc.pickleFile(...)  # RDD containing spam information\n\n# join data stream with spam information to do data cleaning\ncleanedDStream = wordCounts.transform(lambda rdd: rdd.join(spamInfoRDD).filter(...))\n```\n\nExample:\n```scala\nval spamInfoRDD = ssc.sparkContext.newAPIHadoopRDD(...) // RDD containing spam information\n\nval cleanedDStream = wordCounts.transform { rdd =>\n  rdd.join(spamInfoRDD).filter(...) // join data stream with spam information to do data cleaning\n  ...\n}\n```\n\nExample:\n```java\nimport org.apache.spark.streaming.api.java.*;\n// RDD containing spam information\nJavaPairRDD<String, Double> spamInfoRDD = jssc.sparkContext().newAPIHadoopRDD(...);\n\nJavaPairDStream<String, Integer> cleanedDStream = wordCounts.transform(rdd -> {\n  rdd.join(spamInfoRDD).filter(...); // join data stream with spam information to do data cleaning\n  ...\n});\n```\n\nExample:\n```python\n# Reduce last 30 seconds of data, every 10 seconds\nwindowedWordCounts = pairs.reduceByKeyAndWindow(lambda x, y: x + y, lambda x, y: x - y, 30, 10)\n```\n\nExample:\n```scala\n// Reduce last 30 seconds of data, every 10 seconds\nval windowedWordCounts = pairs.reduceByKeyAndWindow((a:Int,b:Int) => (a + b), Seconds(30), Seconds(10))\n```\n\nExample:\n```java\n// Reduce last 30 seconds of data, every 10 seconds\nJavaPairDStream<String, Integer> windowedWordCounts = pairs.reduceByKeyAndWindow((i1, i2) -> i1 + i2, Durations.seconds(30), Durations.seconds(10));\n```\n\nExample:\n```python\nstream1 = ...\nstream2 = ...\njoinedStream = stream1.join(stream2)\n```\n\nExample:\n```scala\nval stream1: DStream[String, String] = ...\nval stream2: DStream[String, String] = ...\nval joinedStream = stream1.join(stream2)\n```\n\nExample:\n```java\nJavaPairDStream<String, String> stream1 = ...\nJavaPairDStream<String, String> stream2 = ...\nJavaPairDStream<String, Tuple2<String, String>> joinedStream = stream1.join(stream2);\n```\n\nExample:\n```python\nwindowedStream1 = stream1.window(20)\nwindowedStream2 = stream2.window(60)\njoinedStream = windowedStream1.join(windowedStream2)\n```\n\nExample:\n```scala\nval windowedStream1 = stream1.window(Seconds(20))\nval windowedStream2 = stream2.window(Minutes(1))\nval joinedStream = windowedStream1.join(windowedStream2)\n```\n\nExample:\n```java\nJavaPairDStream<String, String> windowedStream1 = stream1.window(Durations.seconds(20));\nJavaPairDStream<String, String> windowedStream2 = stream2.window(Durations.minutes(1));\nJavaPairDStream<String, Tuple2<String, String>> joinedStream = windowedStream1.join(windowedStream2);\n```\n\nExample:\n```python\ndataset = ... # some RDD\nwindowedStream = stream.window(20)\njoinedStream = windowedStream.transform(lambda rdd: rdd.join(dataset))\n```\n\nExample:\n```scala\nval dataset: RDD[String, String] = ...\nval windowedStream = stream.window(Seconds(20))...\nval joinedStream = windowedStream.transform { rdd => rdd.join(dataset) }\n```\n\nExample:\n```java\nJavaPairRDD<String, String> dataset = ...\nJavaPairDStream<String, String> windowedStream = stream.window(Durations.seconds(20));\nJavaPairDStream<String, String> joinedStream = windowedStream.transform(rdd -> rdd.join(dataset));\n```\n\nExample:\n```python\ndef sendRecord(rdd):\n    connection = createNewConnection()  # executed at the driver\n    rdd.foreach(lambda record: connection.send(record))\n    connection.close()\n\ndstream.foreachRDD(sendRecord)\n```\n\nExample:\n```scala\ndstream.foreachRDD { rdd =>\n  val connection = createNewConnection()  // executed at the driver\n  rdd.foreach { record =>\n    connection.send(record) // executed at the worker\n  }\n}\n```\n\nExample:\n```java\ndstream.foreachRDD(rdd -> {\n  Connection connection = createNewConnection(); // executed at the driver\n  rdd.foreach(record -> {\n    connection.send(record); // executed at the worker\n  });\n});\n```\n\nExample:\n```python\ndef sendRecord(record):\n    connection = createNewConnection()\n    connection.send(record)\n    connection.close()\n\ndstream.foreachRDD(lambda rdd: rdd.foreach(sendRecord))\n```\n\nExample:\n```scala\ndstream.foreachRDD { rdd =>\n  rdd.foreach { record =>\n    val connection = createNewConnection()\n    connection.send(record)\n    connection.close()\n  }\n}\n```\n\nExample:\n```java\ndstream.foreachRDD(rdd -> {\n  rdd.foreach(record -> {\n    Connection connection = createNewConnection();\n    connection.send(record);\n    connection.close();\n  });\n});\n```\n\nExample:\n```python\ndef sendPartition(iter):\n    connection = createNewConnection()\n    for record in iter:\n        connection.send(record)\n    connection.close()\n\ndstream.foreachRDD(lambda rdd: rdd.foreachPartition(sendPartition))\n```\n\nExample:\n```scala\ndstream.foreachRDD { rdd =>\n  rdd.foreachPartition { partitionOfRecords =>\n    val connection = createNewConnection()\n    partitionOfRecords.foreach(record => connection.send(record))\n    connection.close()\n  }\n}\n```\n\nExample:\n```java\ndstream.foreachRDD(rdd -> {\n  rdd.foreachPartition(partitionOfRecords -> {\n    Connection connection = createNewConnection();\n    while (partitionOfRecords.hasNext()) {\n      connection.send(partitionOfRecords.next());\n    }\n    connection.close();\n  });\n});\n```\n\nExample:\n```python\ndef sendPartition(iter):\n    # ConnectionPool is a static, lazily initialized pool of connections\n    connection = ConnectionPool.getConnection()\n    for record in iter:\n        connection.send(record)\n    # return to the pool for future reuse\n    ConnectionPool.returnConnection(connection)\n\ndstream.foreachRDD(lambda rdd: rdd.foreachPartition(sendPartition))\n```\n\nExample:\n```scala\ndstream.foreachRDD { rdd =>\n  rdd.foreachPartition { partitionOfRecords =>\n    // ConnectionPool is a static, lazily initialized pool of connections\n    val connection = ConnectionPool.getConnection()\n    partitionOfRecords.foreach(record => connection.send(record))\n    ConnectionPool.returnConnection(connection)  // return to the pool for future reuse\n  }\n}\n```\n\nExample:\n```java\ndstream.foreachRDD(rdd -> {\n  rdd.foreachPartition(partitionOfRecords -> {\n    // ConnectionPool is a static, lazily initialized pool of connections\n    Connection connection = ConnectionPool.getConnection();\n    while (partitionOfRecords.hasNext()) {\n      connection.send(partitionOfRecords.next());\n    }\n    ConnectionPool.returnConnection(connection); // return to the pool for future reuse\n  });\n});\n```\n\nExample:\n```python\n# Lazily instantiated global instance of SparkSession\ndef getSparkSessionInstance(sparkConf):\n    if (\"sparkSessionSingletonInstance\" not in globals()):\n        globals()[\"sparkSessionSingletonInstance\"] = SparkSession \\\n            .builder \\\n            .config(conf=sparkConf) \\\n            .getOrCreate()\n    return globals()[\"sparkSessionSingletonInstance\"]\n\n...\n\n# DataFrame operations inside your streaming program\n\nwords = ... # DStream of strings\n\ndef process(time, rdd):\n    print(\"========= %s =========\" % str(time))\n    try:\n        # Get the singleton instance of SparkSession\n        spark = getSparkSessionInstance(rdd.context.getConf())\n\n        # Convert RDD[String] to RDD[Row] to DataFrame\n        rowRdd = rdd.map(lambda w: Row(word=w))\n        wordsDataFrame = spark.createDataFrame(rowRdd)\n\n        # Creates a temporary view using the DataFrame\n        wordsDataFrame.createOrReplaceTempView(\"words\")\n\n        # Do word count on table using SQL and print it\n        wordCountsDataFrame = spark.sql(\"select word, count(*) as total from words group by word\")\n        wordCountsDataFrame.show()\n    except:\n        pass\n\nwords.foreachRDD(process)\n```\n\nExample:\n```scala\n/** DataFrame operations inside your streaming program */\n\nval words: DStream[String] = ...\n\nwords.foreachRDD { rdd =>\n\n  // Get the singleton instance of SparkSession\n  val spark = SparkSession.builder.config(rdd.sparkContext.getConf).getOrCreate()\n  import spark.implicits._\n\n  // Convert RDD[String] to DataFrame\n  val wordsDataFrame = rdd.toDF(\"word\")\n\n  // Create a temporary view\n  wordsDataFrame.createOrReplaceTempView(\"words\")\n\n  // Do word count on DataFrame using SQL and print it\n  val wordCountsDataFrame =\n    spark.sql(\"select word, count(*) as total from words group by word\")\n  wordCountsDataFrame.show()\n}\n```\n\nExample:\n```java\n/** Java Bean class for converting RDD to DataFrame */\npublic class JavaRow implements java.io.Serializable {\n  private String word;\n\n  public String getWord() {\n    return word;\n  }\n\n  public void setWord(String word) {\n    this.word = word;\n  }\n}\n\n...\n\n/** DataFrame operations inside your streaming program */\n\nJavaDStream<String> words = ...\n\nwords.foreachRDD((rdd, time) -> {\n  // Get the singleton instance of SparkSession\n  SparkSession spark = SparkSession.builder().config(rdd.sparkContext().getConf()).getOrCreate();\n\n  // Convert RDD[String] to RDD[case class] to DataFrame\n  JavaRDD<JavaRow> rowRDD = rdd.map(word -> {\n    JavaRow record = new JavaRow();\n    record.setWord(word);\n    return record;\n  });\n  DataFrame wordsDataFrame = spark.createDataFrame(rowRDD, JavaRow.class);\n\n  // Creates a temporary view using the DataFrame\n  wordsDataFrame.createOrReplaceTempView(\"words\");\n\n  // Do word count on table using SQL and print it\n  DataFrame wordCountsDataFrame =\n    spark.sql(\"select word, count(*) as total from words group by word\");\n  wordCountsDataFrame.show();\n});\n```\n\nExample:\n```python\n# Function to create and setup a new StreamingContext\ndef functionToCreateContext():\n    sc = SparkContext(...)  # new context\n    ssc = StreamingContext(...)\n    lines = ssc.socketTextStream(...)  # create DStreams\n    ...\n    ssc.checkpoint(checkpointDirectory)  # set checkpoint directory\n    return ssc\n\n# Get StreamingContext from checkpoint data or create a new one\ncontext = StreamingContext.getOrCreate(checkpointDirectory, functionToCreateContext)\n\n# Do additional setup on context that needs to be done,\n# irrespective of whether it is being started or restarted\ncontext. ...\n\n# Start the context\ncontext.start()\ncontext.awaitTermination()\n```\n\nExample:\n```scala\n// Function to create and setup a new StreamingContext\ndef functionToCreateContext(): StreamingContext = {\n  val ssc = new StreamingContext(...)   // new context\n  val lines = ssc.socketTextStream(...) // create DStreams\n  ...\n  ssc.checkpoint(checkpointDirectory)   // set checkpoint directory\n  ssc\n}\n\n// Get StreamingContext from checkpoint data or create a new one\nval context = StreamingContext.getOrCreate(checkpointDirectory, functionToCreateContext _)\n\n// Do additional setup on context that needs to be done,\n// irrespective of whether it is being started or restarted\ncontext. ...\n\n// Start the context\ncontext.start()\ncontext.awaitTermination()\n```\n\nExample:\n```java\n// Create a factory object that can create and setup a new JavaStreamingContext\nJavaStreamingContextFactory contextFactory = new JavaStreamingContextFactory() {\n  @Override public JavaStreamingContext create() {\n    JavaStreamingContext jssc = new JavaStreamingContext(...);  // new context\n    JavaDStream<String> lines = jssc.socketTextStream(...);     // create DStreams\n    ...\n    jssc.checkpoint(checkpointDirectory);                       // set checkpoint directory\n    return jssc;\n  }\n};\n\n// Get JavaStreamingContext from checkpoint data or create a new one\nJavaStreamingContext context = JavaStreamingContext.getOrCreate(checkpointDirectory, contextFactory);\n\n// Do additional setup on context that needs to be done,\n// irrespective of whether it is being started or restarted\ncontext. ...\n\n// Start the context\ncontext.start();\ncontext.awaitTermination();\n```\n\nExample:\n```python\ndef getWordExcludeList(sparkContext):\n    if (\"wordExcludeList\" not in globals()):\n        globals()[\"wordExcludeList\"] = sparkContext.broadcast([\"a\", \"b\", \"c\"])\n    return globals()[\"wordExcludeList\"]\n\ndef getDroppedWordsCounter(sparkContext):\n    if (\"droppedWordsCounter\" not in globals()):\n        globals()[\"droppedWordsCounter\"] = sparkContext.accumulator(0)\n    return globals()[\"droppedWordsCounter\"]\n\ndef echo(time, rdd):\n    # Get or register the excludeList Broadcast\n    excludeList = getWordExcludeList(rdd.context)\n    # Get or register the droppedWordsCounter Accumulator\n    droppedWordsCounter = getDroppedWordsCounter(rdd.context)\n\n    # Use excludeList to drop words and use droppedWordsCounter to count them\n    def filterFunc(wordCount):\n        if wordCount[0] in excludeList.value:\n            droppedWordsCounter.add(wordCount[1])\n            False\n        else:\n            True\n\n    counts = \"Counts at time %s %s\" % (time, rdd.filter(filterFunc).collect())\n\nwordCounts.foreachRDD(echo)\n```\n\nExample:\n```scala\nobject WordExcludeList {\n\n  @volatile private var instance: Broadcast[Seq[String]] = null\n\n  def getInstance(sc: SparkContext): Broadcast[Seq[String]] = {\n    if (instance == null) {\n      synchronized {\n        if (instance == null) {\n          val wordExcludeList = Seq(\"a\", \"b\", \"c\")\n          instance = sc.broadcast(wordExcludeList)\n        }\n      }\n    }\n    instance\n  }\n}\n\nobject DroppedWordsCounter {\n\n  @volatile private var instance: LongAccumulator = null\n\n  def getInstance(sc: SparkContext): LongAccumulator = {\n    if (instance == null) {\n      synchronized {\n        if (instance == null) {\n          instance = sc.longAccumulator(\"DroppedWordsCounter\")\n        }\n      }\n    }\n    instance\n  }\n}\n\nwordCounts.foreachRDD { (rdd: RDD[(String, Int)], time: Time) =>\n  // Get or register the excludeList Broadcast\n  val excludeList = WordExcludeList.getInstance(rdd.sparkContext)\n  // Get or register the droppedWordsCounter Accumulator\n  val droppedWordsCounter = DroppedWordsCounter.getInstance(rdd.sparkContext)\n  // Use excludeList to drop words and use droppedWordsCounter to count them\n  val counts = rdd.filter { case (word, count) =>\n    if (excludeList.value.contains(word)) {\n      droppedWordsCounter.add(count)\n      false\n    } else {\n      true\n    }\n  }.collect().mkString(\"[\", \", \", \"]\")\n  val output = \"Counts at time \" + time + \" \" + counts\n})\n```\n\nExample:\n```java\nclass JavaWordExcludeList {\n\n  private static volatile Broadcast<List<String>> instance = null;\n\n  public static Broadcast<List<String>> getInstance(JavaSparkContext jsc) {\n    if (instance == null) {\n      synchronized (JavaWordExcludeList.class) {\n        if (instance == null) {\n          List<String> wordExcludeList = Arrays.asList(\"a\", \"b\", \"c\");\n          instance = jsc.broadcast(wordExcludeList);\n        }\n      }\n    }\n    return instance;\n  }\n}\n\nclass JavaDroppedWordsCounter {\n\n  private static volatile LongAccumulator instance = null;\n\n  public static LongAccumulator getInstance(JavaSparkContext jsc) {\n    if (instance == null) {\n      synchronized (JavaDroppedWordsCounter.class) {\n        if (instance == null) {\n          instance = jsc.sc().longAccumulator(\"DroppedWordsCounter\");\n        }\n      }\n    }\n    return instance;\n  }\n}\n\nwordCounts.foreachRDD((rdd, time) -> {\n  // Get or register the excludeList Broadcast\n  Broadcast<List<String>> excludeList = JavaWordExcludeList.getInstance(new JavaSparkContext(rdd.context()));\n  // Get or register the droppedWordsCounter Accumulator\n  LongAccumulator droppedWordsCounter = JavaDroppedWordsCounter.getInstance(new JavaSparkContext(rdd.context()));\n  // Use excludeList to drop words and use droppedWordsCounter to count them\n  String counts = rdd.filter(wordCount -> {\n    if (excludeList.value().contains(wordCount._1())) {\n      droppedWordsCounter.add(wordCount._2());\n      return false;\n    } else {\n      return true;\n    }\n  }).collect().toString();\n  String output = \"Counts at time \" + time + \" \" + counts;\n}\n```\n\nExample:\n```python\nnumStreams = 5\nkafkaStreams = [KafkaUtils.createStream(...) for _ in range (numStreams)]\nunifiedStream = streamingContext.union(*kafkaStreams)\nunifiedStream.pprint()\n```\n\nExample:\n```scala\nval numStreams = 5\nval kafkaStreams = (1 to numStreams).map { i => KafkaUtils.createStream(...) }\nval unifiedStream = streamingContext.union(kafkaStreams)\nunifiedStream.print()\n```\n\nExample:\n```java\nint numStreams = 5;\nList<JavaPairDStream<String, String>> kafkaStreams = new ArrayList<>(numStreams);\nfor (int i = 0; i < numStreams; i++) {\n  kafkaStreams.add(KafkaUtils.createStream(...));\n}\nJavaPairDStream<String, String> unifiedStream = streamingContext.union(kafkaStreams.get(0), kafkaStreams.subList(1, kafkaStreams.size()));\nunifiedStream.print();\n```\n\nExample:\n```text\ndstream.foreachRDD { (rdd, time) =>\n  rdd.foreachPartition { partitionIterator =>\n    val partitionId = TaskContext.get.partitionId()\n    val uniqueId = generateUniqueId(time.milliseconds, partitionId)\n    // use this uniqueId to transactionally commit the data in partitionIterator\n  }\n}\n```\n\n<|endofdoc|>","metadata":{"transformedAt":"2026-08-18T15:21:43.165Z","totalSectionsIncluded":0,"totalCodeBlocksIncluded":80,"totalLines":916,"estimatedTokens":5959}}34{"id":"doc-quickstart_pandas_api_on_spark_pyspark_4_2_0_doc-09201037","source":"documentation","title":"Quickstart: Pandas API on Spark — PySpark 4.2.0 documentation","url":"https://spark.apache.org/docs/latest/api/python/getting_started/quickstart_ps.html","text":"Site Navigation Overview Getting Started Tutorials User Guide API Reference Development More Migration Guides GitHub PyPI Section Navigation Installation Connect API on Spark Testing PySpark Getting Started API on Spark API on Spark# This is a short introduction to pandas API on Spark, geared mainly for new users. This notebook shows you some key differences between pandas and pandas API on Spark. You can run this examples by yourself in ‘Live API on Spark’ at the quickstart page. Customarily, we import pandas API on Spark as follows: [1]: import pandas as pd import numpy as np import pyspark.pandas as ps from pyspark.sql import SparkSession Object Creation# Creating a pandas-on-Spark Series by passing a list of values, letting pandas API on Spark create a default integer index: [2]: s = ps.Series([1, 3, 5, np.nan, 6, 8]) [3]: s [3]: 0 1.0 1 3.0 2 5.0 3 NaN 4 6.0 5 8.0 Creating a pandas-on-Spark DataFrame by passing a dict of objects that can be converted to series-like. [4]: psdf = ps.DataFrame( {'a': [1, 2, 3, 4, 5, 6], 'b': [100, 200, 300, 400, 500, 600], 'c': [\"one\", \"two\", \"three\", \"four\", \"five\", \"six\"]}, index=[10, 20, 30, 40, 50, 60]) [5]: psdf [5]: a b c 10 1 100 one 20 2 200 two 30 3 300 three 40 4 400 four 50 5 500 five 60 6 600 six Creating a pandas DataFrame by passing a numpy array, with a datetime index and labeled columns: [6]: dates = pd.date_range('20130101', periods=6) [7]: dates [7]: DatetimeIndex(['2013-01-01', '2013-01-02', '2013-01-03', '2013-01-04', '2013-01-05', '2013-01-06'], dtype='datetime64[ns]', freq='D') [8]: pdf = pd.DataFrame(np.random.randn(6, 4), index=dates, columns=list('ABCD')) [9]: pdf [9]: A B C D 2013-01-01 0.912558 -0.795645 -0.289115 0.187606 2013-01-02 -0.059703 -1.233897 0.316625 -1.226828 2013-01-03 0.332871 -1.262010 -0.434844 -0.579920 2013-01-04 0.924016 -1.022019 -0.405249 -1.036021 2013-01-05 -0.772209 -1.228099 0.068901 0.896679 2013-01-06 1.485582 -0.709306 -0.202637 -0.248766 Now, this pandas DataFrame can be converted to a pandas-on-Spark DataFrame [10]: psdf = ps.from_pandas(pdf) [11]: type(psdf) [11]: pyspark.pandas.frame.DataFrame It looks and behaves the same as a pandas DataFrame. [12]: psdf [12]: A B C D 2013-01-01 0.912558 -0.795645 -0.289115 0.187606 2013-01-02 -0.059703 -1.233897 0.316625 -1.226828 2013-01-03 0.332871 -1.262010 -0.434844 -0.579920 2013-01-04 0.924016 -1.022019 -0.405249 -1.036021 2013-01-05 -0.772209 -1.228099 0.068901 0.896679 2013-01-06 1.485582 -0.709306 -0.202637 -0.248766 Also, it is possible to create a pandas-on-Spark DataFrame from Spark DataFrame easily. Creating a Spark DataFrame from pandas DataFrame [13]: spark = SparkSession.builder.getOrCreate() [14]: sdf = spark.createDataFrame(pdf) [15]: sdf.show() +--------------------+-------------------+--------------------+--------------------+ | A| B| C| D| +--------------------+-------------------+--------------------+--------------------+ | 0.91255803205208|-0.7956452608556638|-0.28911463069772175| 0.18760566615081622| |-0.05970271470242...| -1.233896949308984| 0.3166246451758431| -1.2268284000402265| | 0.33287106947536615|-1.2620100816441786| -0.4348444277082644| -0.5799199651437185| | 0.9240158461589916|-1.0220190956326003| -0.4052488880650239| -1.0360212104348547| | -0.7722090016558953|-1.2280986385313222| 0.0689011451939635| 0.8966790729426755| | 1.4855822995785612|-0.7093056426018517| -0.2026366848847041|-0.24876619876451092| +--------------------+-------------------+--------------------+--------------------+ Creating pandas-on-Spark DataFrame from Spark DataFrame. [16]: psdf = sdf.pandas_api() [17]: psdf [17]: A B C D 0 0.912558 -0.795645 -0.289115 0.187606 1 -0.059703 -1.233897 0.316625 -1.226828 2 0.332871 -1.262010 -0.434844 -0.579920 3 0.924016 -1.022019 -0.405249 -1.036021 4 -0.772209 -1.228099 0.068901 0.896679 5 1.485582 -0.709306 -0.202637 -0.248766 Having specific dtypes . Types that are common to both Spark and pandas are currently supported. [18]: psdf.dtypes [18]: A float64 B float64 C float64 D float64 Here is how to show top rows from the frame below. Note that the data in a Spark dataframe does not preserve the natural order by default. The natural order can be preserved by setting compute.ordered_head option but it causes a performance overhead with sorting internally. [19]: psdf.head() [19]: A B C D 0 0.912558 -0.795645 -0.289115 0.187606 1 -0.059703 -1.233897 0.316625 -1.226828 2 0.332871 -1.262010 -0.434844 -0.579920 3 0.924016 -1.022019 -0.405249 -1.036021 4 -0.772209 -1.228099 0.068901 0.896679 Displaying the index, columns, and the underlying numpy data. [20]: psdf.index [20]: Index([0, 1, 2, 3, 4, 5], dtype='int64') [21]: psdf.columns [21]: Index(['A', 'B', 'C', 'D'], dtype='object') [22]: psdf.to_numpy() [22]: array([[ 0.91255803, -0.79564526, -0.28911463, 0.18760567], [-0.05970271, -1.23389695, 0.31662465, -1.2268284 ], [ 0.33287107, -1.26201008, -0.43484443, -0.57991997], [ 0.92401585, -1.0220191 , -0.40524889, -1.03602121], [-0.772209 , -1.22809864, 0.06890115, 0.89667907], [ 1.4855823 , -0.70930564, -0.20263668, -0.2487662 ]]) Showing a quick statistic summary of your data [23]: psdf.describe() [23]: A B C D count 6.000000 6.000000 6.000000 6.000000 mean 0.470519 -1.041829 -0.157720 -0.334542 std 0.809428 0.241511 0.294520 0.793014 min -0.772209 -1.262010 -0.434844 -1.226828 25% -0.059703 -1.233897 -0.405249 -1.036021 50% 0.332871 -1.228099 -0.289115 -0.579920 75% 0.924016 -0.795645 0.068901 0.187606 max 1.485582 -0.709306 0.316625 0.896679 Transposing your data [24]: psdf.T [24]: 0 1 2 3 4 5 A 0.912558 -0.059703 0.332871 0.924016 -0.772209 1.485582 B -0.795645 -1.233897 -1.262010 -1.022019 -1.228099 -0.709306 C -0.289115 0.316625 -0.434844 -0.405249 0.068901 -0.202637 D 0.187606 -1.226828 -0.579920 -1.036021 0.896679 -0.248766 Sorting by its index [25]: psdf.sort_index(ascending=False) [25]: A B C D 5 1.485582 -0.709306 -0.202637 -0.248766 4 -0.772209 -1.228099 0.068901 0.896679 3 0.924016 -1.022019 -0.405249 -1.036021 2 0.332871 -1.262010 -0.434844 -0.579920 1 -0.059703 -1.233897 0.316625 -1.226828 0 0.912558 -0.795645 -0.289115 0.187606 Sorting by value [26]: psdf.sort_values(by='B') [26]: A B C D 2 0.332871 -1.262010 -0.434844 -0.579920 1 -0.059703 -1.233897 0.316625 -1.226828 4 -0.772209 -1.228099 0.068901 0.896679 3 0.924016 -1.022019 -0.405249 -1.036021 0 0.912558 -0.795645 -0.289115 0.187606 5 1.485582 -0.709306 -0.202637 -0.248766 Missing Data# Pandas API on Spark primarily uses the value np.nan to represent missing data. It is by default not included in computations. [27]: pdf1 = pdf.reindex(index=dates[0:4], columns=list(pdf.columns) + ['E']) [28]: pdf1.loc[dates[0]:dates[1], 'E'] = 1 [29]: psdf1 = ps.from_pandas(pdf1) [30]: psdf1 [30]: A B C D E 2013-01-01 0.912558 -0.795645 -0.289115 0.187606 1.0 2013-01-02 -0.059703 -1.233897 0.316625 -1.226828 1.0 2013-01-03 0.332871 -1.262010 -0.434844 -0.579920 NaN 2013-01-04 0.924016 -1.022019 -0.405249 -1.036021 NaN To drop any rows that have missing data. [31]: psdf1.dropna(how='any') [31]: A B C D E 2013-01-01 0.912558 -0.795645 -0.289115 0.187606 1.0 2013-01-02 -0.059703 -1.233897 0.316625 -1.226828 1.0 Filling missing data. [32]: psdf1.fillna(value=5) [32]: A B C D E 2013-01-01 0.912558 -0.795645 -0.289115 0.187606 1.0 2013-01-02 -0.059703 -1.233897 0.316625 -1.226828 1.0 2013-01-03 0.332871 -1.262010 -0.434844 -0.579920 5.0 2013-01-04 0.924016 -1.022019 -0.405249 -1.036021 5.0 Operations# Stats# Performing a descriptive statistic: [33]: psdf.mean() [33]: A 0.470519 B -1.041829 C -0.157720 D -0.334542 Spark Configurations# Various configurations in PySpark could be applied internally in pandas API on Spark. For example, you can enable Arrow optimization to hugely speed up internal pandas conversion. See also PySpark Usage Guide for Pandas with Apache Arrow in PySpark documentation. [34]: prev = spark.conf.get(\"spark.sql.execution.arrow.pyspark.enabled\") # Keep its default value. ps.set_option(\"compute.default_index_type\", \"distributed\") # Use default index prevent overhead. import warnings warnings.filterwarnings(\"ignore\") # Ignore warnings coming from Arrow optimizations. [35]: spark.conf.set(\"spark.sql.execution.arrow.pyspark.enabled\", True) %timeit ps.range(300000).to_pandas() 900 ms ± 186 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) [36]: spark.conf.set(\"spark.sql.execution.arrow.pyspark.enabled\", False) %timeit ps.range(300000).to_pandas() 3.08 s ± 227 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) [37]: ps.reset_option(\"compute.default_index_type\") spark.conf.set(\"spark.sql.execution.arrow.pyspark.enabled\", prev) # Set its default value back. Grouping# By “group by” we are referring to a process involving one or more of the following the data into groups based on some criteria Applying a function to each group independently Combining the results into a data structure [38]: psdf = ps.DataFrame({'A': ['foo', 'bar', 'foo', 'bar', 'foo', 'bar', 'foo', 'foo'], 'B': ['one', 'one', 'two', 'three', 'two', 'two', 'one', 'three'], 'C': np.random.randn(8), 'D': np.random.randn(8)}) [39]: psdf [39]: A B C D 0 foo one 1.039632 -0.571950 1 bar one 0.972089 1.085353 2 foo two -1.931621 -2.579164 3 bar three -0.654371 -0.340704 4 foo two -0.157080 0.893736 5 bar two 0.882795 0.024978 6 foo one -0.149384 0.201667 7 foo three -1.355136 0.693883 Grouping and then applying the sum() function to the resulting groups. [40]: psdf.groupby('A').sum() [40]: B C D A bar onethreetwo 1.200513 0.769627 foo onetwotwoonethree -2.553589 -1.361828 Grouping by multiple columns forms a hierarchical index, and again we can apply the sum function. [41]: psdf.groupby(['A', 'B']).sum() [41]: C D A B foo one 0.890248 -0.370283 two -2.088701 -1.685428 bar three -0.654371 -0.340704 foo three -1.355136 0.693883 bar two 0.882795 0.024978 one 0.972089 1.085353 Plotting# [42]: pser = pd.Series(np.random.randn(1000), index=pd.date_range('1/1/2000', periods=1000)) [43]: psser = ps.Series(pser) [44]: psser = psser.cummax() [45]: psser.plot() On a DataFrame, the plot() method is a convenience to plot all of the columns with labels: [46]: pdf = pd.DataFrame(np.random.randn(1000, 4), index=pser.index, columns=['A', 'B', 'C', 'D']) [47]: psdf = ps.from_pandas(pdf) [48]: psdf = psdf.cummax() [49]: psdf.plot() For more details, Plotting documentation. Getting data in/out# CSV# CSV is straightforward and easy to use. See here to write a CSV file and here to read a CSV file. [50]: psdf.to_csv('foo.csv') ps.read_csv('foo.csv').head(10) [50]: A B C D 0 -1.187097 -0.134645 0.377094 -0.627217 1 0.331741 0.166218 0.377094 -0.627217 2 0.331741 0.439450 0.377094 0.365970 3 0.621620 0.439450 1.190180 0.365970 4 0.621620 0.439450 1.190180 0.365970 5 2.169198 1.069183 1.395642 0.365970 6 2.755738 1.069183 1.395642 1.045868 7 2.755738 1.069183 1.395642 1.045868 8 2.755738 1.069183 1.395642 1.045868 9 2.755738 1.508732 1.395642 1.556933 Parquet# Parquet is an efficient and compact file format to read and write faster. See here to write a Parquet file and here to read a Parquet file. [51]: psdf.to_parquet('bar.parquet') ps.read_parquet('bar.parquet').head(10) [51]: A B C D 0 -1.187097 -0.134645 0.377094 -0.627217 1 0.331741 0.166218 0.377094 -0.627217 2 0.331741 0.439450 0.377094 0.365970 3 0.621620 0.439450 1.190180 0.365970 4 0.621620 0.439450 1.190180 0.365970 5 2.169198 1.069183 1.395642 0.365970 6 2.755738 1.069183 1.395642 1.045868 7 2.755738 1.069183 1.395642 1.045868 8 2.755738 1.069183 1.395642 1.045868 9 2.755738 1.508732 1.395642 1.556933 Spark IO# In addition, pandas API on Spark fully supports Spark’s various datasources such as ORC and an external datasource. See here to write it to the specified datasource and here to read it from the datasource. [52]: psdf.spark.to_spark_io('zoo.orc', format=\"orc\") ps.read_spark_io('zoo.orc', format=\"orc\").head(10) [52]: A B C D 0 -1.187097 -0.134645 0.377094 -0.627217 1 0.331741 0.166218 0.377094 -0.627217 2 0.331741 0.439450 0.377094 0.365970 3 0.621620 0.439450 1.190180 0.365970 4 0.621620 0.439450 1.190180 0.365970 5 2.169198 1.069183 1.395642 0.365970 6 2.755738 1.069183 1.395642 1.045868 7 2.755738 1.069183 1.395642 1.045868 8 2.755738 1.069183 1.395642 1.045868 9 2.755738 1.508732 1.395642 1.556933 See the Input/Output documentation for more details. previous Connect next Testing PySpark On this page Object Creation Missing Data Operations Stats Spark Configurations Grouping Plotting Getting data in/out CSV Parquet Spark IO Show Source\n\nExample:\n```text\n[1]:\n```\n\nExample:\n```text\nimport pandas as pd\nimport numpy as np\nimport pyspark.pandas as ps\nfrom pyspark.sql import SparkSession\n```\n\nExample:\n```text\n[2]:\n```\n\nExample:\n```text\ns = ps.Series([1, 3, 5, np.nan, 6, 8])\n```\n\nExample:\n```text\n[3]:\n```\n\nExample:\n```text\ns\n```\n\nExample:\n```text\n0    1.0\n1    3.0\n2    5.0\n3    NaN\n4    6.0\n5    8.0\ndtype: float64\n```\n\nExample:\n```text\n[4]:\n```\n\nExample:\n```text\npsdf = ps.DataFrame(\n    {'a': [1, 2, 3, 4, 5, 6],\n     'b': [100, 200, 300, 400, 500, 600],\n     'c': [\"one\", \"two\", \"three\", \"four\", \"five\", \"six\"]},\n    index=[10, 20, 30, 40, 50, 60])\n```\n\nExample:\n```text\n[5]:\n```\n\nExample:\n```text\npsdf\n```\n\nExample:\n```text\n[6]:\n```\n\nExample:\n```text\ndates = pd.date_range('20130101', periods=6)\n```\n\nExample:\n```text\n[7]:\n```\n\nExample:\n```text\ndates\n```\n\nExample:\n```text\nDatetimeIndex(['2013-01-01', '2013-01-02', '2013-01-03', '2013-01-04',\n               '2013-01-05', '2013-01-06'],\n              dtype='datetime64[ns]', freq='D')\n```\n\nExample:\n```text\n[8]:\n```\n\nExample:\n```text\npdf = pd.DataFrame(np.random.randn(6, 4), index=dates, columns=list('ABCD'))\n```\n\nExample:\n```text\n[9]:\n```\n\nExample:\n```text\npdf\n```\n\nExample:\n```text\n[10]:\n```\n\nExample:\n```text\npsdf = ps.from_pandas(pdf)\n```\n\nExample:\n```text\n[11]:\n```\n\nExample:\n```text\ntype(psdf)\n```\n\nExample:\n```text\npyspark.pandas.frame.DataFrame\n```\n\nExample:\n```text\n[12]:\n```\n\nExample:\n```text\n[13]:\n```\n\nExample:\n```text\nspark = SparkSession.builder.getOrCreate()\n```\n\nExample:\n```text\n[14]:\n```\n\nExample:\n```text\nsdf = spark.createDataFrame(pdf)\n```\n\nExample:\n```text\n[15]:\n```\n\nExample:\n```text\nsdf.show()\n```\n\nExample:\n```text\n+--------------------+-------------------+--------------------+--------------------+\n|                   A|                  B|                   C|                   D|\n+--------------------+-------------------+--------------------+--------------------+\n|    0.91255803205208|-0.7956452608556638|-0.28911463069772175| 0.18760566615081622|\n|-0.05970271470242...| -1.233896949308984|  0.3166246451758431| -1.2268284000402265|\n| 0.33287106947536615|-1.2620100816441786| -0.4348444277082644| -0.5799199651437185|\n|  0.9240158461589916|-1.0220190956326003| -0.4052488880650239| -1.0360212104348547|\n| -0.7722090016558953|-1.2280986385313222|  0.0689011451939635|  0.8966790729426755|\n|  1.4855822995785612|-0.7093056426018517| -0.2026366848847041|-0.24876619876451092|\n+--------------------+-------------------+--------------------+--------------------+\n```\n\nExample:\n```text\n[16]:\n```\n\nExample:\n```text\npsdf = sdf.pandas_api()\n```\n\nExample:\n```text\n[17]:\n```\n\nExample:\n```text\n[18]:\n```\n\nExample:\n```text\npsdf.dtypes\n```\n\nExample:\n```text\nA    float64\nB    float64\nC    float64\nD    float64\ndtype: object\n```\n\nExample:\n```text\n[19]:\n```\n\nExample:\n```text\npsdf.head()\n```\n\nExample:\n```text\n[20]:\n```\n\nExample:\n```text\npsdf.index\n```\n\nExample:\n```text\nIndex([0, 1, 2, 3, 4, 5], dtype='int64')\n```\n\nExample:\n```text\n[21]:\n```\n\nExample:\n```text\npsdf.columns\n```\n\nExample:\n```text\nIndex(['A', 'B', 'C', 'D'], dtype='object')\n```\n\nExample:\n```text\n[22]:\n```\n\nExample:\n```text\npsdf.to_numpy()\n```\n\nExample:\n```text\narray([[ 0.91255803, -0.79564526, -0.28911463,  0.18760567],\n       [-0.05970271, -1.23389695,  0.31662465, -1.2268284 ],\n       [ 0.33287107, -1.26201008, -0.43484443, -0.57991997],\n       [ 0.92401585, -1.0220191 , -0.40524889, -1.03602121],\n       [-0.772209  , -1.22809864,  0.06890115,  0.89667907],\n       [ 1.4855823 , -0.70930564, -0.20263668, -0.2487662 ]])\n```\n\nExample:\n```text\n[23]:\n```\n\nExample:\n```text\npsdf.describe()\n```\n\nExample:\n```text\n[24]:\n```\n\nExample:\n```text\npsdf.T\n```\n\nExample:\n```text\n[25]:\n```\n\nExample:\n```text\npsdf.sort_index(ascending=False)\n```\n\nExample:\n```text\n[26]:\n```\n\nExample:\n```text\npsdf.sort_values(by='B')\n```\n\nExample:\n```text\n[27]:\n```\n\nExample:\n```text\npdf1 = pdf.reindex(index=dates[0:4], columns=list(pdf.columns) + ['E'])\n```\n\nExample:\n```text\n[28]:\n```\n\nExample:\n```text\npdf1.loc[dates[0]:dates[1], 'E'] = 1\n```\n\nExample:\n```text\n[29]:\n```\n\nExample:\n```text\npsdf1 = ps.from_pandas(pdf1)\n```\n\nExample:\n```text\n[30]:\n```\n\nExample:\n```text\npsdf1\n```\n\nExample:\n```text\n[31]:\n```\n\nExample:\n```text\npsdf1.dropna(how='any')\n```\n\nExample:\n```text\n[32]:\n```\n\nExample:\n```text\npsdf1.fillna(value=5)\n```\n\nExample:\n```text\n[33]:\n```\n\nExample:\n```text\npsdf.mean()\n```\n\nExample:\n```text\nA    0.470519\nB   -1.041829\nC   -0.157720\nD   -0.334542\ndtype: float64\n```\n\nExample:\n```text\n[34]:\n```\n\nExample:\n```text\nprev = spark.conf.get(\"spark.sql.execution.arrow.pyspark.enabled\")  # Keep its default value.\nps.set_option(\"compute.default_index_type\", \"distributed\")  # Use default index prevent overhead.\nimport warnings\nwarnings.filterwarnings(\"ignore\")  # Ignore warnings coming from Arrow optimizations.\n```\n\nExample:\n```text\n[35]:\n```\n\nExample:\n```text\nspark.conf.set(\"spark.sql.execution.arrow.pyspark.enabled\", True)\n%timeit ps.range(300000).to_pandas()\n```\n\nExample:\n```text\n900 ms ± 186 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n```\n\nExample:\n```text\n[36]:\n```\n\nExample:\n```text\nspark.conf.set(\"spark.sql.execution.arrow.pyspark.enabled\", False)\n%timeit ps.range(300000).to_pandas()\n```\n\nExample:\n```text\n3.08 s ± 227 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n```\n\nExample:\n```text\n[37]:\n```\n\nExample:\n```text\nps.reset_option(\"compute.default_index_type\")\nspark.conf.set(\"spark.sql.execution.arrow.pyspark.enabled\", prev)  # Set its default value back.\n```\n\nExample:\n```text\n[38]:\n```\n\nExample:\n```text\npsdf = ps.DataFrame({'A': ['foo', 'bar', 'foo', 'bar',\n                          'foo', 'bar', 'foo', 'foo'],\n                    'B': ['one', 'one', 'two', 'three',\n                          'two', 'two', 'one', 'three'],\n                    'C': np.random.randn(8),\n                    'D': np.random.randn(8)})\n```\n\nExample:\n```text\n[39]:\n```\n\nExample:\n```text\n[40]:\n```\n\nExample:\n```text\npsdf.groupby('A').sum()\n```\n\nExample:\n```text\n[41]:\n```\n\nExample:\n```text\npsdf.groupby(['A', 'B']).sum()\n```\n\nExample:\n```text\n[42]:\n```\n\nExample:\n```text\npser = pd.Series(np.random.randn(1000),\n                 index=pd.date_range('1/1/2000', periods=1000))\n```\n\nExample:\n```text\n[43]:\n```\n\nExample:\n```text\npsser = ps.Series(pser)\n```\n\nExample:\n```text\n[44]:\n```\n\nExample:\n```text\npsser = psser.cummax()\n```\n\nExample:\n```text\n[45]:\n```\n\nExample:\n```text\npsser.plot()\n```\n\nExample:\n```text\n[46]:\n```\n\nExample:\n```text\npdf = pd.DataFrame(np.random.randn(1000, 4), index=pser.index,\n                   columns=['A', 'B', 'C', 'D'])\n```\n\nExample:\n```text\n[47]:\n```\n\nExample:\n```text\n[48]:\n```\n\nExample:\n```text\npsdf = psdf.cummax()\n```\n\nExample:\n```text\n[49]:\n```\n\nExample:\n```text\npsdf.plot()\n```\n\nExample:\n```text\n[50]:\n```\n\nExample:\n```text\npsdf.to_csv('foo.csv')\nps.read_csv('foo.csv').head(10)\n```\n\nExample:\n```text\n[51]:\n```\n\nExample:\n```text\npsdf.to_parquet('bar.parquet')\nps.read_parquet('bar.parquet').head(10)\n```\n\nExample:\n```text\n[52]:\n```\n\nExample:\n```text\npsdf.spark.to_spark_io('zoo.orc', format=\"orc\")\nps.read_spark_io('zoo.orc', format=\"orc\").head(10)\n```\n\n<|endofdoc|>","metadata":{"transformedAt":"2026-08-18T15:21:43.167Z","totalSectionsIncluded":1,"totalCodeBlocksIncluded":111,"totalLines":611,"estimatedTokens":4904}}35{"id":"doc-configuration_spark_4_2_0_documentation-0aa651da","source":"documentation","title":"Configuration - Spark 4.2.0 Documentation","url":"https://spark.apache.org/docs/latest/configuration.html","text":"Spark Configuration Spark Properties Dynamically Loading Spark Properties Viewing Spark Properties Available Properties Application Properties Runtime Environment Shuffle Behavior Spark UI Compression and Serialization Memory Management Execution Behavior Executor Metrics Networking Scheduling Barrier Execution Mode Dynamic Allocation Thread Configurations Spark Connect Server Configuration Security Spark SQL Runtime SQL Configuration Static SQL Configuration Spark Streaming SparkR (deprecated) GraphX Cluster Managers YARN Kubernetes Standalone Mode Environment Variables Configuring Logging Plain Text Logging Structured Logging Querying Structured Logs with Spark SQL Overriding configuration directory Inheriting Hadoop Cluster Configuration Custom Hadoop/Hive Configuration Custom Resource Scheduling and Configuration Overview Stage Level Scheduling Overview Push-based shuffle overview External Shuffle service(server) side configuration options Client side configuration options Spark provides three locations to configure the properties control most application parameters and can be set by using a SparkConf object, or through Java system properties. Environment variables can be used to set per-machine settings, such as the IP address, through the conf/spark-env.sh script on each node. Logging can be configured through log4j2.properties. Spark Properties Spark properties control most application settings and are configured separately for each application. These properties can be set directly on a SparkConf passed to your SparkContext. SparkConf allows you to configure some of the common properties (e.g. master URL and application name), as well as arbitrary key-value pairs through the set() method. For example, we could initialize an application with two threads as that we run with local[2], meaning two threads - which represents “minimal” parallelism, which can help detect bugs that only exist when we run in a distributed context. val conf = new SparkConf() .amount 0 Amount of a particular resource type to use on the driver. If this is used, you must also specify the spark.driver.resource.{resourceName}.discoveryScript for the driver to find the resource on startup. 3.0.0 spark.driver.resource.{resourceName}.discoveryScript None A script for the driver to run to discover a particular resource type. This should write to STDOUT a JSON string in the format of the ResourceInformation class. This has a name and an array of addresses. For a client-submitted driver, discovery script must assign different resource addresses to this driver comparing to other drivers on the same host. 3.0.0 spark.driver.resource.{resourceName}.vendor None Vendor of the resources to use for the driver. This option is currently only supported on Kubernetes and is actually both the vendor and domain following the Kubernetes device plugin naming convention. (e.g. For GPUs on Kubernetes this config would be set to nvidia.com or amd.com) 3.0.0 spark.resources.discoveryPlugin org.apache.spark.resource.ResourceDiscoveryScriptPlugin Comma-separated list of class names implementing org.apache.spark.api.resource.ResourceDiscoveryPlugin to load into the application. This is for advanced users to replace the resource discovery class with a custom implementation. Spark will try each class specified until one of them returns the resource information for that resource. It tries the discovery script last if none of the plugins return information for that resource. 3.0.0 spark.executor.memory 1g Amount of memory to use per executor process, in the same format as JVM memory strings with a size unit suffix (\"k\", \"m\", \"g\" or \"t\") (e.g. 512m, 2g), and with minimum value 450m. 0.7.0 spark.executor.pyspark.memory Not set The amount of memory to be allocated to PySpark in each executor, in MiB unless otherwise specified. If set, PySpark memory for an executor will be limited to this amount. If not set, Spark will not limit Python's memory use and it is up to the application to avoid exceeding the overhead memory space shared with other non-JVM processes. When PySpark is run in YARN or Kubernetes, this memory is added to executor resource requests. feature is dependent on Python's resource module; therefore, the behaviors and limitations are inherited. For instance, Windows does not support resource limiting and actual resource is not limited on MacOS. 2.4.0 spark.executor.memoryOverhead executorMemory * spark.executor.memoryOverheadFactor, with minimum of spark.executor.minMemoryOverhead Amount of additional memory to be allocated per executor process, in MiB unless otherwise specified. This is memory that accounts for things like VM overheads, interned strings, other native overheads, etc. This tends to grow with the executor size (typically 6-10%). This option is currently supported on YARN and Kubernetes. memory includes PySpark executor memory (when spark.executor.pyspark.memory is not configured) and memory used by other non-executor processes running in the same container. The maximum memory size of container to running executor is determined by the sum of spark.executor.memoryOverhead, spark.executor.memory, spark.memory.offHeap.size and spark.executor.pyspark.memory. 2.3.0 spark.executor.minMemoryOverhead 384m The minimum amount of non-heap memory to be allocated per executor process, in MiB unless otherwise specified, if spark.executor.memoryOverhead is not defined. This option is currently supported on YARN and Kubernetes. 4.0.0 spark.executor.memoryOverheadFactor 0.10 Fraction of executor memory to be allocated as additional non-heap memory per executor process. This is memory that accounts for things like VM overheads, interned strings, other native overheads, etc. This tends to grow with the container size. This value defaults to 0.10 except for Kubernetes non-JVM jobs, which defaults to 0.40. This is done as non-JVM tasks need more non-JVM heap space and such tasks commonly fail with \"Memory Overhead Exceeded\" errors. This preempts this error with a higher default. This value is ignored if spark.executor.memoryOverhead is set directly. 3.3.0 spark.executor.resource.{resourceName}.amount 0 Amount of a particular resource type to use per executor process. If this is used, you must also specify the spark.executor.resource.{resourceName}.discoveryScript for the executor to find the resource on startup. 3.0.0 spark.executor.resource.{resourceName}.discoveryScript None A script for the executor to run to discover a particular resource type. This should write to STDOUT a JSON string in the format of the ResourceInformation class. This has a name and an array of addresses. 3.0.0 spark.executor.resource.{resourceName}.vendor None Vendor of the resources to use for the executors. This option is currently only supported on Kubernetes and is actually both the vendor and domain following the Kubernetes device plugin naming convention. (e.g. For GPUs on Kubernetes this config would be set to nvidia.com or amd.com) 3.0.0 spark.extraListeners (none) A comma-separated list of classes that implement SparkListener; when initializing SparkContext, instances of these classes will be created and registered with Spark's listener bus. If a class has a single-argument constructor that accepts a SparkConf, that constructor will be called; otherwise, a zero-argument constructor will be called. If no valid constructor can be found, the SparkContext creation will fail with an exception. 1.3.0 spark.local.dir /tmp Directory to use for \"scratch\" space in Spark, including map output files and RDDs that get stored on disk. This should be on a fast, local disk in your system. It can also be a comma-separated list of multiple directories on different disks. will be overridden by SPARK_LOCAL_DIRS (Standalone) or LOCAL_DIRS (YARN) environment variables set by the cluster manager. 0.5.0 spark.logConf false Logs the effective SparkConf as INFO when a SparkContext is started. 0.9.0 spark.master (none) The cluster manager to connect to. See the list of allowed master URL's. 0.9.0 spark.submit.deployMode client The deploy mode of Spark driver program, either \"client\" or \"cluster\", Which means to launch driver program locally (\"client\") or remotely (\"cluster\") on one of the nodes inside the cluster. 1.5.0 spark.log.callerContext (none) Application information that will be written into Yarn RM log/HDFS audit log when running on Yarn/HDFS. Its length depends on the Hadoop configuration hadoop.caller.context.max.size. It should be concise, and typically can have up to 50 characters. 2.2.0 spark.log.level (none) When set, overrides any user-defined log settings as if calling SparkContext.setLogLevel() at Spark startup. Valid log levels include: \"ALL\", \"DEBUG\", \"ERROR\", \"FATAL\", \"INFO\", \"OFF\", \"TRACE\", \"WARN\". 3.5.0 spark.driver.supervise false If true, restarts the driver automatically if it fails with a non-zero exit status. Only has effect in Spark standalone mode. 1.3.0 spark.driver.timeout 0min A timeout for Spark driver in minutes. 0 means infinite. For the positive time value, terminate the driver with the exit code 124 if it runs after timeout duration. To use, it's required to set spark.plugins with org.apache.spark.deploy.DriverTimeoutPlugin. 4.0.0 spark.driver.log.localDir (none) Specifies a local directory to write driver logs and enable Driver Log UI Tab. 4.0.0 spark.driver.log.dfsDir (none) Base directory in which Spark driver logs are synced, if spark.driver.log.persistToDfs.enabled is true. Within this base directory, each application logs the driver logs to an application specific file. Users may want to set this to a unified location like an HDFS directory so driver log files can be persisted for later usage. This directory should allow any Spark user to read/write files and the Spark History Server user to delete files. Additionally, older logs from this directory are cleaned by the Spark History Server if spark.history.fs.driverlog.cleaner.enabled is true and, if they are older than max age configured by setting spark.history.fs.driverlog.cleaner.maxAge. 3.0.0 spark.driver.log.persistToDfs.enabled false If true, spark application running in client mode will write driver logs to a persistent storage, configured in spark.driver.log.dfsDir. If spark.driver.log.dfsDir is not configured, driver logs will not be persisted. Additionally, enable the cleaner by setting spark.history.fs.driverlog.cleaner.enabled to true in Spark History Server. 3.0.0 spark.driver.log.layout %d{yy/MM/dd :ss.SSS} %t %p %c{1}: %m%n%ex The layout for the driver logs that are synced to spark.driver.log.localDir and spark.driver.log.dfsDir. If this is not configured, it uses the layout for the first appender defined in log4j2.properties. If that is also not configured, driver logs use the default layout. 3.0.0 spark.driver.log.allowErasureCoding false Whether to allow driver logs to use erasure coding. On HDFS, erasure coded files will not update as quickly as regular replicated files, so they make take longer to reflect changes written by the application. Note that even if this is true, Spark will still not force the file to use erasure coding, it will simply use file system defaults. 3.0.0 spark.driver.log.redirectConsoleOutputs stdout,stderr Comma-separated list of the console output kind for driver that needs to redirect to logging system. Supported values are `stdout`, `stderr`. It only takes affect when `spark.plugins` is configured with `org.apache.spark.deploy.RedirectConsolePlugin`. 4.1.0 spark.decommission.enabled false When decommission enabled, Spark will try its best to shut down the executor gracefully. Spark will try to migrate all the RDD blocks (controlled by spark.storage.decommission.rddBlocks.enabled) and shuffle blocks (controlled by spark.storage.decommission.shuffleBlocks.enabled) from the decommissioning executor to a remote executor when spark.storage.decommission.enabled is enabled. With decommission enabled, Spark will also decommission an executor instead of killing when spark.dynamicAllocation.enabled enabled. 3.1.0 spark.executor.decommission.killInterval (none) Duration after which a decommissioned executor will be killed forcefully by an outside (e.g. non-spark) service. 3.1.0 spark.executor.decommission.forceKillTimeout (none) Duration after which a Spark will force a decommissioning executor to exit. This should be set to a high value in most situations as low values will prevent block migrations from having enough time to complete. 3.2.0 spark.executor.decommission.signal PWR The signal that used to trigger the executor to start decommission. 3.2.0 spark.executor.maxNumFailures numExecutors * 2, with minimum of 3 The maximum number of executor failures before failing the application. This configuration only takes effect on YARN and Kubernetes. 3.5.0 spark.executor.failuresValidityInterval (none) Interval after which executor failures will be considered independent and not accumulate towards the attempt count. This configuration only takes effect on YARN and Kubernetes. 3.5.0 Apart from these, the following properties are also available, and may be useful in some Environment Property NameDefaultMeaningSince Version spark.driver.extraClassPath (none) Extra classpath entries to prepend to the classpath of the driver. client mode, this config must not be set through the SparkConf directly in your application, because the driver JVM has already started at that point. Instead, please set this through the --driver-class-path command line option or in your default properties file. 1.0.0 spark.driver.defaultJavaOptions (none) A string of default JVM options to prepend to spark.driver.extraJavaOptions. This is intended to be set by administrators. For instance, GC settings or other logging. Note that it is illegal to set maximum heap size (-Xmx) settings with this option. Maximum heap size settings can be set with spark.driver.memory in the cluster mode and through the --driver-memory command line option in the client mode. client mode, this config must not be set through the SparkConf directly in your application, because the driver JVM has already started at that point. Instead, please set this through the --driver-java-options command line option or in your default properties file. 3.0.0 spark.driver.extraJavaOptions (none) A string of extra JVM options to pass to the driver. This is intended to be set by users. For instance, GC settings or other logging. Note that it is illegal to set maximum heap size (-Xmx) settings with this option. Maximum heap size settings can be set with spark.driver.memory in the cluster mode and through the --driver-memory command line option in the client mode. client mode, this config must not be set through the SparkConf directly in your application, because the driver JVM has already started at that point. Instead, please set this through the --driver-java-options command line option or in your default properties file. spark.driver.defaultJavaOptions will be prepended to this configuration. 1.0.0 spark.driver.extraLibraryPath (none) Set a special library path to use when launching the driver JVM. client mode, this config must not be set through the SparkConf directly in your application, because the driver JVM has already started at that point. Instead, please set this through the --driver-library-path command line option or in your default properties file. 1.0.0 spark.driver.userClassPathFirst false (Experimental) Whether to give user-added jars precedence over Spark's own jars when loading classes in the driver. This feature can be used to mitigate conflicts between Spark's dependencies and user dependencies. It is currently an experimental feature. This is used in cluster mode only. 1.3.0 spark.executor.extraClassPath (none) Extra classpath entries to prepend to the classpath of executors. This exists primarily for backwards-compatibility with older versions of Spark. Users typically should not need to set this option. 1.0.0 spark.executor.defaultJavaOptions (none) A string of default JVM options to prepend to spark.executor.extraJavaOptions. This is intended to be set by administrators. For instance, GC settings or other logging. Note that it is illegal to set Spark properties or maximum heap size (-Xmx) settings with this option. Spark properties should be set using a SparkConf object or the spark-defaults.conf file used with the spark-submit script. Maximum heap size settings can be set with spark.executor.memory. The following symbols, if present will be be replaced by application ID and will be replaced by executor ID. For example, to enable verbose gc logging to a file named for the executor ID of the app in /tmp, pass a 'value' :gc -Xloggc:/tmp/-.gc 3.0.0 spark.executor.extraJavaOptions (none) A string of extra JVM options to pass to executors. This is intended to be set by users. For instance, GC settings or other logging. Note that it is illegal to set Spark properties or maximum heap size (-Xmx) settings with this option. Spark properties should be set using a SparkConf object or the spark-defaults.conf file used with the spark-submit script. Maximum heap size settings can be set with spark.executor.memory. The following symbols, if present will be be replaced by application ID and will be replaced by executor ID. For example, to enable verbose gc logging to a file named for the executor ID of the app in /tmp, pass a 'value' :gc -Xloggc:/tmp/-.gc spark.executor.defaultJavaOptions will be prepended to this configuration. 1.0.0 spark.executor.extraLibraryPath (none) Set a special library path to use when launching executor JVM's. 1.0.0 spark.executor.logs.rolling.maxRetainedFiles -1 Sets the number of latest rolling log files that are going to be retained by the system. Older log files will be deleted. Disabled by default. 1.1.0 spark.executor.logs.rolling.enableCompression false Enable executor log compression. If it is enabled, the rolled executor logs will be compressed. Disabled by default. 2.0.2 spark.executor.logs.rolling.maxSize 1024 * 1024 Set the max size of the file in bytes by which the executor logs will be rolled over. Rolling is disabled by default. See spark.executor.logs.rolling.maxRetainedFiles for automatic cleaning of old logs. 1.4.0 spark.executor.logs.rolling.strategy \"\" (disabled) Set the strategy of rolling of executor logs. By default it is disabled. It can be set to \"time\" (time-based rolling) or \"size\" (size-based rolling) or \"\" (disabled). For \"time\", use spark.executor.logs.rolling.time.interval to set the rolling interval. For \"size\", use spark.executor.logs.rolling.maxSize to set the maximum file size for rolling. 1.1.0 spark.executor.logs.rolling.time.interval daily Set the time interval by which the executor logs will be rolled over. Rolling is disabled by default. Valid values are daily, hourly, minutely or any interval in seconds. See spark.executor.logs.rolling.maxRetainedFiles for automatic cleaning of old logs. 1.1.0 spark.executor.logs.redirectConsoleOutputs stdout,stderr Comma-separated list of the console output kind for executor that needs to redirect to logging system. Supported values are `stdout`, `stderr`. It only takes affect when `spark.plugins` is configured with `org.apache.spark.deploy.RedirectConsolePlugin`. 4.1.0 spark.executor.userClassPathFirst false (Experimental) Same functionality as spark.driver.userClassPathFirst, but applied to executor instances. 1.3.0 spark.executorEnv.[EnvironmentVariableName] (none) Add the environment variable specified by EnvironmentVariableName to the Executor process. The user can specify multiple of these to set multiple environment variables. 0.9.0 spark.redaction.regex (?i)secret|password|token|access[.]?key Regex to decide which Spark configuration properties and environment variables in driver and executor environments contain sensitive information. When this regex matches a property key or value, the value is redacted from the environment UI and various logs like YARN and event logs. 2.1.2 spark.redaction.string.regex (none) Regex to decide which parts of strings produced by Spark contain sensitive information. When this regex matches a string part, that string part is replaced by a dummy value. This is currently used to redact the output of SQL explain commands. 2.2.0 spark.python.profile false Enable profiling in Python worker, the profile result will show up by sc.show_profiles(), or it will be displayed before the driver exits. It also can be dumped into disk by sc.dump_profiles(path). If some of the profile results had been displayed manually, they will not be displayed automatically before driver exiting. By default the pyspark.profiler.BasicProfiler will be used, but this can be overridden by passing a profiler class in as a parameter to the SparkContext constructor. 1.2.0 spark.python.profile.dump (none) The directory which is used to dump the profile result before driver exiting. The results will be dumped as separated file for each RDD. They can be loaded by pstats.Stats(). If this is specified, the profile result will not be displayed automatically. 1.2.0 spark.python.worker.memory 512m Amount of memory to use per python worker process during aggregation, in the same format as JVM memory strings with a size unit suffix (\"k\", \"m\", \"g\" or \"t\") (e.g. 512m, 2g). If the memory used during aggregation goes above this amount, it will spill the data into disks. 1.1.0 spark.python.worker.reuse true Reuse Python worker or not. If yes, it will use a fixed number of Python workers, does not need to fork() a Python process for every task. It will be very useful if there is a large broadcast, then the broadcast will not need to be transferred from JVM to Python worker for every task. 1.2.0 spark.python.factory.idleWorkerMaxPoolSize (none) Maximum number of idle Python workers to keep. If unset, the number is unbounded. If set to a positive integer N, at most N idle workers are retained; least-recently used workers are evicted first. 4.1.0 spark.python.worker.killOnIdleTimeout false Whether Spark should terminate the Python worker process when the idle timeout (as defined by spark.python.worker.idleTimeoutSeconds) is reached. If enabled, Spark will terminate the Python worker process in addition to logging the status. 4.1.0 spark.python.worker.tracebackDumpIntervalSeconds 0 The interval (in seconds) for Python workers to dump their tracebacks. If it's positive, the Python worker will periodically dump the traceback into its `stderr`. The default is `0` that means it is disabled. 4.1.0 spark.python.unix.domain.socket.enabled false When set to true, the Python driver uses a Unix domain socket for operations like creating or collecting a DataFrame from local data, using accumulators, and executing Python functions with PySpark such as Python UDFs. This configuration only applies to Spark Classic and Spark Connect server. 4.1.0 spark.files Comma-separated list of files to be placed in the working directory of each executor. Globs are allowed. 1.0.0 spark.submit.pyFiles Comma-separated list of .amount 1 Amount of a particular resource type to allocate for each task, note that this can be a double. If this is specified you must also provide the executor config spark.executor.resource.{resourceName}.amount and any corresponding discovery configs so that your executors are created with that resource type. In addition to whole amounts, a fractional amount (for example, 0.25, which means 1/4th of a resource) may be specified. Fractional amounts must be less than or equal to 0.5, or in other words, the minimum amount of resource sharing is 2 tasks per resource. Additionally, fractional amounts are floored in order to assign resource slots (e.g. a 0.2222 configuration, or 1/0.2222 slots will become 4 tasks/resource, not 5). 3.0.0 spark.task.maxFailures 4 Number of continuous failures of any particular task before giving up on the job. The total number of failures spread across different tasks will not cause the job to fail; a particular task has to fail this number of attempts continuously. If any attempt succeeds, the failure count for the task will be reset. Should be greater than or equal to 1. Number of allowed retries = this value - 1. 0.8.0 spark.task.reaper.enabled false Enables monitoring of killed / interrupted tasks. When set to true, any task which is killed will be monitored by the executor until that task actually finishes executing. See the other spark.task.reaper.* configurations for details on how to control the exact behavior of this monitoring. When set to false (the default), task killing will use an older code path which lacks such monitoring. 2.0.3 spark.task.reaper.pollingInterval 10s When spark.task.reaper.enabled = true, this setting controls the frequency at which executors will poll the status of killed tasks. If a killed task is still running when polled then a warning will be logged and, by default, a thread-dump of the task will be logged (this thread dump can be disabled via the spark.task.reaper.threadDump setting, which is documented below). 2.0.3 spark.task.reaper.threadDump true When spark.task.reaper.enabled = true, this setting controls whether task thread dumps are logged during periodic polling of killed tasks. Set this to false to disable collection of thread dumps. 2.0.3 spark.task.reaper.killTimeout -1 When spark.task.reaper.enabled = true, this setting specifies a timeout after which the executor JVM will kill itself if a killed task has not stopped running. The default value, -1, disables this mechanism and prevents the executor from self-destructing. The purpose of this setting is to act as a safety-net to prevent runaway noncancellable tasks from rendering an executor unusable. 2.0.3 spark.stage.maxConsecutiveAttempts 4 Number of consecutive stage attempts allowed before a stage is aborted. 2.2.0 spark.stage.ignoreDecommissionFetchFailure true Whether ignore stage fetch failure caused by executor decommission when count spark.stage.maxConsecutiveAttempts 3.4.0 Barrier Execution Mode Property NameDefaultMeaningSince Version spark.barrier.sync.timeout 365d The timeout in seconds for each barrier() call from a barrier task. If the coordinator didn't receive all the sync messages from barrier tasks within the configured time, throw a SparkException to fail all the tasks. The default value is set to 31536000(3600 * 24 * 365) so the barrier() call shall wait for one year. 2.4.0 spark.scheduler.barrier.maxConcurrentTasksCheck.interval 15s Time in seconds to wait between a max concurrent tasks check failure and the next check. A max concurrent tasks check ensures the cluster can launch more concurrent tasks than required by a barrier stage on job submitted. The check can fail in case a cluster has just started and not enough executors have registered, so we wait for a little while and try to perform the check again. If the check fails more than a configured max failure times for a job then fail current job submission. Note this config only applies to jobs that contain one or more barrier stages, we won't perform the check on non-barrier jobs. 2.4.0 spark.scheduler.barrier.maxConcurrentTasksCheck.maxFailures 40 Number of max concurrent tasks check failures allowed before fail a job submission. A max concurrent tasks check ensures the cluster can launch more concurrent tasks than required by a barrier stage on job submitted. The check can fail in case a cluster has just started and not enough executors have registered, so we wait for a little while and try to perform the check again. If the check fails more than a configured max failure times for a job then fail current job submission. Note this config only applies to jobs that contain one or more barrier stages, we won't perform the check on non-barrier jobs. 2.4.0 Dynamic Allocation Property NameDefaultMeaningSince Version spark.dynamicAllocation.enabled false Whether to use dynamic resource allocation, which scales the number of executors registered with this application up and down based on the workload. For more detail, see the description here. This requires one of the following ) enabling external shuffle service through spark.shuffle.service.enabled, or 2) enabling shuffle tracking through spark.dynamicAllocation.shuffleTracking.enabled, or 3) enabling shuffle blocks decommission through spark.decommission.enabled and spark.storage.decommission.shuffleBlocks.enabled, or 4) (Experimental) configuring spark.shuffle.sort.io.plugin.class to use a custom ShuffleDataIO who's ShuffleDriverComponents supports reliable storage. The following configurations are also , spark.dynamicAllocation.maxExecutors, and spark.dynamicAllocation.initialExecutors spark.dynamicAllocation.executorAllocationRatio 1.2.0 spark.dynamicAllocation.executorIdleTimeout 60s If dynamic allocation is enabled and an executor has been idle for more than this duration, the executor will be removed. For more detail, see this description. 1.2.0 spark.dynamicAllocation.cachedExecutorIdleTimeout infinity If dynamic allocation is enabled and an executor which has cached data blocks has been idle for more than this duration, the executor will be removed. For more details, see this description. 1.4.0 spark.dynamicAllocation.initialExecutors spark.dynamicAllocation.minExecutors Initial number of executors to run if dynamic allocation is enabled. If --num-executors (or spark.executor.instances) is set and larger than this value, it will be used as the initial number of executors. 1.3.0 spark.dynamicAllocation.maxExecutors infinity Upper bound for the number of executors if dynamic allocation is enabled. 1.2.0 spark.dynamicAllocation.minExecutors 0 Lower bound for the number of executors if dynamic allocation is enabled. 1.2.0 spark.dynamicAllocation.executorAllocationRatio 1 By default, the dynamic allocation will request enough executors to maximize the parallelism according to the number of tasks to process. While this minimizes the latency of the job, with small tasks this setting can waste a lot of resources due to executor allocation overhead, as some executor might not even do any work. This setting allows to set a ratio that will be used to reduce the number of executors w.r.t. full parallelism. Defaults to 1.0 to give maximum parallelism. 0.5 will divide the target number of executors by 2 The target number of executors computed by the dynamicAllocation can still be overridden by the spark.dynamicAllocation.minExecutors and spark.dynamicAllocation.maxExecutors settings 2.4.0 spark.dynamicAllocation.schedulerBacklogTimeout 1s If dynamic allocation is enabled and there have been pending tasks backlogged for more than this duration, new executors will be requested. For more detail, see this description. 1.2.0 spark.dynamicAllocation.sustainedSchedulerBacklogTimeout schedulerBacklogTimeout Same as spark.dynamicAllocation.schedulerBacklogTimeout, but used only for subsequent executor requests. For more detail, see this description. 1.2.0 spark.dynamicAllocation.shuffleTracking.enabled true Enables shuffle file tracking for executors, which allows dynamic allocation without the need for an external shuffle service. This option will try to keep alive executors that are storing shuffle data for active jobs. 3.0.0 spark.dynamicAllocation.shuffleTracking.timeout infinity When shuffle tracking is enabled, controls the timeout for executors that are holding shuffle data. The default value means that Spark will rely on the shuffles being garbage collected to be able to release executors. If for some reason garbage collection is not cleaning up shuffles quickly enough, this option can be used to control when to time out executors even when they are storing shuffle data. 3.0.0 Thread Configurations Depending on jobs and cluster configurations, we can set number of threads in several places in Spark to utilize available resources efficiently to get better performance. Prior to Spark 3.0, these thread configurations apply to all roles of Spark, such as driver, executor, worker and master. From Spark 3.0, we can configure threads in finer granularity starting from driver and executor. Take RPC module as example in below table. For other modules, like shuffle, just replace “rpc” with “shuffle” in the property names except spark.{driver|executor}.rpc.netty.dispatcher.numThreads, which is only for RPC module. Property NameDefaultMeaningSince Version spark.{driver|executor}.rpc.io.serverThreads Fall back on spark.rpc.io.serverThreads Number of threads used in the server thread pool 1.6.0 spark.{driver|executor}.rpc.io.clientThreads Fall back on spark.rpc.io.clientThreads Number of threads used in the client thread pool 1.6.0 spark.{driver|executor}.rpc.netty.dispatcher.numThreads Fall back on spark.rpc.netty.dispatcher.numThreads Number of threads used in RPC message dispatcher thread pool 3.0.0 The default value for number of thread-related config keys is the minimum of the number of cores requested for the driver or executor, or, in the absence of that value, the number of cores available for the JVM (with a hardcoded upper limit of 8). Spark Connect Server Configuration Server configurations are set in Spark Connect server, for example, when you start the Spark Connect server with ./sbin/start-connect-server.sh. They are typically set via the config file and command-line options with --conf/-c. Property NameDefaultMeaningSince Version spark.api.mode classic For Spark Classic applications, specify whether to automatically use Spark Connect by running a local Spark Connect server. The value can be classic or connect. 4.0.0 spark.connect.grpc.binding.address (none) Address for Spark Connect server to bind. 4.0.0 spark.connect.grpc.binding.port 15002 Port for Spark Connect server to bind. 3.4.0 spark.connect.grpc.port.maxRetries 0 The max port retry attempts for the gRPC server binding. By default, it's set to 0, and the server will fail fast in case of port conflicts. 4.0.0 spark.connect.grpc.interceptor.classes (none) Comma separated list of class names that must implement the io.grpc.ServerInterceptor interface 3.4.0 spark.connect.grpc.arrow.maxBatchSize 4m When using Apache Arrow, limit the maximum size of one arrow batch that can be sent from server side to client side. Currently, we conservatively use 70% of it because the size is not accurate but estimated. 3.4.0 spark.connect.grpc.maxInboundMessageSize 134217728 Sets the maximum inbound message size for the gRPC requests. Requests with a larger payload will fail. 3.4.0 spark.connect.extensions.relation.classes (none) Comma separated list of classes that implement the trait org.apache.spark.sql.connect.plugin.RelationPlugin to support custom Relation types in proto. 3.4.0 spark.connect.extensions.expression.classes (none) Comma separated list of classes that implement the trait org.apache.spark.sql.connect.plugin.ExpressionPlugin to support custom Expression types in proto. 3.4.0 spark.connect.extensions.command.classes (none) Comma separated list of classes that implement the trait org.apache.spark.sql.connect.plugin.CommandPlugin to support custom Command types in proto. 3.4.0 spark.connect.ml.backend.classes (none) Comma separated list of classes that implement the trait org.apache.spark.sql.connect.plugin.MLBackendPlugin to replace the specified Spark ML operators with a backend-specific implementation. 4.0.0 spark.connect.jvmStacktrace.maxSize 1024 Sets the maximum stack trace size to display when `spark.sql.pyspark.jvmStacktrace.enabled` is true. 3.5.0 spark.sql.connect.ui.retainedSessions 200 The number of client sessions kept in the Spark Connect UI history. 3.5.0 spark.sql.connect.ui.retainedStatements 200 The number of statements kept in the Spark Connect UI history. 3.5.0 spark.sql.connect.enrichError.enabled true When true, it enriches errors with full exception messages and optionally server-side stacktrace on the client side via an additional RPC. 4.0.0 spark.sql.connect.serverStacktrace.enabled true When true, it sets the server-side stacktrace in the user-facing Spark exception. 4.0.0 spark.connect.grpc.maxMetadataSize 1024 Sets the maximum size of metadata fields. For instance, it restricts metadata fields in `ErrorInfo`. 4.0.0 spark.connect.progress.reportInterval 2s The interval at which the progress of a query is reported to the client. If the value is set to a negative value the progress reports will be disabled. 4.0.0 Security Please refer to the Security page for available options on how to secure different Spark subsystems. Spark SQL Runtime SQL Configuration Runtime SQL configurations are per-session, mutable Spark SQL configurations. They can be set with initial values by the config file and command-line options with --conf/-c prefixed, or by setting SparkConf that are used to create SparkSession. Also, they can be set and queried by SET commands and reset to their initial values by RESET command, or by SparkSession.conf’s setter and getter methods in runtime. Property NameDefaultMeaningSince Version spark.sql.adaptive.advisoryPartitionSizeInBytes (value of spark.sql.adaptive.shuffle.targetPostShuffleInputSize) The advisory size in bytes of the shuffle partition during adaptive optimization (when spark.sql.adaptive.enabled is true). It takes effect when Spark coalesces small shuffle partitions or splits skewed shuffle partition. 3.0.0 spark.sql.adaptive.autoBroadcastJoinThreshold (none) Configures the maximum size in bytes for a table that will be broadcast to all worker nodes when performing a join. By setting this value to -1 broadcasting can be disabled. The default value is same with spark.sql.autoBroadcastJoinThreshold. Note that, this config is used only in adaptive framework. 3.2.0 spark.sql.adaptive.coalescePartitions.enabled true When true and 'spark.sql.adaptive.enabled' is true, Spark will coalesce contiguous shuffle partitions according to the target size (specified by 'spark.sql.adaptive.advisoryPartitionSizeInBytes'), to avoid too many small tasks. 3.0.0 spark.sql.adaptive.coalescePartitions.initialPartitionNum (none) The initial number of shuffle partitions before coalescing. If not set, it equals to spark.sql.shuffle.partitions. This configuration only has an effect when 'spark.sql.adaptive.enabled' and 'spark.sql.adaptive.coalescePartitions.enabled' are both true. 3.0.0 spark.sql.adaptive.coalescePartitions.minPartitionSize 1MB The minimum size of shuffle partitions after coalescing. This is useful when the adaptively calculated target size is too small during partition coalescing. 3.2.0 spark.sql.adaptive.coalescePartitions.parallelismFirst true When true, Spark does not respect the target size specified by 'spark.sql.adaptive.advisoryPartitionSizeInBytes' (default 64MB) when coalescing contiguous shuffle partitions, but adaptively calculate the target size according to the default parallelism of the Spark cluster. The calculated size is usually smaller than the configured target size. This is to maximize the parallelism and avoid performance regressions when enabling adaptive query execution. It's recommended to set this config to false on a busy cluster to make resource utilization more efficient (not many small tasks). 3.2.0 spark.sql.adaptive.customCostEvaluatorClass (none) The custom cost evaluator class to be used for adaptive execution. If not being set, Spark will use its own SimpleCostEvaluator by default. 3.2.0 spark.sql.adaptive.enabled true When true, enable adaptive query execution, which re-optimizes the query plan in the middle of query execution, based on accurate runtime statistics. 1.6.0 spark.sql.adaptive.forceOptimizeSkewedJoin false When true, force enable OptimizeSkewedJoin even if it introduces extra shuffle. 3.3.0 spark.sql.adaptive.localShuffleReader.enabled true When true and 'spark.sql.adaptive.enabled' is true, Spark tries to use local shuffle reader to read the shuffle data when the shuffle partitioning is not needed, for example, after converting sort-merge join to broadcast-hash join. 3.0.0 spark.sql.adaptive.maxShuffledHashJoinLocalMapThreshold 0b Configures the maximum size in bytes per partition that can be allowed to build local hash map. If this value is not smaller than spark.sql.adaptive.advisoryPartitionSizeInBytes and all the partition size are not larger than this config, join selection prefer to use shuffled hash join instead of sort merge join regardless of the value of spark.sql.join.preferSortMergeJoin. 3.2.0 spark.sql.adaptive.optimizeSkewsInRebalancePartitions.enabled true When true and 'spark.sql.adaptive.enabled' is true, Spark will optimize the skewed shuffle partitions in RebalancePartitions and split them to smaller ones according to the target size (specified by 'spark.sql.adaptive.advisoryPartitionSizeInBytes'), to avoid data skew. 3.2.0 spark.sql.adaptive.optimizer.excludedRules (none) Configures a list of rules to be disabled in the adaptive optimizer, in which the rules are specified by their rule names and separated by comma. The optimizer will log the rules that have indeed been excluded. 3.1.0 spark.sql.adaptive.rebalancePartitionsSmallPartitionFactor 0.2 A partition will be merged during splitting if its size is small than this factor multiply spark.sql.adaptive.advisoryPartitionSizeInBytes. 3.3.0 spark.sql.adaptive.skewJoin.enabled true When true and 'spark.sql.adaptive.enabled' is true, Spark dynamically handles skew in shuffled join (sort-merge and shuffled hash) by splitting (and replicating if needed) skewed partitions. 3.0.0 spark.sql.adaptive.skewJoin.skewedPartitionFactor 5.0 A partition is considered as skewed if its size is larger than this factor multiplying the median partition size and also larger than 'spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes' 3.0.0 spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes 256MB A partition is considered as skewed if its size in bytes is larger than this threshold and also larger than 'spark.sql.adaptive.skewJoin.skewedPartitionFactor' multiplying the median partition size. Ideally this config should be set larger than 'spark.sql.adaptive.advisoryPartitionSizeInBytes'. 3.0.0 spark.sql.allowNamedFunctionArguments true If true, Spark will turn on support for named parameters for all functions that has it implemented. 3.5.0 spark.sql.ansi.doubleQuotedIdentifiers false When true and 'spark.sql.ansi.enabled' is true, Spark SQL reads literals enclosed in double quoted (\") as identifiers. When false they are read as string literals. 3.4.0 spark.sql.ansi.enabled true When true, Spark SQL uses an ANSI compliant dialect instead of being Hive compliant. For example, Spark will throw an exception at runtime instead of returning null results when the inputs to a SQL operator/function are invalid. For full details of this dialect, you can find them in the section \"ANSI Compliance\" of Spark's documentation. Some ANSI dialect features may be not from the ANSI SQL standard directly, but their behaviors align with ANSI SQL's style 3.0.0 spark.sql.ansi.enforceReservedKeywords false When true and 'spark.sql.ansi.enabled' is true, the Spark SQL parser enforces the ANSI reserved keywords and forbids SQL queries that use reserved keywords as alias names and/or identifiers for table, view, function, etc. 3.3.0 spark.sql.ansi.relationPrecedence false When true and 'spark.sql.ansi.enabled' is true, JOIN takes precedence over comma when combining relation. For example, t1, t2 JOIN t3 should result to t1 X (t2 X t3). If the config is false, the result is (t1 X t2) X t3. 3.4.0 spark.sql.autoBroadcastJoinThreshold 10MB Configures the maximum size in bytes for a table that will be broadcast to all worker nodes when performing a join. By setting this value to -1 broadcasting can be disabled. 1.1.0 spark.sql.avro.compression.codec snappy Compression codec used in writing of AVRO files. Supported , deflate, snappy, bzip2, xz and zstandard. Default codec is snappy. 2.4.0 spark.sql.avro.deflate.level -1 Compression level for the deflate codec used in writing of AVRO files. Valid value must be in the range of from 1 to 9 inclusive or -1. The default value is -1 which corresponds to 6 level in the current implementation. 2.4.0 spark.sql.avro.filterPushdown.enabled true When true, enable filter pushdown to Avro datasource. 3.1.0 spark.sql.avro.xz.level 6 Compression level for the xz codec used in writing of AVRO files. Valid value must be in the range of from 1 to 9 inclusive The default value is 6. 4.0.0 spark.sql.avro.zstandard.bufferPool.enabled false If true, enable buffer pool of ZSTD JNI library when writing of AVRO files 4.0.0 spark.sql.avro.zstandard.level 3 Compression level for the zstandard codec used in writing of AVRO files. 4.0.0 spark.sql.binaryOutputStyle (none) The output style used display binary data. Valid values are 'UTF-8', 'BASIC', 'BASE64', 'HEX', and 'HEX_DISCRETE'. 4.0.0 spark.sql.broadcastTimeout 300 Timeout in seconds for the broadcast wait time in broadcast joins. 1.3.0 spark.sql.bucketing.coalesceBucketsInJoin.enabled false When true, if two bucketed tables with the different number of buckets are joined, the side with a bigger number of buckets will be coalesced to have the same number of buckets as the other side. Bigger number of buckets is divisible by the smaller number of buckets. Bucket coalescing is applied to sort-merge joins and shuffled hash join. bucketed table can avoid unnecessary shuffling in join, but it also reduces parallelism and could possibly cause OOM for shuffled hash join. 3.1.0 spark.sql.bucketing.coalesceBucketsInJoin.maxBucketRatio 4 The ratio of the number of two buckets being coalesced should be less than or equal to this value for bucket coalescing to be applied. This configuration only has an effect when 'spark.sql.bucketing.coalesceBucketsInJoin.enabled' is set to true. 3.1.0 spark.sql.catalog.spark_catalog builtin A catalog implementation that will be used as the v2 interface to Spark's built-in v1 This catalog shares its identifier namespace with the spark_catalog and must be consistent with it; for example, if a table can be loaded by the spark_catalog, this catalog must also return the table metadata. To delegate operations to the spark_catalog, implementations can extend 'CatalogExtension'. The value should be either 'builtin' which represents the spark's builit-in V2SessionCatalog, or a fully qualified class name of the catalog implementation. 3.0.0 spark.sql.cbo.enabled false Enables CBO for estimation of plan statistics when set true. 2.2.0 spark.sql.cbo.joinReorder.dp.star.filter false Applies star-join filter heuristics to cost based join enumeration. 2.2.0 spark.sql.cbo.joinReorder.dp.threshold 12 The maximum number of joined nodes allowed in the dynamic programming algorithm. 2.2.0 spark.sql.cbo.joinReorder.enabled false Enables join reorder in CBO. 2.2.0 spark.sql.cbo.planStats.enabled false When true, the logical plan will fetch row counts and column statistics from catalog. 3.0.0 spark.sql.cbo.starSchemaDetection false When true, it enables join reordering based on star schema detection. 2.2.0 spark.sql.charAsVarchar false When true, Spark replaces CHAR type with VARCHAR type in CREATE/REPLACE/ALTER TABLE commands, so that newly created/updated tables will not have CHAR type columns/fields. Existing tables with CHAR type columns/fields are not affected by this config. 3.3.0 spark.sql.chunkBase64String.enabled true Whether to truncate string generated by the Base64 function. When true, base64 strings generated by the base64 function are chunked into lines of at most 76 characters. When false, the base64 strings are not chunked. 3.5.2 spark.sql.classic.shuffleDependency.fileCleanup.enabled false When enabled, shuffle files will be cleaned up at the end of classic SQL executions. Note that this cleanup may cause stage retries and regenerate shuffle files if the same dataframe reference is executed again. 4.1.0 spark.sql.cli.print.header false When set to true, spark-sql CLI prints the names of the columns in query output. 3.2.0 spark.sql.collation.allowInMapKeys false Allow for non-UTF8_BINARY collated strings inside of map's keys 4.0.0 spark.sql.columnNameOfCorruptRecord _corrupt_record The name of internal column for storing raw/un-parsed JSON and CSV records that fail to parse. 1.2.0 spark.sql.connect.shuffleDependency.fileCleanup.enabled (value of spark.sql.shuffleDependency.fileCleanup.enabled) When enabled, shuffle files will be cleaned up at the end of Spark Connect SQL executions. 4.1.0 spark.sql.csv.filterPushdown.enabled true When true, enable filter pushdown to CSV datasource. 3.0.0 spark.sql.datetime.java8API.enabled false If the configuration property is set to true, java.time.Instant and java.time.LocalDate classes of Java 8 API are used as external types for Catalyst's TimestampType and DateType. If it is set to false, java.sql.Timestamp and java.sql.Date are used for the same purpose. 3.0.0 spark.sql.debug.maxToStringFields 25 Maximum number of fields of sequence-like entries can be converted to strings in debug output. Any elements beyond the limit will be dropped and replaced by a \"... N more fields\" placeholder. 3.0.0 spark.sql.defaultCacheStorageLevel MEMORY_AND_DISK The default storage level of dataset.cache(), catalog.cacheTable() and sql query CACHE TABLE t. 4.0.0 spark.sql.defaultCatalog spark_catalog Name of the default catalog. This will be the current catalog if users have not explicitly set the current catalog yet. 3.0.0 spark.sql.defaultPath Default SQL PATH used when no SET PATH has been issued in the session; this is also the value to which SET PATH = DEFAULT_PATH expands. Accepts the full SET PATH grammar; an inner DEFAULT_PATH token resolves to the spark-builtin default ordering. The PATH keyword is not allowed in this conf value. When empty, the spark-builtin default ordering controlled by spark.sql.functionResolution.sessionOrder applies. Validated for syntax at set time; redundant entries are tolerated (lookup uses first-match resolution). The interactive SET PATH form still rejects static duplicates as a typo guard. 4.2.0 spark.sql.dropTableOnView.enabled true When true, DROP TABLE command will work on VIEW as well. 4.2.0 spark.sql.error.messageFormat PRETTY When PRETTY, the error message consists of textual representation of error class, message and query context. Stack traces are only shown for internal errors (SQLSTATE XX***). When DEBUG, the output is the same as PRETTY but stack traces are always included. The MINIMAL and STANDARD formats are pretty JSON formats where STANDARD includes an additional JSON field message. This configuration property influences on error messages of Thrift Server and SQL CLI while running queries. 3.4.0 spark.sql.execution.arrow.compression.codec none Compression codec used to compress Arrow IPC data when transferring data between JVM and Python processes (e.g., toPandas, toArrow). This can significantly reduce memory usage and network bandwidth when transferring large datasets. Supported codecs: 'none' (no compression), 'zstd' (Zstandard), 'lz4' (LZ4). Note that compression may add CPU overhead but can provide substantial memory savings especially for datasets with high compression ratios. 4.1.0 spark.sql.execution.arrow.compression.zstd.level 3 Compression level for Zstandard (zstd) codec when compressing Arrow IPC data. This config is only used when spark.sql.execution.arrow.compression.codec is set to 'zstd'. Negative values provide ultra-fast compression with lower compression ratios. Positive values provide normal to maximum compression, with higher values giving better compression but slower speed. The default value 3 provides a good balance between compression speed and compression ratio. 4.1.0 spark.sql.execution.arrow.enabled true (Deprecated since Spark 3.0, please set 'spark.sql.execution.arrow.pyspark.enabled'.) 2.3.0 spark.sql.execution.arrow.fallback.enabled true (Deprecated since Spark 3.0, please set 'spark.sql.execution.arrow.pyspark.fallback.enabled'.) 2.4.0 spark.sql.execution.arrow.localRelationThreshold 48MB When converting Arrow batches to Spark DataFrame, local collections are used in the driver side if the byte size of Arrow batches is smaller than this threshold. Otherwise, the Arrow batches are sent and deserialized to Spark internal rows in the executors. 3.4.0 spark.sql.execution.arrow.maxRecordsPerBatch 10000 When using Apache Arrow, limit the maximum number of records that can be written to a single ArrowRecordBatch in memory. If set to zero or negative there is no limit. See also spark.sql.execution.arrow.maxBytesPerBatch. If both are set, each batch is created when any condition of both is met. 2.3.0 spark.sql.execution.arrow.pyspark.enabled (value of spark.sql.execution.arrow.enabled) When true, make use of Apache Arrow for columnar data transfers in PySpark. This optimization applies pyspark.sql.DataFrame.toPandas. 2. pyspark.sql.SparkSession.createDataFrame when its input is a Pandas DataFrame or a NumPy ndarray. The following data type is of TimestampType. 3.0.0 spark.sql.execution.arrow.pyspark.fallback.enabled (value of spark.sql.execution.arrow.fallback.enabled) When true, optimizations enabled by 'spark.sql.execution.arrow.pyspark.enabled' will fallback automatically to non-optimized implementations if an error occurs. 3.0.0 spark.sql.execution.arrow.pyspark.selfDestruct.enabled false (Experimental) When true, make use of Apache Arrow's self-destruct and split-blocks options for columnar data transfers in PySpark, when converting from Arrow to Pandas. This reduces memory usage at the cost of some CPU time. This optimization applies when 'spark.sql.execution.arrow.pyspark.enabled' is set. 3.2.0 spark.sql.execution.arrow.pyspark.validateSchema.enabled false When true, validate the schema of Arrow batches returned by mapInArrow, mapInPandas and DataSource against the expected schema to ensure that they are compatible. 4.1.0 spark.sql.execution.arrow.pythonUDF.columnarInput.enabled true When true, Arrow-based Python UDFs (pandas UDFs) can accept columnar input directly from upstream operators that produce Arrow-backed ColumnarBatch (e.g., DataSource V2 connectors), bypassing the ColumnarToRow and ArrowWriter conversion. This optimization reduces data transfer overhead between the JVM and Python worker processes. 4.2.0 spark.sql.execution.arrow.sparkr.enabled false When true, make use of Apache Arrow for columnar data transfers in SparkR. This optimization applies createDataFrame when its input is an R DataFrame 2. collect 3. dapply 4. gapply The following data types are , BinaryType, ArrayType, StructType and MapType. 3.0.0 spark.sql.execution.arrow.transformWithStateInPySpark.maxStateRecordsPerBatch 10000 When using TransformWithState in PySpark (both Python Row and Pandas), limit the maximum number of state records that can be written to a single ArrowRecordBatch in memory. 4.0.0 spark.sql.execution.arrow.useLargeVarTypes false When using Apache Arrow, use large variable width vectors for string and binary types. Regular string and binary types have a 2GiB limit for a column in a single record batch. Large variable types remove this limitation at the cost of higher memory usage per value. 3.5.0 spark.sql.execution.interruptOnCancel true When true, all running tasks will be interrupted if one cancels a query. 4.0.0 spark.sql.execution.pandas.inferPandasDictAsMap false When true, spark.createDataFrame will infer dict from Pandas DataFrame as a MapType. When false, spark.createDataFrame infers dict from Pandas DataFrame as a StructType which is default inferring from PyArrow. 4.0.0 spark.sql.execution.pandas.structHandlingMode legacy The conversion mode of struct type when creating pandas DataFrame. When \"legacy\", 1. when Arrow optimization is disabled, convert to Row object, 2. when Arrow optimization is enabled, convert to dict or raise an Exception if there are duplicated nested field names. When \"row\", convert to Row object regardless of Arrow optimization. When \"dict\", convert to dict and use suffixed key names, e.g., a_0, a_1, if there are duplicated nested field names, regardless of Arrow optimization. 3.5.0 spark.sql.execution.pandas.udf.buffer.size (value of spark.buffer.size) Same as spark.buffer.size but only applies to Pandas UDF executions. If it is not set, the fallback is spark.buffer.size. Note that Pandas execution requires more than 4 bytes. Lowering this value could make small Pandas UDF batch iterated and pipelined; however, it might degrade performance. See SPARK-27870. 3.0.0 spark.sql.execution.pyspark.binaryAsBytes true When true, BinaryType is consistently mapped to bytes in PySpark. When false, restores the PySpark behavior before 4.1.0. Before 4.1.0, BinaryType is mapped to bytearray for regular UDF and UDTF without Arrow optimization, DataFrame APIs (both Spark Classic and Spark Connect), and data sources; BinaryType is mapped to bytes for Arrow-optimized UDF and UDTF with legacy pandas conversion. 4.1.0 spark.sql.execution.pyspark.udf.daemonKillWorkerOnFlushFailure (value of spark.python.daemon.killWorkerOnFlushFailure) Same as spark.python.daemon.killWorkerOnFlushFailure for Python execution with DataFrame and SQL. It can change during runtime. 4.1.0 spark.sql.execution.pyspark.udf.faulthandler.enabled (value of spark.python.worker.faulthandler.enabled) Same as spark.python.worker.faulthandler.enabled for Python execution with DataFrame and SQL. It can change during runtime. 4.0.0 spark.sql.execution.pyspark.udf.hideTraceback.enabled false When true, only show the message of the exception from Python UDFs, hiding the stack trace. If this is enabled, simplifiedTraceback has no effect. 4.0.0 spark.sql.execution.pyspark.udf.idleTimeoutSeconds (value of spark.python.worker.idleTimeoutSeconds) Same as spark.python.worker.idleTimeoutSeconds for Python execution with DataFrame and SQL. It can change during runtime. 4.0.0 spark.sql.execution.pyspark.udf.killOnIdleTimeout (value of spark.python.worker.killOnIdleTimeout) Same as spark.python.worker.killOnIdleTimeout for Python execution with DataFrame and SQL. It can change during runtime. 4.1.0 spark.sql.execution.pyspark.udf.simplifiedTraceback.enabled true When true, the traceback from Python UDFs is simplified. It hides the Python worker, (de)serialization, etc from PySpark in tracebacks, and only shows the exception messages from UDFs. Note that this works only with CPython 3.7+. 3.1.0 spark.sql.execution.pyspark.udf.tracebackDumpIntervalSeconds (value of spark.python.worker.tracebackDumpIntervalSeconds) Same as spark.python.worker.tracebackDumpIntervalSeconds for Python execution with DataFrame and SQL. It can change during runtime. 4.1.0 spark.sql.execution.python.udf.buffer.size (value of spark.buffer.size) Same as spark.buffer.size but only applies to Python UDF executions. If it is not set, the fallback is spark.buffer.size. 4.0.0 spark.sql.execution.python.udf.maxRecordsPerBatch 100 When using Python UDFs, limit the maximum number of records that can be batched for serialization/deserialization. 4.0.0 spark.sql.execution.pythonUDF.arrow.concurrency.level (none) The level of concurrency to execute Arrow-optimized Python UDF. This can be useful if Python UDFs use I/O intensively. 4.0.0 spark.sql.execution.pythonUDF.arrow.enabled true Enable Arrow optimization in regular Python UDFs. This optimization can only be enabled when the given function takes at least one argument. 3.4.0 spark.sql.execution.pythonUDF.pandas.intToDecimalCoercionEnabled false When true, convert int to Decimal python objects before converting Pandas.Series to Arrow array during serialization.Disabled by default, impacts performance. 4.1.0 spark.sql.execution.pythonUDF.pandas.preferIntExtensionDtype false When true, convert integers to Pandas ExtensionDtype (e.g. pandas.Int64Dtype) for Pandas UDF execution. Otherwise, depends on the behavior of pyarrow.Array.to_pandas on each input arrow batch. 4.2.0 spark.sql.execution.pythonUDTF.arrow.enabled true Enable Arrow optimization for Python UDTFs. 3.5.0 spark.sql.execution.topKSortFallbackThreshold 2147483632 In SQL queries with a SORT followed by a LIMIT like 'SELECT x FROM t ORDER BY y LIMIT m', if m is under this threshold, do a top-K sort in memory, otherwise do a global sort which spills to disk if necessary. 2.4.0 spark.sql.extendedExplainProviders (none) A comma-separated list of classes that implement the org.apache.spark.sql.ExtendedExplainGenerator trait. If provided, Spark will print extended plan information from the providers in explain plan and in the UI 4.0.0 spark.sql.fileSource.insert.enforceNotNull false When true, Spark enforces NOT NULL constraints when inserting data into file-based data source tables (e.g., Parquet, ORC, JSON), consistent with the behavior for other data sources and V2 catalog tables. When false (default), null values are silently accepted into NOT NULL columns. 4.2.0 spark.sql.files.ignoreCorruptFiles false Whether to ignore corrupt files. If true, the Spark jobs will continue to run when encountering corrupted files and the contents that have been read will still be returned. This configuration is effective only when using file-based sources such as Parquet, JSON and ORC. 2.1.1 spark.sql.files.ignoreInvalidPartitionPaths false Whether to ignore invalid partition paths that do not match <column>=<value>. When the option is enabled, table with two partition directories 'table/invalid' and 'table/col=1' will only load the latter directory and ignore the invalid partition 4.0.0 spark.sql.files.ignoreMissingFiles false Whether to ignore missing files. If true, the Spark jobs will continue to run when encountering missing files and the contents that have been read will still be returned. This configuration is effective only when using file-based sources such as Parquet, JSON and ORC. 2.3.0 spark.sql.files.maxPartitionBytes 128MB The maximum number of bytes to pack into a single partition when reading files. This configuration is effective only when using file-based sources such as Parquet, JSON and ORC. 2.0.0 spark.sql.files.maxPartitionNum (none) The suggested (not guaranteed) maximum number of split file partitions. If it is set, Spark will rescale each partition to make the number of partitions is close to this value if the initial number of partitions exceeds this value. This configuration is effective only when using file-based sources such as Parquet, JSON and ORC. 3.5.0 spark.sql.files.maxRecordsPerFile 0 Maximum number of records to write out to a single file. If this value is zero or negative, there is no limit. 2.2.0 spark.sql.files.minPartitionNum (none) The suggested (not guaranteed) minimum number of split file partitions. If not set, the default value is spark.sql.leafNodeDefaultParallelism. This configuration is effective only when using file-based sources such as Parquet, JSON and ORC. 3.1.0 spark.sql.function.concatBinaryAsString false When this option is set to false and all inputs are binary, functions.concat returns an output as binary. Otherwise, it returns as a string. 2.3.0 spark.sql.function.eltOutputAsString false When this option is set to false and all inputs are binary, elt returns an output as binary. Otherwise, it returns as a string. 2.3.0 spark.sql.function.protobufExtensions.enabled false When true, the from_protobuf and to_protobuf operators will support proto2 extensions when a binary file descriptor set is provided. This property will have no effect for the overloads taking a Java class name instead of a file descriptor set. 4.2.0 spark.sql.groupByAliases true When true, aliases in a select list can be used in group by clauses. When false, an analysis exception is thrown in the case. 2.2.0 spark.sql.groupByOrdinal true When true, the ordinal numbers in group by clauses are treated as the position in the select list. When false, the ordinal numbers are ignored. 2.0.0 spark.sql.hive.convertInsertingPartitionedTable true When set to true, and spark.sql.hive.convertMetastoreParquet or spark.sql.hive.convertMetastoreOrc is true, the built-in ORC/Parquet writer is usedto process inserting into partitioned ORC/Parquet tables created by using the HiveSQL syntax. 3.0.0 spark.sql.hive.convertInsertingUnpartitionedTable true When set to true, and spark.sql.hive.convertMetastoreParquet or spark.sql.hive.convertMetastoreOrc is true, the built-in ORC/Parquet writer is usedto process inserting into unpartitioned ORC/Parquet tables created by using the HiveSQL syntax. 4.0.0 spark.sql.hive.convertMetastoreAsNullable false When set to true, apply nullable to the schema when Spark use datasource APIs instead of Hive serde to read/write Hive tables in Parquet or ORC formats. This flag is effective only if convertMetastoreParquet or convertMetastoreOrc is enabled respectively. It's recommended to set to true, when the nullability of table schema is inconsistent between the metastore and the data files. 4.1.0 spark.sql.hive.convertMetastoreCtas true When set to true, Spark will try to use built-in data source writer instead of Hive serde in CTAS. This flag is effective only if spark.sql.hive.convertMetastoreParquet or spark.sql.hive.convertMetastoreOrc is enabled respectively for Parquet and ORC formats 3.0.0 spark.sql.hive.convertMetastoreInsertDir true When set to true, Spark will try to use built-in data source writer instead of Hive serde in INSERT OVERWRITE DIRECTORY. This flag is effective only if spark.sql.hive.convertMetastoreParquet or spark.sql.hive.convertMetastoreOrc is enabled respectively for Parquet and ORC formats 3.3.0 spark.sql.hive.convertMetastoreOrc true When set to true, the built-in ORC reader and writer are used to process ORC tables created by using the HiveQL syntax, instead of Hive serde. 2.0.0 spark.sql.hive.convertMetastoreParquet true When set to true, the built-in Parquet reader and writer are used to process parquet tables created by using the HiveQL syntax, instead of Hive serde. 1.1.1 spark.sql.hive.convertMetastoreParquet.mergeSchema false When true, also tries to merge possibly different but compatible Parquet schemas in different Parquet data files. This configuration is only effective when \"spark.sql.hive.convertMetastoreParquet\" is true. 1.3.1 spark.sql.hive.dropPartitionByName.enabled false When true, Spark will get partition name rather than partition object to drop partition, which can improve the performance of drop partition. 3.4.0 spark.sql.hive.filesourcePartitionFileCacheSize 262144000 When nonzero, enable caching of partition file metadata in memory. All tables share a cache that can use up to specified num bytes for file metadata. This conf only has an effect when hive filesource partition management is enabled. 2.1.1 spark.sql.hive.manageFilesourcePartitions true When true, enable metastore partition management for file source tables as well. This includes both datasource and converted Hive tables. When partition management is enabled, datasource tables store partition in the Hive metastore, and use the metastore to prune partitions during query planning when spark.sql.hive.metastorePartitionPruning is set to true. 2.1.1 spark.sql.hive.metastorePartitionPruning true When true, some predicates will be pushed down into the Hive metastore so that unmatching partitions can be eliminated earlier. 1.5.0 spark.sql.hive.metastorePartitionPruningFallbackOnException false Whether to fallback to get all partitions from Hive metastore and perform partition pruning on Spark client side, when encountering MetaException from the metastore. Note that Spark query performance may degrade if this is enabled and there are many partitions to be listed. If this is disabled, Spark will fail the query instead. 3.3.0 spark.sql.hive.metastorePartitionPruningFastFallback false When this config is enabled, if the predicates are not supported by Hive or Spark does fallback due to encountering MetaException from the metastore, Spark will instead prune partitions by getting the partition names first and then evaluating the filter expressions on the client side. Note that the predicates with TimeZoneAwareExpression is not supported. 3.3.0 spark.sql.hive.thriftServer.async true When set to true, Hive Thrift server executes SQL queries in an asynchronous way. 1.5.0 spark.sql.icu.caseMappings.enabled true When enabled we use the ICU library (instead of the JVM) to implement case mappings for strings under UTF8_BINARY collation. 4.0.0 spark.sql.inMemoryColumnarStorage.batchSize 10000 Controls the size of batches for columnar caching. Larger batch sizes can improve memory utilization and compression, but risk OOMs when caching data. 1.1.1 spark.sql.inMemoryColumnarStorage.compressed true When set to true Spark SQL will automatically select a compression codec for each column based on statistics of the data. 1.0.1 spark.sql.inMemoryColumnarStorage.enableVectorizedReader true Enables vectorized reader for columnar caching. 2.3.1 spark.sql.inMemoryColumnarStorage.hugeVectorReserveRatio 1.2 When spark.sql.inMemoryColumnarStorage.hugeVectorThreshold <= 0 or the required memory is smaller than spark.sql.inMemoryColumnarStorage.hugeVectorThreshold, spark reserves required memory * 2 memory; otherwise, spark reserves required memory * this ratio memory, and will release this column vector memory before reading the next batch rows. 4.0.0 spark.sql.inMemoryColumnarStorage.hugeVectorThreshold -1b When the required memory is larger than this, spark reserves required memory * spark.sql.inMemoryColumnarStorage.hugeVectorReserveRatio memory next time and release this column vector memory before reading the next batch rows. -1 means disabling the optimization. 4.0.0 spark.sql.insertIntoReplaceOn.enabled true Enable the SQL syntax INSERT INTO ... REPLACE ON (...). The command atomically inserts new rows into a table after deleting all existing rows that match the new rows according to the specified matching condition. The inserted rows are specified by a VALUES expression or the result of a query. 4.2.0 spark.sql.insertIntoReplaceUsing.enabled true Enable the SQL syntax INSERT INTO ... REPLACE USING (...). The command atomically inserts new rows into a table after deleting all existing rows that match the new rows according to the key columns specified in the statement. The inserted rows are specified by a VALUES expression or the result of a query. 4.2.0 spark.sql.json.filterPushdown.enabled true When true, enable filter pushdown to JSON datasource. 3.1.0 spark.sql.json.useUnsafeRow false When set to true, use UnsafeRow to represent struct result in the JSON parser. It can be overwritten by the JSON option useUnsafeRow. 4.0.0 spark.sql.jsonGenerator.ignoreNullFields true Whether to ignore null fields when generating JSON objects in JSON data source and JSON functions such as to_json. If false, it generates null for null fields in JSON objects. 3.0.0 spark.sql.leafNodeDefaultParallelism (none) The default parallelism of Spark SQL leaf nodes that produce data, such as the file scan node, the local data scan node, the range node, etc. The default value of this config is 'SparkContext#defaultParallelism'. 3.2.0 spark.sql.legacy.hive.thriftServer.useZeroBasedColumnOrdinalPosition false When set to true, Hive Thrift server returns 0-based ORDINAL_POSITION in the result of GetColumns operation, instead of the corrected 1-based. 4.1.0 spark.sql.mapKeyDedupPolicy EXCEPTION The policy to deduplicate map keys in builtin , MapFromArrays, MapFromEntries, StringToMap, MapConcat and TransformKeys. When EXCEPTION, the query fails if duplicated map keys are detected. When LAST_WIN, the map key that is inserted at last takes precedence. 3.0.0 spark.sql.maven.additionalRemoteRepositories https://maven-central.storage-download.googleapis.com/maven2/ A comma-delimited string config of the optional additional remote Maven mirror repositories. This is only used for downloading Hive jars in IsolatedClientLoader if the default Maven Central repo is unreachable. 3.0.0 spark.sql.maxBroadcastTableSize 8589934592b The maximum table size in bytes that can be broadcast in broadcast joins. 4.1.0 spark.sql.maxMetadataStringLength 100 Maximum number of characters to output for a metadata string. e.g. file location in DataSourceScanExec, every value will be abbreviated if exceed length. 3.1.0 spark.sql.maxPlanStringLength 2147483632 Maximum number of characters to output for a plan string. If the plan is longer, further output will be truncated. The default setting always generates a full plan. Set this to a lower value such as 8k if plan strings are taking up too much memory or are causing OutOfMemory errors in the driver or UI processes. 3.0.0 spark.sql.maxSinglePartitionBytes 128m The maximum number of bytes allowed for a single partition. Otherwise, The planner will introduce shuffle to improve parallelism. 3.4.0 spark.sql.operatorPipeSyntaxEnabled true If true, enable operator pipe syntax for Apache Spark SQL. This uses the operator pipe marker |> to indicate separation between clauses of SQL in a manner that describes the sequence of steps that the query performs in a composable fashion. 4.0.0 spark.sql.optimizer.avoidCollapseUDFWithExpensiveExpr true Whether to avoid collapsing projections that would duplicate expensive expressions in UDFs. 4.0.0 spark.sql.optimizer.avoidDoubleFilterEval true When true avoid pushing expensive (UDF, etc.) filters down if it could result indouble evaluation. This was the behaviour prior to 3.X. 4.2.0 spark.sql.optimizer.collapseProjectAlwaysInline false Whether to always collapse two adjacent projections and inline expressions even if it causes extra duplication. 3.3.0 spark.sql.optimizer.dynamicPartitionPruning.enabled true When true, we will generate predicate for partition column when it's used as join key 3.0.0 spark.sql.optimizer.enableCsvExpressionOptimization true Whether to optimize CSV expressions in SQL optimizer. It includes pruning unnecessary columns from from_csv. 3.2.0 spark.sql.optimizer.enableJsonExpressionOptimization true Whether to optimize JSON expressions in SQL optimizer. It includes pruning unnecessary columns from from_json, simplifying from_json + to_json, to_json + named_struct(from_json.col1, from_json.col2, ....). 3.1.0 spark.sql.optimizer.excludedRules (none) Configures a list of rules to be disabled in the optimizer, in which the rules are specified by their rule names and separated by comma. It is not guaranteed that all the rules in this configuration will eventually be excluded, as some rules are necessary for correctness. The optimizer will log the rules that have indeed been excluded. 2.4.0 spark.sql.optimizer.mergeSubplans.filterPropagation.enabled true When set to true, subquery plans that differ only in their filter conditions can be merged by propagating filters up to enclosing non-grouping aggregates. 4.2.0 spark.sql.optimizer.mergeSubplans.filterPropagation.symmetricFilterPropagation.enabled false When set to true, two non-grouping aggregate subplans that both have filter conditions (but with different predicates) can be merged into a single scan using FILTER (WHERE ...) clauses on each aggregate expression. Merging two filtered scans broadens the combined filter to OR(f1, f2), which may reduce IO pruning (e.g. partition or file skipping) compared to the individual filters. Disabled by default; enable once the behaviour has been validated in your workload, particularly on heavily partitioned or file-pruned tables. Has no effect when spark.sql.optimizer.mergeSubplans.filterPropagation.enabled is false. 4.2.0 spark.sql.optimizer.mergeSubplans.filterPropagation.throughJoin.enabled false When set to true, filter attributes can propagate through Join nodes during subplan merging, allowing subplans that differ only in their filter conditions and share a common join to be merged into a single scan. A filter attribute is only propagated through a join when it originates from the non-nullable (preserved) left side of LeftOuter/LeftSemi/LeftAnti, the right side of RightOuter, or either side of Inner/Cross. FullOuter joins are never eligible. Has no effect when spark.sql.optimizer.mergeSubplans.filterPropagation.enabled is false. 4.2.0 spark.sql.optimizer.pushDownJoinThroughUnion.enabled false When true, pushes down Join through Union when the right side is small enough to broadcast. This can improve performance by allowing each Union branch to directly perform a broadcast join, avoiding materializing the entire Union result. 4.2.0 spark.sql.optimizer.runtime.bloomFilter.applicationSideScanSizeThreshold 10GB Byte size threshold of the Bloom filter application side plan's aggregated scan size. Aggregated scan byte size of the Bloom filter application side needs to be over this value to inject a bloom filter. 3.3.0 spark.sql.optimizer.runtime.bloomFilter.creationSideThreshold 10MB Size threshold of the bloom filter creation side plan. Estimated size needs to be under this value to try to inject bloom filter. 3.3.0 spark.sql.optimizer.runtime.bloomFilter.enabled true When true and if one side of a shuffle join has a selective predicate, we attempt to insert a bloom filter in the other side to reduce the amount of shuffle data. 3.3.0 spark.sql.optimizer.runtime.bloomFilter.expectedNumItems 1000000 The default number of expected items for the runtime bloomfilter 3.3.0 spark.sql.optimizer.runtime.bloomFilter.maxNumBits 67108864 The max number of bits to use for the runtime bloom filter 3.3.0 spark.sql.optimizer.runtime.bloomFilter.maxNumItems 4000000 The max allowed number of expected items for the runtime bloom filter 3.3.0 spark.sql.optimizer.runtime.bloomFilter.numBits 8388608 The default number of bits to use for the runtime bloom filter 3.3.0 spark.sql.optimizer.runtime.rowLevelOperationGroupFilter.enabled true Enables runtime filtering for group-based and delta-based row-level operations. Data sources may prune entire file groups at runtime when planning a row-level operation scan. Planning-time filter pushdown is limited as not all expressions can be converted into data source filters and some expressions can only be evaluated by Spark (e.g. subqueries). Since rewriting groups or scanning unnecessary files is expensive, Spark can execute a lightweight query at runtime to find what records match the condition of the row-level operation. The information about matching records will be passed back to the row-level operation scan, allowing data sources to skip files that don't have to be processed. 3.4.0 spark.sql.optimizer.runtimeFilter.number.threshold 10 The total number of injected runtime filters (non-DPP) for a single query. This is to prevent driver OOMs with too many Bloom filters. 3.3.0 spark.sql.orc.aggregatePushdown false If true, aggregates will be pushed down to ORC for optimization. Support MIN, MAX and COUNT as aggregate expression. For MIN/MAX, support boolean, integer, float and date type. For COUNT, support all data types. If statistics is missing from any ORC file footer, exception would be thrown. 3.3.0 spark.sql.orc.columnarReaderBatchSize 4096 The number of rows to include in a orc vectorized reader batch. The number should be carefully chosen to minimize overhead and avoid OOMs in reading data. 2.4.0 spark.sql.orc.columnarWriterBatchSize 1024 The number of rows to include in a orc vectorized writer batch. The number should be carefully chosen to minimize overhead and avoid OOMs in writing data. 3.4.0 spark.sql.orc.compression.codec zstd Sets the compression codec used when writing ORC files. If either compression or orc.compress is specified in the table-specific options/properties, the precedence would be compression, orc.compress, spark.sql.orc.compression.codec. Acceptable values , uncompressed, snappy, zlib, lzo, zstd, lz4, brotli. 2.3.0 spark.sql.orc.enableNestedColumnVectorizedReader true Enables vectorized orc decoding for nested column. 3.2.0 spark.sql.orc.enableVectorizedReader true Enables vectorized orc decoding. 2.3.0 spark.sql.orc.filterPushdown true When true, enable filter pushdown for ORC files. 1.4.0 spark.sql.orc.mergeSchema false When true, the Orc data source merges schemas collected from all data files, otherwise the schema is picked from a random data file. 3.0.0 spark.sql.orderByOrdinal true When true, the ordinal numbers are treated as the position in the select list. When false, the ordinal numbers in order/sort by clause are ignored. 2.0.0 spark.sql.parquet.aggregatePushdown false If true, aggregates will be pushed down to Parquet for optimization. Support MIN, MAX and COUNT as aggregate expression. For MIN/MAX, support boolean, integer, float and date type. For COUNT, support all data types. If statistics is missing from any Parquet file footer, exception would be thrown. 3.3.0 spark.sql.parquet.binaryAsString false Some other Parquet-producing systems, in particular Impala and older versions of Spark SQL, do not differentiate between binary data and strings when writing out the Parquet schema. This flag tells Spark SQL to interpret binary data as a string to provide compatibility with these systems. 1.1.1 spark.sql.parquet.columnarReaderBatchSize 4096 The number of rows to include in a parquet vectorized reader batch. The number should be carefully chosen to minimize overhead and avoid OOMs in reading data. 2.4.0 spark.sql.parquet.compression.codec snappy Sets the compression codec used when writing Parquet files. If either compression or parquet.compression is specified in the table-specific options/properties, the precedence would be compression, parquet.compression, spark.sql.parquet.compression.codec. Acceptable values , uncompressed, snappy, gzip, lzo, brotli, lz4, lz4_raw, zstd. 1.1.1 spark.sql.parquet.enableNestedColumnVectorizedReader true Enables vectorized Parquet decoding for nested columns (e.g., struct, list, map). Requires spark.sql.parquet.enableVectorizedReader to be enabled. 3.3.0 spark.sql.parquet.enableVectorizedReader true Enables vectorized parquet decoding. 2.0.0 spark.sql.parquet.fieldId.read.enabled false Field ID is a native field of the Parquet schema spec. When enabled, Parquet readers will use field IDs (if present) in the requested Spark schema to look up Parquet fields instead of using column names 3.3.0 spark.sql.parquet.fieldId.read.ignoreMissing false When the Parquet file doesn't have any field IDs but the Spark read schema is using field IDs to read, we will silently return nulls when this flag is enabled, or error otherwise. 3.3.0 spark.sql.parquet.fieldId.write.enabled true Field ID is a native field of the Parquet schema spec. When enabled, Parquet writers will populate the field Id metadata (if present) in the Spark schema to the Parquet schema. 3.3.0 spark.sql.parquet.filterPushdown true Enables Parquet filter push-down optimization when set to true. 1.2.0 spark.sql.parquet.inferTimestampNTZ.enabled true When enabled, Parquet timestamp columns with annotation isAdjustedToUTC = false are inferred as TIMESTAMP_NTZ type during schema inference. Otherwise, all the Parquet timestamp columns are inferred as TIMESTAMP_LTZ types. Note that Spark writes the output schema into Parquet's footer metadata on file writing and leverages it on file reading. Thus this configuration only affects the schema inference on Parquet files which are not written by Spark. 3.4.0 spark.sql.parquet.int96AsTimestamp true Some Parquet-producing systems, in particular Impala, store Timestamp into INT96. Spark would also store Timestamp as INT96 because we need to avoid precision lost of the nanoseconds field. This flag tells Spark SQL to interpret INT96 data as a timestamp to provide compatibility with these systems. 1.3.0 spark.sql.parquet.int96TimestampConversion false This controls whether timestamp adjustments should be applied to INT96 data when converting to timestamps, for data written by Impala. This is necessary because Impala stores INT96 data with a different timezone offset than Hive & Spark. 2.3.0 spark.sql.parquet.mergeSchema false When true, the Parquet data source merges schemas collected from all data files, otherwise the schema is picked from the summary file or a random data file if no summary file is available. 1.5.0 spark.sql.parquet.outputTimestampType INT96 Sets which Parquet timestamp type to use when Spark writes data to Parquet files. INT96 is a non-standard but commonly used timestamp type in Parquet. TIMESTAMP_MICROS is a standard timestamp type in Parquet, which stores number of microseconds from the Unix epoch. TIMESTAMP_MILLIS is also standard, but with millisecond precision, which means Spark has to truncate the microsecond portion of its timestamp value. 2.3.0 spark.sql.parquet.recordLevelFilter.enabled false If true, enables Parquet's native record-level filtering using the pushed down filters. This configuration only has an effect when 'spark.sql.parquet.filterPushdown' is enabled and the vectorized reader is not used. You can ensure the vectorized reader is not used by setting 'spark.sql.parquet.enableVectorizedReader' to false. 2.3.0 spark.sql.parquet.respectSummaryFiles false When true, we make assumption that all part-files of Parquet are consistent with summary files and we will ignore them when merging schema. Otherwise, if this is false, which is the default, we will merge all part-files. This should be considered as expert-only option, and shouldn't be enabled before knowing what it means exactly. 1.5.0 spark.sql.parquet.writeLegacyFormat false If true, data will be written in a way of Spark 1.4 and earlier. For example, decimal values will be written in Apache Parquet's fixed-length byte array format, which other systems such as Apache Hive and Apache Impala use. If false, the newer format in Parquet will be used. For example, decimals will be written in int-based format. If Parquet output is intended for use with systems that do not support this newer format, set to true. 1.6.0 spark.sql.parser.quotedRegexColumnNames false When true, quoted Identifiers (using backticks) in SELECT statement are interpreted as regular expressions. 2.3.0 spark.sql.parser.singleCharacterPipeOperator.enabled true When true, the single character pipe token '|' can be used as an alternative to '|>' for SQL pipe operators. When false, only '|>' is recognized as a pipe operator, and '|' is only used for bitwise OR operations. This provides syntax compatibility with other languages like Splunk SPL and Kusto that use '|' for pipe operations. 4.2.0 spark.sql.path.enabled false When true, enables the SQL Standard PATH PATH, path-based routine resolution, and CURRENT_PATH(). When false, SET PATH is rejected and resolution uses the default path only. 4.2.0 spark.sql.pipelines.maxFlowRetryAttempts 2 Maximum number of times a flow can be retried 4.1.0 spark.sql.pivotMaxValues 10000 When doing a pivot without specifying values for the pivot column this is the maximum number of (distinct) values that will be collected without error. 1.6.0 spark.sql.planner.pythonExecution.memory (none) Specifies the memory allocation for executing Python code in Spark driver, in MiB. When set, it caps the memory for Python execution to the specified amount. If not set, Spark will not limit Python's memory usage and it is up to the application to avoid exceeding the overhead memory space shared with other non-JVM processes. does not support resource limiting and actual resource is not limited on MacOS. 4.0.0 spark.sql.preserveCharVarcharTypeInfo false When true, Spark does not replace CHAR/VARCHAR types the STRING type, which is the default behavior of Spark 3.0 and earlier versions. This means the length checks for CHAR/VARCHAR types is enforced and CHAR type is also properly padded. 4.0.0 spark.sql.pyspark.dataSource.profiler (none) Configure the Python Data Source profiler by enabling or disabling it with the option to choose between \"perf\" and \"memory\" types, or unsetting the config disables the profiler. This is disabled by default. 4.2.0 spark.sql.pyspark.inferNestedDictAsStruct.enabled false PySpark's SparkSession.createDataFrame infers the nested dict as a map by default. When it set to true, it infers the nested dict as a struct. 3.3.0 spark.sql.pyspark.jvmStacktrace.enabled false When true, it shows the JVM stacktrace in the user-facing PySpark exception together with Python stacktrace. By default, it is disabled to hide JVM stacktrace and shows a Python-friendly exception only. Note that this is independent from log level settings. 3.0.0 spark.sql.pyspark.plotting.max_rows 1000 The visual limit on plots. If set to 1000 for top-n-based plots (pie, bar, barh), the first 1000 data points will be used for plotting. For sampled-based plots (scatter, area, line), 1000 data points will be randomly sampled. 4.0.0 spark.sql.pyspark.toJSON.returnDataFrame false When true, DataFrame.toJSON in PySpark Classic returns a Dataframe instead of RDD. 4.2.0 spark.sql.pyspark.udf.profiler (none) Configure the Python/Pandas UDF profiler by enabling or disabling it with the option to choose between \"perf\" and \"memory\" types, or unsetting the config disables the profiler. This is disabled by default. 4.0.0 spark.sql.pyspark.worker.logging.enabled false When set to true, this configuration enables comprehensive logging within Python worker processes that execute User-Defined Functions (UDFs), User-Defined Table Functions (UDTFs), and other Python-based operations in Spark SQL. 4.1.0 spark.sql.readSideCharPadding true When true, Spark applies string padding when reading CHAR type columns/fields, in addition to the write-side padding. This config is true by default to better enforce CHAR type semantic in cases such as external tables. 3.4.0 spark.sql.redaction.options.regex (?i)url Regex to decide which keys in a Spark SQL command's options map contain sensitive information. The values of options whose names that match this regex will be redacted in the explain output. This redaction is applied on top of the global redaction configuration defined by spark.redaction.regex. 2.2.2 spark.sql.redaction.string.regex (value of spark.redaction.string.regex) Regex to decide which parts of strings produced by Spark contain sensitive information. When this regex matches a string part, that string part is replaced by a dummy value. This is currently used to redact the output of SQL explain commands. When this conf is not set, the value from spark.redaction.string.regex is used. 2.3.0 spark.sql.repl.eagerEval.enabled false Enables eager evaluation or not. When true, the top K rows of Dataset will be displayed if and only if the REPL supports the eager evaluation. Currently, the eager evaluation is supported in PySpark and SparkR. In PySpark, for the notebooks like Jupyter, the HTML table (generated by repr_html) will be returned. For plain Python REPL, the returned outputs are formatted like dataframe.show(). In SparkR, the returned outputs are showed similar to R data.frame would. 2.4.0 spark.sql.repl.eagerEval.maxNumRows 20 The max number of rows that are returned by eager evaluation. This only takes effect when spark.sql.repl.eagerEval.enabled is set to true. The valid range of this config is from 0 to (Int.MaxValue - 1), so the invalid config like negative and greater than (Int.MaxValue - 1) will be normalized to 0 and (Int.MaxValue - 1). 2.4.0 spark.sql.repl.eagerEval.truncate 20 The max number of characters for each cell that is returned by eager evaluation. This only takes effect when spark.sql.repl.eagerEval.enabled is set to true. 2.4.0 spark.sql.scripting.enabled true SQL Scripting feature is under development and its use should be done under this feature flag. SQL Scripting enables users to write procedural SQL including control flow and error handling. 4.0.0 spark.sql.session.localRelationCacheThreshold 1048576 The threshold for the size in bytes of local relations to be cached at the driver side after serialization. 3.5.0 spark.sql.session.localRelationChunkSizeBytes 16777216 The chunk size in bytes when splitting ChunkedCachedLocalRelation.data into batches. A new chunk is created when either spark.sql.session.localRelationChunkSizeBytes or spark.sql.session.localRelationChunkSizeRows is reached. Limited by the spark.sql.session.localRelationBatchOfChunksSizeBytes, a minimum of the two confs is used to determine the chunk size. 4.1.0 spark.sql.session.localRelationChunkSizeRows 10000 The chunk size in number of rows when splitting ChunkedCachedLocalRelation.data into batches. A new chunk is created when either spark.sql.session.localRelationChunkSizeBytes or spark.sql.session.localRelationChunkSizeRows is reached. 4.1.0 spark.sql.session.timeZone (value of local timezone) The ID of session local timezone in the format of either region-based zone IDs or zone offsets. Region IDs must have the form 'area/city', such as 'America/Los_Angeles'. Zone offsets must be in the format '(+|-)HH', '(+|-)HH:mm' or '(+|-)HH:mm:ss', e.g '-08', '+01:00' or '-13:33:33'. Also 'UTC' and 'Z' are supported as aliases of '+00:00'. Other short names are not recommended to use because they can be ambiguous. 2.2.0 spark.sql.shuffle.orderIndependentChecksum.enableFullRetryOnMismatch true Whether to retry all tasks of a consumer stage when we detect checksum mismatches with its producer stages. 4.1.0 spark.sql.shuffle.orderIndependentChecksum.enabled true Whether to calculate order independent checksum for the shuffle data or not. If enabled, Spark will calculate a checksum that is independent of the input row order for each mapper and returns the checksums from executors to driver. This is different from the checksum computed when spark.shuffle.checksum.enabled is enabled which is sensitive to shuffle data ordering to detect file corruption. While this checksum will be the same even if the shuffle row order changes and it is used to detect whether different task attempts of the same partition produce different output data or not (same set of keyValue pairs). In case the output data has changed across retries, Spark will need to retry all tasks of the consumer stages to avoid correctness issues. 4.1.0 spark.sql.shuffle.partitions 200 The default number of partitions to use when shuffling data for joins or aggregations. 1.1.0 spark.sql.shuffleDependency.fileCleanup.enabled false (Deprecated since Spark 4.1, please set 'spark.sql.connect.shuffleDependency.fileCleanup.enabled'.) 4.0.0 spark.sql.shuffleDependency.skipMigration.enabled false When enabled, shuffle dependencies for a Spark Connect SQL execution are marked at the end of the execution, and they will not be migrated during decommissions. 4.0.0 spark.sql.shuffledHashJoinFactor 3 The shuffle hash join can be selected if the data size of small side multiplied by this factor is still smaller than the large side. 3.3.0 spark.sql.sources.bucketing.autoBucketedScan.enabled true When true, decide whether to do bucketed scan on input tables based on query plan automatically. Do not use bucketed scan if 1. query does not have operators to utilize bucketing (e.g. join, group-by, etc), or 2. there's an exchange operator between these operators and table scan. Note when 'spark.sql.sources.bucketing.enabled' is set to false, this configuration does not take any effect. 3.1.0 spark.sql.sources.bucketing.enabled true When false, we will treat bucketed table as normal table 2.0.0 spark.sql.sources.bucketing.maxBuckets 100000 The maximum number of buckets allowed. 2.4.0 spark.sql.sources.default parquet The default data source to use in input/output. 1.3.0 spark.sql.sources.parallelPartitionDiscovery.threshold 32 The maximum number of paths allowed for listing files at driver side. If the number of detected paths exceeds this value during partition discovery, it tries to list the files with another Spark distributed job. This configuration is effective only when using file-based sources such as Parquet, JSON and ORC. 1.5.0 spark.sql.sources.partitionColumnTypeInference.enabled true When true, automatically infer the data types for partitioned columns. 1.5.0 spark.sql.sources.partitionOverwriteMode STATIC When INSERT OVERWRITE a partitioned data source table, we currently support 2 and dynamic. In static mode, Spark deletes all the partitions that match the partition specification(e.g. PARTITION(a=1,b)) in the INSERT statement, before overwriting. In dynamic mode, Spark doesn't delete partitions ahead, and only overwrite those partitions that have data written into it at runtime. By default we use static mode to keep the same behavior of Spark prior to 2.3. Note that this config doesn't affect Hive serde tables, as they are always overwritten with dynamic mode. This can also be set as an output option for a data source using key partitionOverwriteMode (which takes precedence over this setting), e.g. dataframe.write.option(\"partitionOverwriteMode\", \"dynamic\").save(path). 2.3.0 spark.sql.sources.v2.bucketing.allowCompatibleTransforms.enabled false Whether to allow storage-partition join in the case where the partition transforms are compatible but not identical. This config requires both spark.sql.sources.v2.bucketing.enabled and spark.sql.sources.v2.bucketing.pushPartValues.enabled to be enabled and spark.sql.sources.v2.bucketing.partiallyClusteredDistribution.enabled to be disabled. 4.0.0 spark.sql.sources.v2.bucketing.allowJoinKeysSubsetOfPartitionKeys.enabled false Whether to allow storage-partition join in the case where join keys are a subset of the partition keys of the source tables. At planning time, Spark will group the partitions by only those keys that are in the join keys. This is currently enabled only if spark.sql.requireAllClusterKeysForDistribution is false. 4.0.0 spark.sql.sources.v2.bucketing.enabled true Similar to spark.sql.sources.bucketing.enabled, this config is used to enable bucketing for V2 data sources. When turned on, Spark will recognize the specific distribution reported by a V2 data source through SupportsReportPartitioning, and will try to avoid shuffle if necessary. 3.3.0 spark.sql.sources.v2.bucketing.partiallyClusteredDistribution.enabled false During a storage-partitioned join, whether to allow input partitions to be partially clustered, when both sides of the join are of KeyedPartitioning. At planning time, Spark will pick the side with less data size based on table statistics, group and replicate them to match the other side. This is an optimization on skew join and can help to reduce data skewness when certain partitions are assigned large amount of data. This config requires both spark.sql.sources.v2.bucketing.enabled and spark.sql.sources.v2.bucketing.pushPartValues.enabled to be enabled 3.4.0 spark.sql.sources.v2.bucketing.partition.filter.enabled false Whether to filter partitions when running storage-partition join. When enabled, partitions without matches on the other side can be omitted for scanning, if allowed by the join type. This config requires both spark.sql.sources.v2.bucketing.enabled and spark.sql.sources.v2.bucketing.pushPartValues.enabled to be enabled. 4.0.0 spark.sql.sources.v2.bucketing.partitionKeyOrdering.enabled false When enabled, Spark derives output ordering from the partition key expressions of a V2 data source that reports a KeyedPartitioning but does not report explicit ordering via SupportsReportOrdering. Within a single partition all rows share the same key value, so the data is trivially sorted by those expressions. Requires spark.sql.sources.v2.bucketing.enabled to be enabled. 4.2.0 spark.sql.sources.v2.bucketing.preserveKeyOrderingOnCoalesce.enabled false When enabled, Spark preserves sort orders over partition key expressions when GroupPartitionsExec coalesces multiple input partitions into one output partition. Because all merged partitions share the same partition key value, sort orders over those key expressions remain valid after the merge. This applies to both key-derived ordering (from SupportsReportOrdering) and ordering derived from spark.sql.sources.v2.bucketing.partitionKeyOrdering.enabled. Requires spark.sql.sources.v2.bucketing.enabled to be enabled. 4.2.0 spark.sql.sources.v2.bucketing.preserveOrderingOnCoalesce.enabled false When turned on, GroupPartitionsExec will use sorted merge to preserve full ordering (as opposed to the key-derived ordering preserved by spark.sql.sources.v2.bucketing.preserveKeyOrderingOnCoalesce.enabled) when coalescing multiple partitions with the same key. This allows eliminating downstream sorts when data is both partitioned and sorted. When this config is enabled, the effect of spark.sql.sources.v2.bucketing.preserveKeyOrderingOnCoalesce.enabled is fully ordering implies key-derived ordering. However, sorted merge uses more resources (priority queue, comparison overhead) than simple concatenation, especially when coalescing many partitions. When turned off, only key-derived ordering is preserved during coalescing. This config requires spark.sql.sources.v2.bucketing.enabled to be enabled. 4.2.0 spark.sql.sources.v2.bucketing.pushPartValues.enabled true Whether to pushdown common partition values when spark.sql.sources.v2.bucketing.enabled is enabled. When turned on, if both sides of a join are of KeyedPartitioning and if they share compatible partition keys, even if they don't have the exact same partition values, Spark will calculate a superset of partition values and pushdown that info to group partition nodes, which will use empty partitions for the missing partition values on either side. This could help to eliminate unnecessary shuffles 3.4.0 spark.sql.sources.v2.bucketing.shuffle.enabled false During a storage-partitioned join, whether to allow to shuffle only one side. When only one side is KeyedPartitioning, if the conditions are met, spark will only shuffle the other side. This optimization will reduce the amount of data that needs to be shuffle. This config requires spark.sql.sources.v2.bucketing.enabled to be enabled 4.0.0 spark.sql.sources.v2.bucketing.sorting.enabled false When turned on, Spark will recognize the specific distribution reported by a V2 data source through SupportsReportPartitioning, and will try to avoid a shuffle if possible when sorting by those columns. This config requires spark.sql.sources.v2.bucketing.enabled to be enabled. 4.0.0 spark.sql.stackTracesInDataFrameContext 1 The number of non-Spark stack traces in the captured DataFrame query context. 4.0.0 spark.sql.statistics.fallBackToHdfs false When true, it will fall back to HDFS if the table statistics are not available from table metadata. This is useful in determining if a table is small enough to use broadcast joins. This flag is effective only for non-partitioned Hive tables. For non-partitioned data source tables, it will be automatically recalculated if table statistics are not available. For partitioned data source and partitioned Hive tables, It is 'spark.sql.defaultSizeInBytes' if table statistics are not available. 2.0.0 spark.sql.statistics.histogram.enabled false Generates histograms when computing column statistics if enabled. Histograms can provide better estimation accuracy. Currently, Spark only supports equi-height histogram. Note that collecting histograms takes extra cost. For example, collecting column statistics usually takes only one table scan, but generating equi-height histogram will cause an extra table scan. 2.3.0 spark.sql.statistics.size.autoUpdate.enabled false Enables automatic update for table size once table's data is changed. Note that if the total number of files of the table is very large, this can be expensive and slow down data change commands. 2.3.0 spark.sql.statistics.updatePartitionStatsInAnalyzeTable.enabled false When this config is enabled, Spark will also update partition statistics in analyze table command (i.e., ANALYZE TABLE .. COMPUTE STATISTICS [NOSCAN]). Note the command will also become more expensive. When this config is disabled, Spark will only update table level statistics. 4.0.0 spark.sql.storeAssignmentPolicy ANSI When inserting a value into a column with different data type, Spark will perform type coercion. Currently, we support 3 policies for the type coercion , legacy and strict. With ANSI policy, Spark performs the type coercion as per ANSI SQL. In practice, the behavior is mostly the same as PostgreSQL. It disallows certain unreasonable type conversions such as converting string to int or double to boolean. With legacy policy, Spark allows the type coercion as long as it is a valid Cast, which is very loose. e.g. converting string to int or double to boolean is allowed. It is also the only behavior in Spark 2.x and it is compatible with Hive. With strict policy, Spark doesn't allow any possible precision loss or data truncation in type coercion, e.g. converting double to int or decimal to double is not allowed. 3.0.0 spark.sql.streaming.checkpointLocation (none) The default location for storing checkpoint data for streaming queries. 2.0.0 spark.sql.streaming.continuous.epochBacklogQueueSize 10000 The max number of entries to be stored in queue to wait for late epochs. If this parameter is exceeded by the size of the queue, stream will stop with an error. 3.0.0 spark.sql.streaming.disabledV2Writers A comma-separated list of fully qualified data source register class names for which StreamWriteSupport is disabled. Writes to these sources will fall back to the V1 Sinks. 2.3.1 spark.sql.streaming.fileSource.cleaner.numThreads 1 Number of threads used in the file source completed file cleaner. 3.0.0 spark.sql.streaming.forceDeleteTempCheckpointLocation false When true, enable temporary checkpoint locations force delete. 3.0.0 spark.sql.streaming.metricsEnabled false Whether Dropwizard/Codahale metrics will be reported for active streaming queries. 2.0.2 spark.sql.streaming.multipleWatermarkPolicy min Policy to calculate the global watermark value when there are multiple watermark operators in a streaming query. The default value is 'min' which chooses the minimum watermark reported across multiple operators. Other alternative value is 'max' which chooses the maximum across multiple operators. configuration cannot be changed between query restarts from the same checkpoint location. 2.4.0 spark.sql.streaming.noDataMicroBatches.enabled true Whether streaming micro-batch engine will execute batches without data for eager state management for stateful streaming queries. 2.4.1 spark.sql.streaming.numRecentProgressUpdates 100 The number of progress updates to retain for a streaming query 2.1.1 spark.sql.streaming.realTimeMode.allowlistCheck true Whether to check all operators, sinks used in real-time mode are in the allowlist. 4.1.0 spark.sql.streaming.realTimeMode.minBatchDuration 5000ms The minimum long-running batch duration in milliseconds for real-time mode. 4.1.0 spark.sql.streaming.sessionWindow.merge.sessions.in.local.partition false When true, streaming session window sorts and merge sessions in local partition prior to shuffle. This is to reduce the rows to shuffle, but only beneficial when there're lots of rows in a batch being assigned to same sessions. 3.2.0 spark.sql.streaming.stateStore.commitValidation.enabled true When true, Spark will validate that all StateStore instances have committed for stateful streaming queries using foreachBatch. This helps detect cases where user-defined functions in foreachBatch (e.g., show(), limit()) don't process all partitions, which can lead to incorrect results. The validation only applies to foreachBatch sinks without global aggregates or limits. 4.1.0 spark.sql.streaming.stateStore.encodingFormat unsaferow The encoding format used for stateful operators to store information in the state store 4.0.0 spark.sql.streaming.stateStore.stateSchemaCheck true When true, Spark will validate the state schema against schema on existing state and fail query if it's incompatible. 3.1.0 spark.sql.streaming.stopActiveRunOnRestart true Running multiple runs of the same streaming query concurrently is not supported. If we find a concurrent active run for a streaming query (in the same or different SparkSessions on the same cluster) and this flag is true, we will stop the old streaming query run to start the new one. 3.0.0 spark.sql.streaming.stopTimeout 0 How long to wait in milliseconds for the streaming execution thread to stop when calling the streaming query's stop() method. 0 or negative values wait indefinitely. 3.0.0 spark.sql.streaming.transformWithState.stateSchemaVersion 3 The version of the state schema used by the transformWithState operator 4.0.0 spark.sql.streaming.validateEventTimeWatermarkColumn true When true, check that eventTime in withWatermark is a top-level column. 4.2.0 spark.sql.thriftServer.interruptOnCancel (value of spark.sql.execution.interruptOnCancel) When true, all running tasks will be interrupted if one cancels a query. When false, all running tasks will remain until finished. 3.2.0 spark.sql.thriftServer.queryTimeout 0ms Set a query duration timeout in seconds in Thrift Server. If the timeout is set to a positive value, a running query will be cancelled automatically when the timeout is exceeded, otherwise the query continues to run till completion. If timeout values are set for each statement via java.sql.Statement.setQueryTimeout and they are smaller than this configuration value, they take precedence. If you set this timeout and prefer to cancel the queries right away without waiting task to finish, consider enabling spark.sql.thriftServer.interruptOnCancel together. 3.1.0 spark.sql.thriftserver.scheduler.pool (none) Set a Fair Scheduler pool for a JDBC client session. 1.1.1 spark.sql.thriftserver.shuffleDependency.fileCleanup.enabled false When enabled, shuffle files will be cleaned up at the end of Thrift server SQL executions. 4.2.0 spark.sql.thriftserver.ui.retainedSessions 200 The number of SQL client sessions kept in the JDBC/ODBC web UI history. 1.4.0 spark.sql.thriftserver.ui.retainedStatements 200 The number of SQL statements kept in the JDBC/ODBC web UI history. 1.4.0 spark.sql.timeTravelTimestampKey timestampAsOf The option name to specify the time travel timestamp when reading a table. 4.0.0 spark.sql.timeTravelVersionKey versionAsOf The option name to specify the time travel table version when reading a table. 4.0.0 spark.sql.timestampType TIMESTAMP_LTZ Configures the default timestamp type of Spark SQL, including SQL DDL, Cast clause, type literal and the schema inference of data sources. Setting the configuration as TIMESTAMP_NTZ will use TIMESTAMP WITHOUT TIME ZONE as the default type while putting it as TIMESTAMP_LTZ will use TIMESTAMP WITH LOCAL TIME ZONE. Before the 3.4.0 release, Spark only supports the TIMESTAMP WITH LOCAL TIME ZONE type. 3.4.0 spark.sql.transposeMaxValues 500 When doing a transpose without specifying values for the index column this is the maximum number of values that will be transposed without error. 4.0.0 spark.sql.tvf.allowMultipleTableArguments.enabled false When true, allows multiple table arguments for table-valued functions, receiving the cartesian product of all the rows of these tables. 3.5.0 spark.sql.udt.allowCreatingUDTFromString true When true, Spark loads and instantiates the UserDefinedType class named in a schema string (for example the schema stored in Parquet/ORC file metadata) while inferring or parsing a schema. Because the class name is taken from the data being read, a crafted file can make Spark load an arbitrary class from the classpath. Set this to false to block loading UDT classes by name, optionally allowing specific classes via 'spark.sql.udt.allowedDynamicUDTClasses'. 4.1.3 spark.sql.udt.allowedDynamicUDTClasses When 'spark.sql.udt.allowCreatingUDTFromString' is false, UserDefinedType classes listed here (by fully qualified class name) may still be loaded and instantiated from a schema string. Has no effect when UDT loading is enabled. 4.1.3 spark.sql.ui.explainMode formatted Configures the query explain mode used in the Spark SQL UI. The value can be 'simple', 'extended', 'codegen', 'cost', or 'formatted'. The default value is 'formatted'. 3.1.0 spark.sql.variable.substitute true This enables substitution using syntax like ${var}, ${system:var}, and ${env:var}. 2.0.0 spark.sql.window.segmentTree.blockSize 65536 Block size, in rows, for the block-chunked segment tree used by moving window frames. Each leaf of the tree aggregates this many consecutive rows. Smaller values reduce per-partition memory and speed up tree build for small partitions, but make the tree deeper and increase query cost for wide frames. Larger values amortize build cost and shrink the tree but increase the per-block prefix/suffix scan cost within a block. The default is tuned for partitions on the order of tens of thousands to millions of rows. 4.2.0 spark.sql.window.segmentTree.enabled false Use block-chunked segment tree for moving aggregate window frames whose functions are all DeclarativeAggregate without FILTER/DISTINCT. 4.2.0 spark.sql.xml.variant.respectInferSchema true Kill switch for the SPARK-56554 fix. When true (default), the XML to Variant parser honors the 'inferSchema' 'inferSchema' is false, primitive leaf values (text and attributes) are preserved as strings inside the Variant instead of being inferred as boolean, long, or decimal. Set this conf to false to restore the pre-SPARK-56554 behavior of always inferring types regardless of the 'inferSchema' option. 4.1.0 Static SQL Configuration Static SQL configurations are cross-session, immutable Spark SQL configurations. They can be set with final values by the config file and command-line options with --conf/-c prefixed, or by setting SparkConf that are used to create SparkSession. External users can query the static sql config values via SparkSession.conf or via set command, e.g. SET spark.sql.extensions;, but cannot set/unset them. Property NameDefaultMeaningSince Version spark.sql.cache.serializer org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer The name of a class that implements org.apache.spark.sql.columnar.CachedBatchSerializer. It will be used to translate SQL data into a format that can more efficiently be cached. The underlying API is subject to change so use with caution. Multiple classes cannot be specified. The class must have a no-arg constructor. 3.1.0 spark.sql.catalog.spark_catalog.defaultDatabase default The default database for session catalog. 3.4.0 spark.sql.event.truncate.length 2147483647 Threshold of SQL length beyond which it will be truncated before adding to event. Defaults to no truncation. If set to 0, callsite will be logged instead. 3.0.0 spark.sql.extensions (none) A comma-separated list of classes that implement Function1[SparkSessionExtensions, Unit] used to configure Spark Session extensions. The classes must have a no-args constructor. If multiple extensions are specified, they are applied in the specified order. For the case of rules and planner strategies, they are applied in the specified order. For the case of parsers, the last parser is used and each parser can delegate to its predecessor. For the case of function name conflicts, the last registered function name is used. 2.2.0 spark.sql.extensions.test.loadFromCp true Flag that determines if we should load extensions from the classpath using the SparkSessionExtensionsProvider mechanism. This is a test only flag. spark.sql.hive.metastore.barrierPrefixes A comma separated list of class prefixes that should explicitly be reloaded for each version of Hive that Spark SQL is communicating with. For example, Hive UDFs that are declared in a prefix that typically would be shared (i.e. org.apache.spark.*). 1.4.0 spark.sql.hive.metastore.jars builtin Location of the jars that should be used to instantiate the HiveMetastoreClient. This property can be one of four \"builtin\" Use Hive 2.3.10, which is bundled with the Spark assembly when -Phive is enabled. When this option is chosen, spark.sql.hive.metastore.version must be either 2.3.10 or not defined. 2. \"maven\" Use Hive jars of specified version downloaded from Maven repositories. 3. \"path\" Use Hive jars configured by spark.sql.hive.metastore.jars.path in comma separated format. Support both local or remote paths.The provided jars should be the same version as spark.sql.hive.metastore.version. 4. A classpath in the standard format for both Hive and Hadoop. The provided jars should be the same version as spark.sql.hive.metastore.version. 1.4.0 spark.sql.hive.metastore.jars.path Comma-separated paths of the jars that used to instantiate the HiveMetastoreClient. This configuration is useful only when spark.sql.hive.metastore.jars is set as path. The paths can be any of the following file://path/to/jar/foo.jar 2. hdfs://nameservice/path/to/jar/foo.jar 3. /path/to/jar/ (path without URI scheme follow conf fs.defaultFS's URI schema) 4. [http/https/ftp]://path/to/jar/foo.jar Note that 1, 2, and 3 support wildcard. For file://path/to/jar/,file://path2/to/jar//.jar 2. hdfs://nameservice/path/to/jar/,hdfs://nameservice2/path/to/jar//.jar 3.1.0 spark.sql.hive.metastore.sharedPrefixes com.mysql.jdbc,com.mysql.cj,org.postgresql,com.microsoft.sqlserver,oracle.jdbc A comma separated list of class prefixes that should be loaded using the classloader that is shared between Spark SQL and a specific version of Hive. An example of classes that should be shared is JDBC drivers that are needed to talk to the metastore. Other classes that need to be shared are those that interact with classes that are already shared. For example, custom appenders that are used by log4j. 1.4.0 spark.sql.hive.metastore.version 2.3.10 Version of the Hive metastore. Available options are 2.0.0 through 2.3.10, 3.0.0 through 3.1.3 and 4.0.0 through 4.1.0. 1.4.0 spark.sql.hive.thriftServer.singleSession false When set to true, Hive Thrift server is running in a single session mode. All the JDBC/ODBC connections share the temporary views, function registries, SQL configuration and the current database. 1.6.0 spark.sql.hive.version 2.3.10 The compiled, a.k.a, builtin Hive version of the Spark distribution bundled with. Note that, this a read-only conf and only used to report the built-in hive version. If you want a different metastore client for Spark to call, please refer to spark.sql.hive.metastore.version. 1.1.1 spark.sql.metadataCacheTTLSeconds -1000ms Time-to-live (TTL) value for the metadata file metadata cache and session catalog cache. This configuration only has an effect when this value having a positive value (> 0). It also requires setting 'spark.sql.catalogImplementation' to hive, setting 'spark.sql.hive.filesourcePartitionFileCacheSize' > 0 and setting 'spark.sql.hive.manageFilesourcePartitions' to true to be applied to the partition file metadata cache. 3.1.0 spark.sql.queryExecutionListeners (none) List of class names implementing QueryExecutionListener that will be automatically added to newly created sessions. The classes should have either a no-arg constructor, or a constructor that expects a SparkConf argument. 2.3.0 spark.sql.sources.disabledJdbcConnProviderList Configures a list of JDBC connection providers, which are disabled. The list contains the name of the JDBC connection providers separated by comma. 3.1.0 spark.sql.streaming.streamingQueryListeners (none) List of class names implementing StreamingQueryListener that will be automatically added to newly created sessions. The classes should have either a no-arg constructor, or a constructor that expects a SparkConf argument. 2.4.0 spark.sql.streaming.ui.enabled true Whether to run the Structured Streaming Web UI for the Spark application when the Spark Web UI is enabled. 3.0.0 spark.sql.streaming.ui.retainedProgressUpdates 100 The number of progress updates to retain for a streaming query for Structured Streaming UI. 3.0.0 spark.sql.streaming.ui.retainedQueries 100 The number of inactive queries to retain for Structured Streaming UI. 3.0.0 spark.sql.ui.retainedExecutions 1000 Number of executions to retain in the Spark UI. 1.5.0 spark.sql.warehouse.dir (value of $PWD/spark-warehouse) The default location for managed databases and tables. 2.0.0 Spark Streaming Property NameDefaultMeaningSince Version spark.streaming.backpressure.enabled false Enables or disables Spark Streaming's internal backpressure mechanism (since 1.5). This enables the Spark Streaming to control the receiving rate based on the current batch scheduling delays and processing times so that the system receives only as fast as the system can process. Internally, this dynamically sets the maximum receiving rate of receivers. This rate is upper bounded by the values spark.streaming.receiver.maxRate and spark.streaming.kafka.maxRatePerPartition if they are set (see below). 1.5.0 spark.streaming.backpressure.initialRate not set This is the initial maximum receiving rate at which each receiver will receive data for the first batch when the backpressure mechanism is enabled. 2.0.0 spark.streaming.blockInterval 200ms Interval at which data received by Spark Streaming receivers is chunked into blocks of data before storing them in Spark. Minimum recommended - 50 ms. See the performance tuning section in the Spark Streaming programming guide for more details. 0.8.0 spark.streaming.receiver.maxRate not set Maximum rate (number of records per second) at which each receiver will receive data. Effectively, each stream will consume at most this number of records per second. Setting this configuration to 0 or a negative number will put no limit on the rate. See the deployment guide in the Spark Streaming programming guide for mode details. 1.0.2 spark.streaming.receiver.writeAheadLog.enable false Enable write-ahead logs for receivers. All the input data received through receivers will be saved to write-ahead logs that will allow it to be recovered after driver failures. See the deployment guide in the Spark Streaming programming guide for more details. 1.2.1 spark.streaming.unpersist true Force RDDs generated and persisted by Spark Streaming to be automatically unpersisted from Spark's memory. The raw input data received by Spark Streaming is also automatically cleared. Setting this to false will allow the raw data and persisted RDDs to be accessible outside the streaming application as they will not be cleared automatically. But it comes at the cost of higher memory usage in Spark. 0.9.0 spark.streaming.stopGracefullyOnShutdown false If true, Spark shuts down the StreamingContext gracefully on JVM shutdown rather than immediately. 1.4.0 spark.streaming.kafka.maxRatePerPartition not set Maximum rate (number of records per second) at which data will be read from each Kafka partition when using the new Kafka direct stream API. See the Kafka Integration guide for more details. 1.3.0 spark.streaming.kafka.minRatePerPartition 1 Minimum rate (number of records per second) at which data will be read from each Kafka partition when using the new Kafka direct stream API. 2.4.0 spark.streaming.ui.retainedBatches 1000 How many batches the Spark Streaming UI and status APIs remember before garbage collecting. 1.0.0 spark.streaming.driver.writeAheadLog.closeFileAfterWrite false Whether to close the file after writing a write-ahead log record on the driver. Set this to 'true' when you want to use S3 (or any file system that does not support flushing) for the metadata WAL on the driver. 1.6.0 spark.streaming.receiver.writeAheadLog.closeFileAfterWrite false Whether to close the file after writing a write-ahead log record on the receivers. Set this to 'true' when you want to use S3 (or any file system that does not support flushing) for the data WAL on the receivers. 1.6.0 SparkR (deprecated) Property NameDefaultMeaningSince Version spark.r.numRBackendThreads 2 Number of threads used by RBackend to handle RPC calls from SparkR package. 1.4.0 spark.r.command Rscript Executable for executing R scripts in cluster modes for both driver and workers. 1.5.3 spark.r.driver.command spark.r.command Executable for executing R scripts in client modes for driver. Ignored in cluster modes. 1.5.3 spark.r.shell.command R Executable for executing sparkR shell in client modes for driver. Ignored in cluster modes. It is the same as environment variable SPARKR_DRIVER_R, but take precedence over it. spark.r.shell.command is used for sparkR shell while spark.r.driver.command is used for running R script. 2.1.0 spark.r.backendConnectionTimeout 6000 Connection timeout set by R process on its connection to RBackend in seconds. 2.1.0 spark.r.heartBeatInterval 100 Interval for heartbeats sent from SparkR backend to R process to prevent connection timeout. 2.1.0 GraphX Property NameDefaultMeaningSince Version spark.graphx.pregel.checkpointInterval -1 Checkpoint interval for graph and message in Pregel. It used to avoid stackOverflowError due to long lineage chains after lots of iterations. The checkpoint is disabled by default. 2.2.0 Cluster Managers Each cluster manager in Spark has additional configuration options. Configurations can be found on the pages for each Kubernetes Standalone Mode Environment Variables Certain Spark settings can be configured through environment variables, which are read from the conf/spark-env.sh script in the directory where Spark is installed (or conf/spark-env.cmd on Windows). In Standalone mode, this file can give machine specific information such as hostnames. It is also sourced when running local Spark applications or submission scripts. Note that conf/spark-env.sh does not exist by default when Spark is installed. However, you can copy conf/spark-env.sh.template to create it. Make sure you make the copy executable. The following variables can be set in spark-env.sh: Environment VariableMeaning JAVA_HOME Location where Java is installed (if it's not on your default PATH). PYSPARK_PYTHON Python binary executable to use for PySpark in both driver and workers (default is python3 if available, otherwise python). Property spark.pyspark.python take precedence if it is set PYSPARK_DRIVER_PYTHON Python binary executable to use for PySpark in driver only (default is PYSPARK_PYTHON). Property spark.pyspark.driver.python take precedence if it is set SPARKR_DRIVER_R R binary executable to use for SparkR shell (default is R). Property spark.r.shell.command take precedence if it is set SPARK_LOCAL_IP IP address of the machine to bind to. SPARK_PUBLIC_DNS Hostname your Spark program will advertise to other machines. In addition to the above, there are also options for setting up the Spark standalone cluster scripts, such as number of cores to use on each machine and maximum memory. Since spark-env.sh is a shell script, some of these can be set programmatically – for example, you might compute SPARK_LOCAL_IP by looking up the IP of a specific network interface. running Spark on YARN in cluster mode, environment variables need to be set using the spark.yarn.appMasterEnv.[EnvironmentVariableName] property in your conf/spark-defaults.conf file. Environment variables that are set in spark-env.sh will not be reflected in the YARN Application Master process in cluster mode. See the YARN-related Spark Properties for more information. Configuring Logging Spark uses log4j for logging. You can configure it by adding a log4j2.properties file in the conf directory. To get started, copy one of the provided (for plain text logging) or log4j2-json-layout.properties.template (for structured logging). Plain Text Logging The default logging format is plain text, using Log4j’s Pattern Layout. MDC (Mapped Diagnostic Context) information is not included by default in plain text logs. To include it, update the PatternLayout configuration in the log4j2.properties file. For example, add %X{task_name} to include the task name in logs. Additionally, use spark.sparkContext.setLocalProperty(\"key\", \"value\") to add custom data to the MDC. Structured Logging Starting with version 4.0.0, spark-submit supports optional structured logging using the JSON Template Layout. This format enables efficient querying of logs with Spark SQL using the JSON data source and includes all MDC information for improved searchability and debugging. To enable structured logging and include MDC information, set the configuration spark.log.structuredLogging.enabled to true (default is false). For additional customization, copy log4j2-json-layout.properties.template to conf/log4j2.properties and adjust as needed. Querying Structured Logs with Spark SQL To query structured logs in JSON format, use the following code : from pyspark.logger import SPARK_LOG_SCHEMA logDf = spark.read.schema(SPARK_LOG_SCHEMA).json(\"path/to/logs\") org.apache.spark.util.LogUtils.SPARK_LOG_SCHEMA val logDf = spark.read.schema(SPARK_LOG_SCHEMA).json(\"path/to/logs\") you’re using the interactive shell (pyspark shell or spark-shell), you can omit the import statement in the code because SPARK_LOG_SCHEMA is already available in the shell’s context. Overriding configuration directory To specify a different configuration directory other than the default “SPARK_HOME/conf”, you can set SPARK_CONF_DIR. Spark will use the configuration files (spark-defaults.conf, spark-env.sh, log4j2.properties, etc) from this directory. Inheriting Hadoop Cluster Configuration If you plan to read and write from HDFS using Spark, there are two Hadoop configuration files that should be included on Spark’s , which provides default behaviors for the HDFS client. core-site.xml, which sets the default filesystem name. The location of these configuration files varies across Hadoop versions, but a common location is inside of /etc/hadoop/conf. Some tools create configurations on-the-fly, but offer a mechanism to download copies of them. To make these files visible to Spark, set HADOOP_CONF_DIR in $SPARK_HOME/conf/spark-env.sh to a location containing the configuration files. Custom Hadoop/Hive Configuration If your Spark application is interacting with Hadoop, Hive, or both, there are probably Hadoop/Hive configuration files in Spark’s classpath. Multiple running applications might require different Hadoop/Hive client side configurations. You can copy and modify hdfs-site.xml, core-site.xml, yarn-site.xml, hive-site.xml in Spark’s classpath for each application. In a Spark cluster running on YARN, these configuration files are set cluster-wide, and cannot safely be changed by the application. The better choice is to use spark hadoop properties in the form of spark.hadoop.*, and use spark hive properties in the form of spark.hive.*. For example, adding configuration “spark.hadoop.abc.def=xyz” represents adding hadoop property “abc.def=xyz”, and adding configuration “spark.hive.abc=xyz” represents adding hive property “hive.abc=xyz”. They can be considered as same as normal spark properties which can be set in $SPARK_HOME/conf/spark-defaults.conf In some cases, you may want to avoid hard-coding certain configurations in a SparkConf. For instance, Spark allows you to simply create an empty conf and set spark/spark hadoop/spark hive properties. val conf = new SparkConf().set(\"spark.hadoop.abc.def\", \"xyz\") val sc = new SparkContext(conf) Also, you can modify or add configurations at /bin/spark-submit \\ --name \"My app\" \\ --master \"local[4]\" \\ --conf spark.eventLog.enabled=false \\ --conf \"spark.executor.extraJavaOptions=-XX:+PrintGCDetails -XX:+PrintGCTimeStamps\" \\ --conf spark.hadoop.abc.def=xyz \\ --conf spark.hive.abc=xyz myApp.jar Custom Resource Scheduling and Configuration Overview GPUs and other accelerators have been widely used for accelerating special workloads, e.g., deep learning and signal processing. Spark now supports requesting and scheduling generic resources, such as GPUs, with a few caveats. The current implementation requires that the resource have addresses that can be allocated by the scheduler. It requires your cluster manager to support and be properly configured with the resources. There are configurations available to request resources for the {resourceName}.amount, request resources for the executor(s): spark.executor.resource.{resourceName}.amount and specify the requirements for each {resourceName}.amount. The spark.driver.resource.{resourceName}.discoveryScript config is required on YARN, Kubernetes and a client side Driver on Spark Standalone. spark.executor.resource.{resourceName}.discoveryScript config is required for YARN and Kubernetes. Kubernetes also requires spark.driver.resource.{resourceName}.vendor and/or spark.executor.resource.{resourceName}.vendor. See the config descriptions above for more information on each. Spark will use the configurations specified to first request containers with the corresponding resources from the cluster manager. Once it gets the container, Spark launches an Executor in that container which will discover what resources the container has and the addresses associated with each resource. The Executor will register with the Driver and report back the resources available to that Executor. The Spark scheduler can then schedule tasks to each Executor and assign specific resource addresses based on the resource requirements the user specified. The user can see the resources assigned to a task using the TaskContext.get().resources api. On the driver, the user can see the resources assigned with the SparkContext resources call. It’s then up to the user to use the assigned addresses to do the processing they want or pass those into the ML/AI framework they are using. See your cluster manager specific page for requirements and details on each of - YARN, Kubernetes and Standalone Mode. It is currently not available with local mode. And please also note that local-cluster mode with multiple workers is not supported(see Standalone documentation). Stage Level Scheduling Overview The stage level scheduling feature allows users to specify task and executor resource requirements at the stage level. This allows for different stages to run with executors that have different resources. A prime example of this is one ETL stage runs with executors with just CPUs, the next stage is an ML stage that needs GPUs. Stage level scheduling allows for user to request different executors that have GPUs when the ML stage runs rather then having to acquire executors with GPUs at the start of the application and them be idle while the ETL stage is being run. This is only available for the RDD API in Scala, Java, and Python. It is available on YARN, Kubernetes and Standalone when dynamic allocation is enabled. When dynamic allocation is disabled, it allows users to specify different task resource requirements at stage level, and this is supported on YARN, Kubernetes and Standalone cluster right now. See the YARN page or Kubernetes page or Standalone page for more implementation details. See the RDD.withResources and ResourceProfileBuilder API’s for using this feature. When dynamic allocation is disabled, tasks with different task resource requirements will share executors with DEFAULT_RESOURCE_PROFILE. While when dynamic allocation is enabled, the current implementation acquires new executors for each ResourceProfile created and currently has to be an exact match. Spark does not try to fit tasks into an executor that require a different ResourceProfile than the executor was created with. Executors that are not in use will idle timeout with the dynamic allocation logic. The default configuration for this feature is to only allow one ResourceProfile per stage. If the user associates more then 1 ResourceProfile to an RDD, Spark will throw an exception by default. See config spark.scheduler.resource.profileMergeConflicts to control that behavior. The current merge strategy Spark implements when spark.scheduler.resource.profileMergeConflicts is enabled is a simple max of each resource within the conflicting ResourceProfiles. Spark will create a new ResourceProfile with the max of each of the resources. Push-based shuffle overview Push-based shuffle helps improve the reliability and performance of spark shuffle. It takes a best-effort approach to push the shuffle blocks generated by the map tasks to remote external shuffle services to be merged per shuffle partition. Reduce tasks fetch a combination of merged shuffle partitions and original shuffle blocks as their input data, resulting in converting small random disk reads by external shuffle services into large sequential reads. Possibility of better data locality for reduce tasks additionally helps minimize network IO. Push-based shuffle takes priority over batch fetch for some scenarios, like partition coalesce when merged output is available. Push-based shuffle improves performance for long running jobs/queries which involves large disk I/O during shuffle. Currently it is not well suited for jobs/queries which runs quickly dealing with lesser amount of shuffle data. This will be further improved in the future releases. Currently push-based shuffle is only supported for Spark on YARN with external shuffle service. External Shuffle service(server) side configuration options Property NameDefaultMeaningSince Version spark.shuffle.push.server.mergedShuffleFileManagerImpl org.apache.spark.network.shuffle.NoOpMergedShuffleFileManager Class name of the implementation of MergedShuffleFileManager that manages push-based shuffle. This acts as a server side config to disable or enable push-based shuffle. By default, push-based shuffle is disabled at the server side. To enable push-based shuffle on the server side, set this config to org.apache.spark.network.shuffle.RemoteBlockPushResolver 3.2.0 spark.shuffle.push.server.minChunkSizeInMergedShuffleFile 2m The minimum size of a chunk when dividing a merged shuffle file into multiple chunks during push-based shuffle. A merged shuffle file consists of multiple small shuffle blocks. Fetching the complete merged shuffle file in a single disk I/O increases the memory requirements for both the clients and the external shuffle services. Instead, the external shuffle service serves the merged file in MB-sized chunks. This configuration controls how big a chunk can get. A corresponding index file for each merged shuffle file will be generated indicating chunk boundaries. Setting this too high would increase the memory requirements on both the clients and the external shuffle service. Setting this too low would increase the overall number of RPC requests to external shuffle service unnecessarily. 3.2.0 spark.shuffle.push.server.mergedIndexCacheSize 100m The maximum size of cache in memory which could be used in push-based shuffle for storing merged index files. This cache is in addition to the one configured via spark.shuffle.service.index.cache.size. 3.2.0 Client side configuration options Property NameDefaultMeaningSince Version spark.shuffle.push.enabled false Set to true to enable push-based shuffle on the client side and works in conjunction with the server side flag spark.shuffle.push.server.mergedShuffleFileManagerImpl. 3.2.0 spark.shuffle.push.finalize.timeout 10s The amount of time driver waits in seconds, after all mappers have finished for a given shuffle map stage, before it sends merge finalize requests to remote external shuffle services. This gives the external shuffle services extra time to merge blocks. Setting this too long could potentially lead to performance regression. 3.2.0 spark.shuffle.push.maxRetainedMergerLocations 500 Maximum number of merger locations cached for push-based shuffle. Currently, merger locations are hosts of external shuffle services responsible for handling pushed blocks, merging them and serving merged blocks for later shuffle fetch. 3.2.0 spark.shuffle.push.mergersMinThresholdRatio 0.05 Ratio used to compute the minimum number of shuffle merger locations required for a stage based on the number of partitions for the reducer stage. For example, a reduce stage which has 100 partitions and uses the default value 0.05 requires at least 5 unique merger locations to enable push-based shuffle. 3.2.0 spark.shuffle.push.mergersMinStaticThreshold 5 The static threshold for number of shuffle push merger locations should be available in order to enable push-based shuffle for a stage. Note this config works in conjunction with spark.shuffle.push.mergersMinThresholdRatio. Maximum of spark.shuffle.push.mergersMinStaticThreshold and spark.shuffle.push.mergersMinThresholdRatio ratio number of mergers needed to enable push-based shuffle for a stage. For 1000 partitions for the child stage with spark.shuffle.push.mergersMinStaticThreshold as 5 and spark.shuffle.push.mergersMinThresholdRatio set to 0.05, we would need at least 50 mergers to enable push-based shuffle for that stage. 3.2.0 spark.shuffle.push.numPushThreads (none) Specify the number of threads in the block pusher pool. These threads assist in creating connections and pushing blocks to remote external shuffle services. By default, the threadpool size is equal to the number of spark executor cores. 3.2.0 spark.shuffle.push.maxBlockSizeToPush 1m The max size of an individual block to push to the remote external shuffle services. Blocks larger than this threshold are not pushed to be merged remotely. These shuffle blocks will be fetched in the original manner. Setting this too high would result in more blocks to be pushed to remote external shuffle services but those are already efficiently fetched with the existing mechanisms resulting in additional overhead of pushing the large blocks to remote external shuffle services. It is recommended to set spark.shuffle.push.maxBlockSizeToPush lesser than spark.shuffle.push.maxBlockBatchSize config's value. Setting this too low would result in lesser number of blocks getting merged and directly fetched from mapper external shuffle service results in higher small random reads affecting overall disk I/O performance. 3.2.0 spark.shuffle.push.maxBlockBatchSize 3m The max size of a batch of shuffle blocks to be grouped into a single push request. Default is set to 3m in order to keep it slightly higher than spark.storage.memoryMapThreshold default which is 2m as it is very likely that each batch of block gets memory mapped which incurs higher overhead. 3.2.0 spark.shuffle.push.merge.finalizeThreads 8 Number of threads used by driver to finalize shuffle merge. Since it could potentially take seconds for a large shuffle to finalize, having multiple threads helps driver to handle concurrent shuffle merge finalize requests when push-based shuffle is enabled. 3.3.0 spark.shuffle.push.minShuffleSizeToWait 500m Driver will wait for merge finalization to complete only if total shuffle data size is more than this threshold. If total shuffle size is less, driver will immediately finalize the shuffle output. 3.3.0 spark.shuffle.push.minCompletedPushRatio 1.0 Fraction of minimum map partitions that should be push complete before driver starts shuffle merge finalization during push based shuffle. 3.3.0\n\nExample:\n```scala\nval conf = new SparkConf()\n             .setMaster(\"local[2]\")\n             .setAppName(\"CountingSheep\")\nval sc = new SparkContext(conf)\n```\n\nExample:\n```text\n25ms (milliseconds)\n5s (seconds)\n10m or 10min (minutes)\n3h (hours)\n5d (days)\n1y (years)\n```\n\nExample:\n```text\n1b (bytes)\n1k or 1kb (kibibytes = 1024 bytes)\n1m or 1mb (mebibytes = 1024 kibibytes)\n1g or 1gb (gibibytes = 1024 mebibytes)\n1t or 1tb (tebibytes = 1024 gibibytes)\n1p or 1pb (pebibytes = 1024 tebibytes)\n```\n\nExample:\n```scala\nval sc = new SparkContext(new SparkConf())\n```\n\nExample:\n```text\n./bin/spark-submit \\\n  --name \"My app\" \\\n  --master \"local[4]\" \\\n  --conf spark.eventLog.enabled=false \\\n  --conf \"spark.executor.extraJavaOptions=-XX:+PrintGCDetails -XX:+PrintGCTimeStamps\" \\\n  myApp.jar\n```\n\nExample:\n```text\nspark.master            spark://5.6.7.8:7077\nspark.executor.memory   4g\nspark.eventLog.enabled  true\nspark.serializer        org.apache.spark.serializer.KryoSerializer\n```\n\nExample:\n```text\nfrom pyspark.logger import SPARK_LOG_SCHEMA\n\nlogDf = spark.read.schema(SPARK_LOG_SCHEMA).json(\"path/to/logs\")\n```\n\nExample:\n```text\nimport org.apache.spark.util.LogUtils.SPARK_LOG_SCHEMA\n\nval logDf = spark.read.schema(SPARK_LOG_SCHEMA).json(\"path/to/logs\")\n```\n\nExample:\n```scala\nval conf = new SparkConf().set(\"spark.hadoop.abc.def\", \"xyz\")\nval sc = new SparkContext(conf)\n```\n\nExample:\n```bash\n./bin/spark-submit \\\n  --name \"My app\" \\\n  --master \"local[4]\" \\\n  --conf spark.eventLog.enabled=false \\\n  --conf \"spark.executor.extraJavaOptions=-XX:+PrintGCDetails -XX:+PrintGCTimeStamps\" \\\n  --conf spark.hadoop.abc.def=xyz \\\n  --conf spark.hive.abc=xyz\n  myApp.jar\n```\n\n<|endofdoc|>","metadata":{"transformedAt":"2026-08-18T15:21:43.192Z","totalSectionsIncluded":1,"totalCodeBlocksIncluded":10,"totalLines":86,"estimatedTokens":36214}}36