flytekitplugins.spark.Spark#

class flytekitplugins.spark.Spark(spark_conf=None, hadoop_conf=None, executor_path=None, applications_path=None, driver_pod=None, executor_pod=None)#

Use this to configure a SparkContext for a your task. Task’s marked with this will automatically execute natively onto K8s as a distributed execution of spark

Parameters:
spark_conf#

Spark configuration dictionary.

Type:

Optional[Dict[str, str]]

hadoop_conf#

Hadoop configuration dictionary.

Type:

Optional[Dict[str, str]]

executor_path#

Path to the Python binary for PySpark execution.

Type:

Optional[str]

applications_path#

Path to the main application file.

Type:

Optional[str]

driver_pod#

The pod template for the Spark driver pod.

Type:

Optional[PodTemplate]

executor_pod#

The pod template for the Spark executor pod.

Type:

Optional[PodTemplate]

Methods

Attributes

applications_path: str | None = None
driver_pod: PodTemplate | None = None
executor_path: str | None = None
executor_pod: PodTemplate | None = None
hadoop_conf: Dict[str, str] | None = None
spark_conf: Dict[str, str] | None = None