开发者指南
开发者指南
克隆代码仓库
克隆 Spark 操作符代码库并切换到该目录：
git clone git@github.com:kubeflow/spark-operator.git
cd spark-operator
(可选) 配置 Git 预提交钩子
Git hooks 对于在提交代码审查之前识别简单问题非常有用。我们在每次提交时运行 hooks，以自动从 README.md.gotmpl 文件生成 helm chart README.md 文件。在运行 git hooks 之前，您需要安装 pre-commit 包管理器，方法如下：
# Using pip
pip install pre-commit
# Using conda
conda install -c conda-forge pre-commit
# Using Homebrew
brew install pre-commit
要设置预提交钩子，请运行以下命令：
pre-commit install
pre-commit install-hooks
使用Makefile
我们使用Makefile来自动化常见任务。例如，要构建操作符，运行如下的 build-operator 目标， spark-operator 二进制文件将被构建并放置在 bin 目录中：
make build-operator
依赖项将根据需要自动下载到 bin 目录。例如，如果您运行 make manifests 目标，则 controller-gen 工具将使用 go install 命令自动下载，然后将其重命名为 controller-gen-vX.Y.Z 并放置在 bin 目录中。
要查看可用目标的完整列表，请运行以下命令：
$ make help
Usage:
make <target>
General
help Display this help.
version Print version information.
Development
manifests Generate CustomResourceDefinition, RBAC and WebhookConfiguration manifests.
generate Generate code containing DeepCopy, DeepCopyInto, and DeepCopyObject method implementations.
update-crd Update CRD files in the Helm chart.
go-clean Clean up caches and output.
go-fmt Run go fmt against code.
go-vet Run go vet against code.
lint Run golangci-lint linter.
lint-fix Run golangci-lint linter and perform fixes.
unit-test Run unit tests.
e2e-test Run the e2e tests against a Kind k8s instance that is spun up.
Build
build-operator Build Spark operator.
build-sparkctl Build sparkctl binary.
install-sparkctl Install sparkctl binary.
clean Clean spark-operator and sparkctl binaries.
build-api-docs Build api documentation.
docker-build Build docker image with the operator.
docker-push Push docker image with the operator.
docker-buildx Build and push docker image for the operator for cross-platform support
Helm
detect-crds-drift Detect CRD drift.
helm-unittest Run Helm chart unittests.
helm-lint Run Helm chart lint test.
helm-docs Generates markdown documentation for helm charts from requirements and values files.
Deployment
kind-create-cluster Create a kind cluster for integration tests.
kind-load-image Load the image into the kind cluster.
kind-delete-custer Delete the created kind cluster.
install-crd Install CRDs into the K8s cluster specified in ~/.kube/config.
uninstall-crd Uninstall CRDs from the K8s cluster specified in ~/.kube/config. Call with ignore-not-found=true to ignore resource not found errors during deletion.
deploy Deploy controller to the K8s cluster specified in ~/.kube/config.
undeploy Undeploy controller from the K8s cluster specified in ~/.kube/config. Call with ignore-not-found=true to ignore resource not found errors during deletion.
Dependencies
kustomize Download kustomize locally if necessary.
controller-gen Download controller-gen locally if necessary.
kind Download kind locally if necessary.
envtest Download setup-envtest locally if necessary.
golangci-lint Download golangci-lint locally if necessary.
gen-crd-api-reference-docs Download gen-crd-api-reference-docs locally if necessary.
helm Download helm locally if necessary.
helm-unittest-plugin Download helm unittest plugin locally if necessary.
helm-docs-plugin Download helm-docs plugin locally if necessary.
使用 Spark Operator 开发
构建二进制文件
要构建该操作符，请运行以下命令：
make build-operator
构建Docker镜像
如果您想从源代码构建操作符，例如测试您编写的修复或功能，您可以按照以下说明进行操作。
构建操作器的最简单方法是使用Dockerfile构建一个镜像，而不必担心它的依赖项。
make docker-build IMAGE_TAG=<image-tag>
操作符镜像是基于一个默认使用 spark:3.5.2 的基础 Spark 镜像构建的。如果你想使用自己的 Spark 镜像（例如，使用不同版本的 Spark 或一些自定义依赖的镜像），可以按以下示例指定参数 SPARK_IMAGE：
docker build --build-arg SPARK_IMAGE=<your Spark image> -t <image-tag> .
更新API定义
如果您已更新API定义，则还需要更新自动生成的代码。要更新包含DeepCopy、DeepCopyInto和DeepCopyObject方法实现的自动生成代码，请运行以下命令：
make generate
要更新自动生成的 CustomResourceDefinition (CRD)、RBAC 和 WebhookConfiguration 清单，请运行以下命令：
make manifests
更新CRD文件后，运行以下命令将CRD文件复制到helm图表目录：
make update-crd
此外，API 规格文档 docs/api-docs.md 也需要更新。要更新文档，请运行以下命令：
make build-api-docs
运行单元测试
要运行单元测试，请运行以下命令：
make unit-test
运行端到端测试
要运行端到端测试，请运行以下命令：
# Create a kind cluster
make kind-create-cluster
# Build docker image
make docker-build IMAGE_TAG=local
# Load docker image to kind cluster
make kind-load-image
# Run e2e tests
make e2e-test
# Delete the kind cluster
make kind-delete-cluster
使用Helm Chart进行开发
运行 Helm Chart 语法检查测试
要运行Helm图表lint测试，请运行以下命令：
$ make helm-lint
Linting charts...
------------------------------------------------------------------------------------------------------------------------
Charts to be processed:
------------------------------------------------------------------------------------------------------------------------
spark-operator => (version: "1.2.4", path: "charts/spark-operator-chart")
------------------------------------------------------------------------------------------------------------------------
Linting chart "spark-operator => (version: \"1.2.4\", path: \"charts/spark-operator-chart\")"
Checking chart "spark-operator => (version: \"1.2.4\", path: \"charts/spark-operator-chart\")" for a version bump...
Old chart version: 1.2.1
New chart version: 1.2.4
Chart version ok.
Validating /Users/user/go/src/github.com/kubeflow/spark-operator/charts/spark-operator-chart/Chart.yaml...
Validation success! 👍
Validating maintainers...
Linting chart with values file "charts/spark-operator-chart/ci/ci-values.yaml"...
==> Linting charts/spark-operator-chart
[INFO] Chart.yaml: icon is recommended
1 chart(s) linted, 0 chart(s) failed
------------------------------------------------------------------------------------------------------------------------
✔︎ spark-operator => (version: "1.2.4", path: "charts/spark-operator-chart")
------------------------------------------------------------------------------------------------------------------------
All charts linted successfully
运行 Helm 图表单元测试
有关如何编写 Helm chart 单元测试的详细信息，请参阅 helm-unittest。要运行 Helm chart 单元测试，请运行以下命令：
$ make helm-unittest
### Chart [ spark-operator ] charts/spark-operator-chart
PASS Test controller deployment charts/spark-operator-chart/tests/controller/deployment_test.yaml
PASS Test controller pod disruption budget charts/spark-operator-chart/tests/controller/poddisruptionbudget_test.yaml
PASS Test controller rbac charts/spark-operator-chart/tests/controller/rbac_test.yaml
PASS Test controller deployment charts/spark-operator-chart/tests/controller/service_test.yaml
PASS Test controller service account charts/spark-operator-chart/tests/controller/serviceaccount_test.yaml
PASS Test prometheus pod monitor charts/spark-operator-chart/tests/prometheus/podmonitor_test.yaml
PASS Test Spark RBAC charts/spark-operator-chart/tests/spark/rbac_test.yaml
PASS Test spark service account charts/spark-operator-chart/tests/spark/serviceaccount_test.yaml
PASS Test webhook deployment charts/spark-operator-chart/tests/webhook/deployment_test.yaml
PASS Test mutating webhook configuration charts/spark-operator-chart/tests/webhook/mutatingwebhookconfiguration_test.yaml
PASS Test webhook pod disruption budget charts/spark-operator-chart/tests/webhook/poddisruptionbudget_test.yaml
PASS Test webhook rbac charts/spark-operator-chart/tests/webhook/rbac_test.yaml
PASS Test webhook service charts/spark-operator-chart/tests/webhook/service_test.yaml
PASS Test validating webhook configuration charts/spark-operator-chart/tests/webhook/validatingwebhookconfiguration_test.yaml
Charts: 1 passed, 1 total
Test Suites: 14 passed, 14 total
Tests: 137 passed, 137 total
Snapshot: 0 passed, 0 total
Time: 477.748ms
构建 Helm 文档
Helm 图表 README.md 文件是由 helm-docs 工具生成的。如果您想更新 Helm 文档，请记得修改 README.md.gotmpl 而不是 README.md，然后运行 make helm-docs 来生成 README.md 文件：
$ make helm-docs
INFO[2024-04-14T07:29:26Z] Found Chart directories [charts/spark-operator-chart]
INFO[2024-04-14T07:29:26Z] Generating README Documentation for chart charts/spark-operator-chart
请注意，如果设置了 git 预提交钩子， helm-docs 将在提交任何更改之前自动运行。如果对 README.md 文件有任何更改，提交过程将被中止。
签署您的提交
在您对代码进行了更改后，请使用 -s 或 --signoff 标志签署您的提交，以便 DCO 检查 CI 能够通过：
git commit -s -m "Your commit message"
Last modified September 28, 2024: 更新 spark-operator 开发者指南 (#3882) (026d8ec)