databricks-jobs

Develop and deploy Lakeflow Jobs on Databricks via DABs, Python SDK, or the CLI. Use when creating data engineering jobs with notebooks, Python wheels, SQL, dbt, or pipelines. Invoke BEFORE starting implementation. For compute (serverless, performance mode, classic), also use databricks-serverless.

1,006
installs: 0 up / 0 down, score 0.00
votes: 383
repo stars
Updated: less than a month ago
In the catalog since: 7 Nov 2026
From: databricks/databricks-agent-skills

Get it through your agent npx skills add databricks/databricks-agent-skills@databricks-jobs

Skill instructions

Rendered from the published SKILL.md, 345 lines.

Lakeflow Jobs Development

FIRST: Use the parent databricks-core skill for CLI basics, authentication, profile selection, and data exploration commands.

Compute: jobs run on serverless by default. Before you choose compute or write any cluster setting, use the databricks-serverless skill; it decides serverless or classic and the performance mode.

Lakeflow Jobs orchestrate data workflows with multi-task DAGs, flexible triggers, and comprehensive monitoring. Jobs support diverse task types and can be managed via Declarative Automation Bundles (DABs), Python SDK, or CLI.

Reference Files

Use Case	Reference File
Configure task types (notebook, Python, SQL, dbt, pipeline, JAR, run_job, for_each)	references/task-types.md
Set up triggers and schedules (cron, periodic, file arrival, table update, continuous)	references/triggers-schedules.md
Configure notifications, health rules, retries, timeouts, queues	references/notifications-monitoring.md
Complete worked examples (ETL, warehouse refresh, event-driven, ML training, multi-env, streaming, cross-job)	references/examples.md

Scaffolding a New Job Project

Use databricks bundle init with a config file to scaffold non-interactively. This creates a project in the <project_name>/ directory:

databricks bundle init default-python --config-file <(echo '{"project_name": "my_job", "include_job": "yes", "include_pipeline": "no", "include_python": "yes", "serverless": "yes"}') --profile <PROFILE> < /dev/null

project_name: letters, numbers, underscores only

bundle init also creates an AGENTS.md in the project. Customize that file for the project instead of creating new agent instruction files.

Project Structure

my-job-project/
├── databricks.yml              # Bundle configuration
├── resources/
│   └── my_job.job.yml          # Job definition
├── src/
│   ├── my_notebook.ipynb       # Notebook tasks
│   └── my_module/              # Python wheel package
│       ├── __init__.py
│       └── main.py
├── tests/
│   └── test_main.py
└── pyproject.toml              # Python project config (if using wheels)

Quick Start

Declarative Automation Bundles (DABs) — recommended

# resources/jobs.yml
resources:
  jobs:
    my_etl_job:
      name: "[${bundle.target}] My ETL Job"
      tasks:
        - task_key: extract
          notebook_task:
            notebook_path: ../src/notebooks/extract.py
      # no cluster block = serverless; compute details: databricks-serverless

Python SDK

from databricks.sdk import WorkspaceClient
from databricks.sdk.service.jobs import Task, NotebookTask, Source

w = WorkspaceClient()

job = w.jobs.create(
    name="my-etl-job",
    tasks=[
        Task(
            task_key="extract",
            notebook_task=NotebookTask(
                notebook_path="/Workspace/Shared/etl/extract",
                source=Source.WORKSPACE,
            ),
        ),
    ],
)
print(f"Created job: {job.job_id}")

CLI

databricks jobs create --json '{
  "name": "my-etl-job",
  "tasks": [{
    "task_key": "extract",
    "notebook_task": {
      "notebook_path": "/Workspace/Shared/etl/extract",
      "source": "WORKSPACE"
    }
  }]
}'

Core Concepts

Multi-Task Workflows

Jobs support DAG-based task dependencies:

tasks:
  - task_key: extract
    notebook_task:
      notebook_path: ../src/extract.py

  - task_key: transform
    depends_on:
      - task_key: extract
    notebook_task:
      notebook_path: ../src/transform.py

Show the whole skill (225 more lines)

task_key: load depends_on:
- task_key: transform run_if: ALL_SUCCESS # Only run if all dependencies succeed notebook_task: notebook_path: ../src/load.py


**run_if conditions:**

- `ALL_SUCCESS` (default) — run when all dependencies succeed
- `ALL_DONE` — run when all dependencies complete (success or failure)
- `AT_LEAST_ONE_SUCCESS` — run when at least one dependency succeeds
- `NONE_FAILED` — run when no dependencies failed
- `ALL_FAILED` — run when all dependencies failed
- `AT_LEAST_ONE_FAILED` — run when at least one dependency failed

### Task Types Summary

| Task Type | Use Case | Reference |
|-----------|----------|-----------|
| `notebook_task` | Run notebooks | [references/task-types.md#notebook-task](references/task-types.md#notebook-task) |
| `spark_python_task` | Run Python scripts | [references/task-types.md#spark-python-task](references/task-types.md#spark-python-task) |
| `python_wheel_task` | Run Python wheels | [references/task-types.md#python-wheel-task](references/task-types.md#python-wheel-task) |
| `sql_task` | Run SQL queries/files/dashboards/alerts | [references/task-types.md#sql-task](references/task-types.md#sql-task) |
| `dbt_task` | Run dbt projects | [references/task-types.md#dbt-task](references/task-types.md#dbt-task) |
| `pipeline_task` | Trigger SDP (formerly DLT) pipelines | [references/task-types.md#pipeline-task](references/task-types.md#pipeline-task) |
| `spark_jar_task` | Run Spark JARs | [references/task-types.md#spark-jar-task](references/task-types.md#spark-jar-task) |
| `run_job_task` | Trigger other jobs | [references/task-types.md#run-job-task](references/task-types.md#run-job-task) |
| `for_each_task` | Loop over inputs | [references/task-types.md#for-each-task](references/task-types.md#for-each-task) |

### Trigger Types Summary

| Trigger Type | Use Case | Reference |
|--------------|----------|-----------|
| `schedule` | Cron-based scheduling | [references/triggers-schedules.md#cron-schedule](references/triggers-schedules.md#cron-schedule) |
| `trigger.periodic` | Interval-based | [references/triggers-schedules.md#periodic-trigger](references/triggers-schedules.md#periodic-trigger) |
| `trigger.file_arrival` | File arrival events | [references/triggers-schedules.md#file-arrival-trigger](references/triggers-schedules.md#file-arrival-trigger) |
| `trigger.table_update` | Unity Catalog table change events | [references/triggers-schedules.md#table-update-trigger](references/triggers-schedules.md#table-update-trigger) |
| `continuous` | Always-running jobs | [references/triggers-schedules.md#continuous-jobs](references/triggers-schedules.md#continuous-jobs) |

## Compute Configuration

Serverless is the default. Leave out cluster settings (`job_clusters`, `new_cluster`, `job_cluster_key`,
`existing_cluster_id`); a notebook, Python script, wheel or JAR task without them runs on serverless.
SQL tasks, and the SQL of dbt tasks, run on a SQL warehouse (`warehouse_id`, see
[Task Types](references/task-types.md)); a pipeline task runs on the pipeline's own compute
(`serverless: true` in the pipeline), and the job's `performance_target` sets its mode.
If creating or running the job fails because serverless is not available in the workspace, put the
whole job on classic and say so (job clusters for the tasks that run on job compute).

For everything else about compute, **use the `databricks-serverless` skill**: the performance mode
(`performance_target`), libraries (`environments`), existing code that may not run on serverless, and
the few workloads that need a classic cluster. If that skill is not installed, run
`databricks aitools install`. Either way, set `performance_target` on every job: `STANDARD` to optimize
cost, `PERFORMANCE_OPTIMIZED` when the job is time-sensitive, interactive, or other workloads depend on it.

## Job Parameters

Parameters defined at job level are passed to ALL tasks (no need to repeat per task):

```yaml
parameters:
  - name: env
    default: "dev"
  - name: date
    default: "{{start_date}}"  # Dynamic value reference

Access in notebooks:

catalog = dbutils.widgets.get("env")
load_date = dbutils.widgets.get("date")

Pass to specific tasks:

tasks:
  - task_key: my_task
    notebook_task:
      notebook_path: ../src/notebook.py
      base_parameters:
        env: "{{job.parameters.env}}"
        custom_param: "value"

Common Operations

Python SDK

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()

# List jobs
jobs = w.jobs.list()

# Get job details
job = w.jobs.get(job_id=12345)

# Run job now
run = w.jobs.run_now(job_id=12345)

# Run with parameters
run = w.jobs.run_now(
    job_id=12345,
    job_parameters={"env": "prod", "date": "2024-01-15"},
)

# Cancel run
w.jobs.cancel_run(run_id=run.run_id)

# Delete job
w.jobs.delete(job_id=12345)

CLI

# List jobs
databricks jobs list

# Get job details
databricks jobs get 12345

# Run job
databricks jobs run-now 12345

# Run with parameters (must use --json with job_id inside)
databricks jobs run-now --json '{"job_id": 12345, "job_parameters": {"env": "prod"}}'

# Cancel run
databricks jobs cancel-run 67890

# Delete job
databricks jobs delete 12345

Declarative Automation Bundle Operations

# Validate configuration
databricks bundle validate --profile <profile>

# Deploy to a target
databricks bundle deploy -t dev --profile <profile>

# Run a job
databricks bundle run <job_name> -t dev --profile <profile>

# Check run status
databricks jobs get-run --run-id <id> --profile <profile>

# Destroy resources
databricks bundle destroy --auto-approve

Permissions (DABs)

resources:
  jobs:
    my_job:
      name: "My Job"
      permissions:
        - level: CAN_VIEW
          group_name: "data-analysts"
        - level: CAN_MANAGE_RUN
          group_name: "data-engineers"
        - level: CAN_MANAGE
          user_name: "admin@example.com"

Permission levels:

CAN_VIEW — view job and run history
CAN_MANAGE_RUN — view, trigger, and cancel runs
CAN_MANAGE — full control including edit and delete

Unit Testing

Run unit tests locally:

uv run pytest

Development Workflow

Validate: databricks bundle validate --profile <profile>
Deploy: databricks bundle deploy -t dev --profile <profile>
Run: databricks bundle run <job_name> -t dev --profile <profile>
Check run status: databricks jobs get-run --run-id <id> --profile <profile>

Common Issues

Issue	Solution
Serverless job startup slow	`performance_target: PERFORMANCE_OPTIMIZED`; see databricks-serverless
Task dependencies not working	Verify `task_key` references match exactly in `depends_on`
Schedule not triggering	Check `pause_status: UNPAUSED` and valid timezone
File arrival not detecting	Ensure path has proper permissions and uses cloud storage URL
Table update trigger missing events	Verify Unity Catalog table and proper grants
Parameter not accessible	Use `dbutils.widgets.get()` in notebooks
`admins` group error	Cannot modify admins permissions on jobs
Serverless task fails	See the Common Issues of databricks-serverless (libraries, RDD code, DBFS mounts)

Related Skills

databricks-dabs — DABs configuration patterns shared by jobs and pipelines
databricks-pipelines — SDP (formerly DLT) pipelines triggered by pipeline_task
databricks-serverless — compute decisions: serverless, performance mode, classic fallback

Documentation

Get it through your agent

Call the Graph-of-Skills MCP server's retrieve_skill_bundle tool and it returns this skill's text inside your budget.

{
  "tool": "retrieve_skill_bundle",
  "arguments": {
    "query": "databricks-jobs"
  }
}

New to this? Set up Graph-of-Skills or get a free key. The text on this page comes from its authors: how skills are sourced and what we check. If none fits your task, your agent can write a skill for your own library. To keep a skill from your own library on a machine, see how to install one of your skills.

Share on X Share on LinkedIn Share on Bluesky Share on Hacker News

Share on Reddit

Source

Source repository databricks/databricks-agent-skills
Listing on skills.sh

From databricks/databricks-agent-skills via skills.sh

Related skills

databricks-pipelines
Develop Lakeflow Spark Declarative Pipelines (formerly Delta Live Tables) on Databricks. Use when building batch or streaming data pipelines with Python or SQL, including Auto CDC from event streams or periodic complete snapshots. Invoke BEFORE starting implementation.
1,044 installs
databricks-core
Databricks CLI operations and the parent/entry-point skill for Databricks CLI use: authentication, profile selection, and bundles. Load this first for CLI, auth, profile, and bundle tasks, then load the matching product skill. For finding or exploring data, answering questions about the data, or generating SQL, load the databricks-data-discovery skill (it routes to Genie One). Contains up-to-date guidelines for Databricks-related CLI tasks.
2,062 installs
data-engineering-patterns-fabric-databricks
600+ patterns and concepts for Azure Databricks, Microsoft Fabric, and PySpark data engineering - covering lakehouse architecture, Delta Lake, pipelines, and production best practices.
1,578 installs
spark-engineer
Use when writing Spark jobs, debugging performance issues, or configuring cluster settings for Apache Spark applications, distributed data processing pipelines, or big data workloads. Invoke to write DataFrame transformations, optimize Spark SQL queries, implement RDD pipelines, tune shuffle operations, configure executor memory, process .parquet files, handle data partitioning, or build structured streaming analytics.
4,088 installs
devops
Deploy to Cloudflare (Workers, R2, D1), Docker, GCP (Cloud Run, GKE), Kubernetes (kubectl, Helm). Use for serverless, containers, CI/CD, GitOps, security audit.
979 installs

More from databricks/databricks-agent-skills

databricks-core
Databricks CLI operations and the parent/entry-point skill for Databricks CLI use: authentication, profile selection, and bundles. Load this first for CLI, auth, profile, and bundle tasks, then load the matching product skill. For finding or exploring data, answering questions about the data, or generating SQL, load the databricks-data-discovery skill (it routes to Genie One). Contains up-to-date guidelines for Databricks-related CLI tasks.
2,062 installs
databricks-apps
Build apps on Databricks Apps platform. Use when asked to create data apps, analytics tools, or custom interactive visualizations. A plain "create a dashboard" request means a managed AI/BI (Lakeview) dashboard → use databricks-aibi-dashboards, not this skill. Evaluates data access patterns (analytics vs Lakebase synced tables) before scaffolding. Invoke BEFORE starting implementation.
1,136 installs
databricks-pipelines
Develop Lakeflow Spark Declarative Pipelines (formerly Delta Live Tables) on Databricks. Use when building batch or streaming data pipelines with Python or SQL, including Auto CDC from event streams or periodic complete snapshots. Invoke BEFORE starting implementation.
1,044 installs

Content is provided as published by its creators; you are responsible for how you use it.

To request removal, use the contact form.