Google Cloud Managed Airflow adds AI agents for debugging, upgrade checks and DAG writing; its MCP server is now generally available

Google Cloud says Managed Airflow now offers AI agent help across the pipeline lifecycle, including the Managed Airflow Agent in the console and the Data Agent Kit in the IDE; its MCP server is now generally available.

News date: About 6 min read

Contents
  1. 1. What do the new Managed Airflow agent features change?
  2. 2. How do you upgrade to Airflow 3 and event-driven scheduling?
  3. 3. Which tools are there and where do you use them?
  4. 4. How will this affect real workflows?

In its official blog, Google Cloud announced that Managed Service for Apache Airflow (formerly Cloud Composer) now offers agent assistance across the entire pipeline lifecycle. Airflow is an open-source tool for scheduling and running data-processing workflows, each described as a DAG (directed acyclic graph: the tasks and their order, defined in code). The only item the article explicitly labels as generally available is the Managed Airflow MCP server; the launch stage, plans and regions for the console agent and the Data Agent Kit are not stated. Google Cloud notes that when a pipeline fails, the hardest part is usually finding the problem, which often takes deep Airflow experience and frequently happens during on-call incident response; the final decisions remain with people.

What do the new Managed Airflow agent features change?

According to Google Cloud, the Managed Airflow Agent in the console investigates failed tasks, suggests environment improvements, and checks whether DAGs are ready to upgrade to a newer Airflow version. In an IDE (the code editor developers work in) or coding agent, the Data Agent Kit writes, migrates and deploys DAGs.

The blog uses a fictional company, Cymbal Shops, as a demonstration: it processed click events in hourly batches on Airflow 2.11.1, and as traffic grew, a single hour of data rose from about 350 MB to about 1.4 GB, exceeding the 4 GB memory of the worker (the compute node that actually runs tasks). The article says autoscaling does not solve this, because each task still runs on a single worker.

The agent's suggestions include loading files directly into BigQuery, processing them in batches, or adding worker memory; it also recommends raising worker memory from 4 GB to 8 GB and points out three gaps: database retention is not enabled, the environment is not highly resilient, and there are no scheduled snapshots. The article explains that the agent is read-only and the user makes the changes.

How do you upgrade to Airflow 3 and event-driven scheduling?

The article says Airflow 3 natively supports event-driven scheduling, meaning a run starts when new data arrives rather than once every hour: an asset watcher monitors an external source such as a Pub/Sub (Google Cloud's messaging service) subscription, updates an asset when a new message arrives, and DAGs scheduled on that asset start running. Before the upgrade, the agent checked compatibility against Airflow 3.3.1 as the target and found two blockers (a maintenance DAG that manually cleans the database, and a DAG using the removed execution_date), plus four DAGs that need schedule_interval changed to schedule.

Afterwards, the Data Agent Kit's migrations skill adjusts the DAGs, deploys them and verifies them. The example switches to PubsubPullTrigger with AssetWatcher, where the triggerer (the Airflow component that waits for events in the background) checks the subscription every 10 seconds, and dynamic task mapping gives each new file its own task, with a file limit of 128 MB.

Comparison of the fictional scenario in the Google Cloud blog (figures are illustrative)
AspectAirflow 2 hourly batch (example)Airflow 3 event-driven (example)
Unit of workOne clock hour: about 350 MB normally, about 1.4 GB at campaign peakA single file, a few MB each (up to 128 MB)
Peak memory per taskGrows with traffic; Black Friday projected well above 8 GBStable, well under 1 GB
Click to BigQueryMore than an hourA few minutes
Eight-times peak traffic load test—DAG runs more often, per-task memory unchanged

Which tools are there and where do you use them?

Compiled from the Google Cloud blog
ToolWhere usedWhat it doesLaunch status
Managed Airflow AgentGoogle Cloud consoleInvestigates failed tasks, environment suggestions, upgrade compatibility checksDescribed as something Managed Airflow "now offers"; plans and regions not stated
Data Agent KitIDE / coding agentWrites, migrates and deploys DAGsLaunch stage not stated; VS Code extension and plugins for several agents
Managed Airflow MCP serverAny MCP-compatible agentLists and triggers DAGs, checks status, views failed tasksGenerally available; controlled via IAM
Common AI providerInside Airflow 3 DAGsLLM branching, agentic retries, schema drift detectionOnly listed as an available capability in the article

How will this affect real workflows?

Google Cloud says the console agent inspects production telemetry with the user's permissions and doesn't change anything, while code changes are reviewed, tested and deployed in the IDE. The article says "Agents propose and people decide," and that static checks can assess migration effort in minutes.

If you want to try it, Google Cloud lists these Data Agent Kit installation routes: documentation, a VS Code extension, plugins supporting Antigravity, Claude Code, Gemini CLI and Codex CLI, and open-source Airflow agent skills; see the Google Cloud documentation for details.

Frequently asked questions

Will the Managed Airflow Agent change my environment automatically?

According to Google Cloud, the console agent is read-only: it only diagnoses and recommends. In the example, the engineer adjusts worker memory themselves and reviews each change.

What can the MCP server do?

MCP is a protocol that lets AI agents connect to external tools. Google Cloud says the Managed Airflow MCP server is now generally available, letting MCP-compatible agents list and trigger DAGs, check run status and inspect failed task instances. Access is controlled by IAM, and read-only tools are flagged.

Can the figures in the article be taken as real performance?

No. The blog notes that Cymbal Shops is a fictional company, and the agent output as well as the traffic and memory figures are shortened and illustrative.

What about pricing or regional availability?

Google Cloud's article does not mention pricing, plan tiers or regional information; please rely on the official Google Cloud documentation.

Browse the latest news in this topic

Found this guide helpful? Share it with someone who needs it

Sources

Compiled from the references and verification dates listed above; fares, schedules, product features, and terms can change, so please confirm the latest details on the relevant official website before travelling, booking, or purchasing.

Lifestyle