Google Cloud Managed Airflow adds AI agents for debugging, upgrade checks and DAG writing; its MCP server is now generally available
Google Cloud says Managed Airflow now offers AI agent help across the pipeline lifecycle, including the Managed Airflow Agent in the console and the Data Agent Kit in the IDE; its MCP server is now generally available.
News date: About 6 min read
Contents
In its official blog, Google Cloud announced that Managed Service for Apache Airflow (formerly Cloud Composer) now offers agent assistance across the entire pipeline lifecycle. Airflow is an open-source tool for scheduling and running data-processing workflows, each described as a DAG (directed acyclic graph: the tasks and their order, defined in code). The only item the article explicitly labels as generally available is the Managed Airflow MCP server; the launch stage, plans and regions for the console agent and the Data Agent Kit are not stated. Google Cloud notes that when a pipeline fails, the hardest part is usually finding the problem, which often takes deep Airflow experience and frequently happens during on-call incident response; the final decisions remain with people.
What do the new Managed Airflow agent features change?
According to Google Cloud, the Managed Airflow Agent in the console investigates failed tasks, suggests environment improvements, and checks whether DAGs are ready to upgrade to a newer Airflow version. In an IDE (the code editor developers work in) or coding agent, the Data Agent Kit writes, migrates and deploys DAGs.
The blog uses a fictional company, Cymbal Shops, as a demonstration: it processed click events in hourly batches on Airflow 2.11.1, and as traffic grew, a single hour of data rose from about 350 MB to about 1.4 GB, exceeding the 4 GB memory of the worker (the compute node that actually runs tasks). The article says autoscaling does not solve this, because each task still runs on a single worker.
The agent's suggestions include loading files directly into BigQuery, processing them in batches, or adding worker memory; it also recommends raising worker memory from 4 GB to 8 GB and points out three gaps: database retention is not enabled, the environment is not highly resilient, and there are no scheduled snapshots. The article explains that the agent is read-only and the user makes the changes.
How do you upgrade to Airflow 3 and event-driven scheduling?
The article says Airflow 3 natively supports event-driven scheduling, meaning a run starts when new data arrives rather than once every hour: an asset watcher monitors an external source such as a Pub/Sub (Google Cloud's messaging service) subscription, updates an asset when a new message arrives, and DAGs scheduled on that asset start running. Before the upgrade, the agent checked compatibility against Airflow 3.3.1 as the target and found two blockers (a maintenance DAG that manually cleans the database, and a DAG using the removed execution_date), plus four DAGs that need schedule_interval changed to schedule.
Afterwards, the Data Agent Kit's migrations skill adjusts the DAGs, deploys them and verifies them. The example switches to PubsubPullTrigger with AssetWatcher, where the triggerer (the Airflow component that waits for events in the background) checks the subscription every 10 seconds, and dynamic task mapping gives each new file its own task, with a file limit of 128 MB.
| Aspect | Airflow 2 hourly batch (example) | Airflow 3 event-driven (example) |
|---|---|---|
| Unit of work | One clock hour: about 350 MB normally, about 1.4 GB at campaign peak | A single file, a few MB each (up to 128 MB) |
| Peak memory per task | Grows with traffic; Black Friday projected well above 8 GB | Stable, well under 1 GB |
| Click to BigQuery | More than an hour | A few minutes |
| Eight-times peak traffic load test | — | DAG runs more often, per-task memory unchanged |
Which tools are there and where do you use them?
| Tool | Where used | What it does | Launch status |
|---|---|---|---|
| Managed Airflow Agent | Google Cloud console | Investigates failed tasks, environment suggestions, upgrade compatibility checks | Described as something Managed Airflow "now offers"; plans and regions not stated |
| Data Agent Kit | IDE / coding agent | Writes, migrates and deploys DAGs | Launch stage not stated; VS Code extension and plugins for several agents |
| Managed Airflow MCP server | Any MCP-compatible agent | Lists and triggers DAGs, checks status, views failed tasks | Generally available; controlled via IAM |
| Common AI provider | Inside Airflow 3 DAGs | LLM branching, agentic retries, schema drift detection | Only listed as an available capability in the article |
How will this affect real workflows?
Google Cloud says the console agent inspects production telemetry with the user's permissions and doesn't change anything, while code changes are reviewed, tested and deployed in the IDE. The article says "Agents propose and people decide," and that static checks can assess migration effort in minutes.
If you want to try it, Google Cloud lists these Data Agent Kit installation routes: documentation, a VS Code extension, plugins supporting Antigravity, Claude Code, Gemini CLI and Codex CLI, and open-source Airflow agent skills; see the Google Cloud documentation for details.
Frequently asked questions
Will the Managed Airflow Agent change my environment automatically?
According to Google Cloud, the console agent is read-only: it only diagnoses and recommends. In the example, the engineer adjusts worker memory themselves and reviews each change.
What can the MCP server do?
MCP is a protocol that lets AI agents connect to external tools. Google Cloud says the Managed Airflow MCP server is now generally available, letting MCP-compatible agents list and trigger DAGs, check run status and inspect failed task instances. Access is controlled by IAM, and read-only tools are flagged.
Can the figures in the article be taken as real performance?
No. The blog notes that Cymbal Shops is a fictional company, and the agent output as well as the traffic and memory figures are shortened and illustrative.
What about pricing or regional availability?
Google Cloud's article does not mention pricing, plan tiers or regional information; please rely on the official Google Cloud documentation.
Found this guide helpful? Share it with someone who needs it
Claude Code 2.1.330: Enterprise Admin Console Sign-In, easel Chart Module, and Scheduled Tasks Extended to 30 Days
Anthropic's Claude Code 2.1.330 changelog lists Enterprise Admin Console sign-in, a built-in easel chart module, scheduled tasks of up to 30 days, and stricter approval for rm commands.
AWS Quick adds chat-built flows, multi-agent answers and Meta ads/Amplitude connectors, available today in all Quick regions
AWS Distributed KV Caching Agent Skill for Self-Hosted LLMs: AWS Says Cost Drops Up to 40% and Time to First Token Up to 80%
Amazon SageMaker HyperPod Slurm Adds Cluster Extensions: Managed Observability and User Management, Available in All HyperPod Regions
Sources
Compiled from the references and verification dates listed above; fares, schedules, product features, and terms can change, so please confirm the latest details on the relevant official website before travelling, booking, or purchasing.