{"id":6329,"date":"2025-04-11T16:49:43","date_gmt":"2025-04-11T07:49:43","guid":{"rendered":"https:\/\/blog.since2020.jp\/?p=6329"},"modified":"2026-07-30T17:56:34","modified_gmt":"2026-07-30T08:56:34","slug":"airflow_glossary","status":"publish","type":"post","link":"https:\/\/since2020.jp\/media\/airflow_glossary\/","title":{"rendered":"Airflow\u3068\u306f\uff1f\u30102026\u5e74\u6700\u65b0\u7248\u3011\u6700\u65b0\u6a5f\u80fd\u30fb\u6d3b\u7528\u4e8b\u4f8b\u30fb\u4ed6\u30c4\u30fc\u30eb\u3068\u306e\u9055\u3044\u307e\u3067\u5fb9\u5e95\u89e3\u8aac"},"content":{"rendered":"\n<p>Apache Airflow\u3068\u306f\u4f55\u304b\u30922026\u5e74\u6700\u65b0\u60c5\u5831\u3092\u3082\u3068\u306b\u5fb9\u5e95\u89e3\u8aac\u3002Airflow 3.x\u306e\u65b0\u6a5f\u80fd\u3001DAG\u306e\u4ed5\u7d44\u307f\u3001AI\u30fb\u30c7\u30fc\u30bf\u57fa\u76e4\u3067\u306e\u6d3b\u7528\u4e8b\u4f8b\u3001Databricks\u30fbSnowflake\u30fbdbt\u3068\u306e\u9023\u643a\u3001\u30e1\u30ea\u30c3\u30c8\u30fb\u30c7\u30e1\u30ea\u30c3\u30c8\u3001\u4eca\u5f8c\u306e\u5c55\u671b\u307e\u3067\u308f\u304b\u308a\u3084\u3059\u304f\u7d39\u4ecb\u3057\u307e\u3059\u3002<\/p>\n\n\n<h2>\u306f\u3058\u3081\u306b<\/h2>\n<p><b>Airflow\u3068\u306f\uff1f<\/b><\/p>\n<p>Apache Airflow\u306f\u3001\u30c7\u30fc\u30bf\u30d1\u30a4\u30d7\u30e9\u30a4\u30f3\u3084AI\u30ef\u30fc\u30af\u30d5\u30ed\u30fc\u3092\u30b3\u30fc\u30c9\u3067\u7ba1\u7406\u30fb\u81ea\u52d5\u5316\u3067\u304d\u308b\u3001\u4e16\u754c\u3067\u3082\u3063\u3068\u3082\u5229\u7528\u3055\u308c\u3066\u3044\u308b\u30aa\u30fc\u30d7\u30f3\u30bd\u30fc\u30b9\u306e\u30ef\u30fc\u30af\u30d5\u30ed\u30fc\u30aa\u30fc\u30b1\u30b9\u30c8\u30ec\u30fc\u30b7\u30e7\u30f3\u30c4\u30fc\u30eb\u3067\u3059\u3002<\/p>\n<p>Python\u3067\u30ef\u30fc\u30af\u30d5\u30ed\u30fc\u3092\u8a18\u8ff0\u3067\u304d\u308b\u305f\u3081\u3001ETL\u3060\u3051\u3067\u306a\u304f\u3001<\/p>\n<p><strong>\u30fb<\/strong>\u30c7\u30fc\u30bf\u5206\u6790<br \/><strong>\u30fb<\/strong>\u6a5f\u68b0\u5b66\u7fd2<br \/><strong>\u30fb<\/strong>Generative AI<br \/><strong>\u30fb<\/strong>\u30c7\u30fc\u30bf\u54c1\u8cea\u30c1\u30a7\u30c3\u30af<br \/><strong>\u30fb<\/strong>dbt\u5b9f\u884c<br \/><strong>\u30fb<\/strong>\u30af\u30e9\u30a6\u30c9\u30b5\u30fc\u30d3\u30b9\u9023\u643a<\/p>\n<p>\u306a\u3069\u5e45\u5e83\u3044\u7528\u9014\u3067\u5229\u7528\u3055\u308c\u3066\u3044\u307e\u3059\u3002<\/p>\n<p>2026\u5e74\u73fe\u5728\u3067\u306f\u3001\u591a\u304f\u306e\u4f01\u696d\u304c<\/p>\n<p><strong>\u30fb<\/strong>Databricks<br \/><strong>\u30fb<\/strong>Snowflake<br \/><strong>\u30fb<\/strong>BigQuery<br \/><strong>\u30fb<\/strong>Spark<br \/><strong>\u30fb<\/strong>Kubernetes<br \/><strong>\u30fb<\/strong>AWS<br \/><strong>\u30fb<\/strong>Azure<br \/><strong>\u30fb<\/strong>Google Cloud<\/p>\n<p>\u306a\u3069\u3092Airflow\u3067\u7d71\u5408\u7ba1\u7406\u3057\u3066\u304a\u308a\u3001\u300c\u30c7\u30fc\u30bf\u30d7\u30e9\u30c3\u30c8\u30d5\u30a9\u30fc\u30e0\u5168\u4f53\u3092\u5236\u5fa1\u3059\u308b\u53f8\u4ee4\u5854\u300d\u3068\u3057\u3066\u6d3b\u7528\u3055\u308c\u3066\u3044\u307e\u3059\u3002<!-- notionvc: e3b1d599-28e0-4a57-b5c4-53dc506c3457 --><\/p>\n<p><!-- notionvc: 38e1c650-cfbd-4565-b072-7ddeb43e2971 --><\/p>\n<h2>Airflow\u306e\u6700\u65b0\u60c5\u5831\uff082026\u5e74\u7248\uff09<!-- notionvc: 26c33f45-ce8a-441a-b285-cd3b7c1ccf61 --><\/h2>\n<p>2025\u301c2026\u5e74\u306b\u304b\u3051\u3066Airflow\u306f\u5927\u304d\u304f\u9032\u5316\u3057\u307e\u3057\u305f\u3002<\/p>\n<p>\u7279\u306b\u6ce8\u76ee\u3055\u308c\u3066\u3044\u308b\u306e\u304cAirflow 3.x\u7cfb\u3067\u3059\u3002<\/p>\n<p>\u4e3b\u306a\u30a2\u30c3\u30d7\u30c7\u30fc\u30c8\u306f\u6b21\u306e\u3068\u304a\u308a\u3067\u3059\u3002<\/p>\n<p>\u30fbAirflow 3.x\u30a2\u30fc\u30ad\u30c6\u30af\u30c1\u30e3\u3078\u306e\u5237\u65b0<br \/>\u30fbPython\u306e\u6700\u65b0\u30d0\u30fc\u30b8\u30e7\u30f3\u3078\u306e\u5bfe\u5fdc<br \/>\u30fbUI\u306e\u5927\u5e45\u6539\u5584<br \/>\u30fbDAG\u5b9f\u884c\u901f\u5ea6\u306e\u5411\u4e0a<br \/>\u30fb\u30c7\u30fc\u30bf\u30a2\u30bb\u30c3\u30c8\u7ba1\u7406\uff08Datasets\uff09\u306e\u5f37\u5316<br \/>\u30fbEvent-driven Scheduling\u306e\u62e1\u5145<br \/>\u30fbKubernetes\u3068\u306e\u7d71\u5408\u5f37\u5316<br \/>\u30fbREST API\u306e\u6a5f\u80fd\u62e1\u5f35<br \/>\u30fbOpenLineage\u3068\u306e\u9023\u643a\u5f37\u5316<br \/>\u30fbAI\u30ef\u30fc\u30af\u30d5\u30ed\u30fc\u3078\u306e\u6700\u9069\u5316<\/p>\n<p>\u4ee5\u524d\u306f\u300ccron\u30b8\u30e7\u30d6\u7ba1\u7406\u30c4\u30fc\u30eb\u300d\u3068\u3044\u3046\u5370\u8c61\u3082\u3042\u308a\u307e\u3057\u305f\u304c\u3001\u73fe\u5728\u3067\u306f\u30a4\u30d9\u30f3\u30c8\u99c6\u52d5\u578b\u30c7\u30fc\u30bf\u57fa\u76e4\u306e\u4e2d\u6838\u3068\u3057\u3066\u5229\u7528\u3055\u308c\u3066\u3044\u307e\u3059\u3002<!-- notionvc: 3187aef3-4130-4879-b5c0-303d80644b7f --><\/p>\n<p><!-- notionvc: 1a1a6bc9-cee6-4579-8dcc-f826ebe9e9be --><\/p>\n<p><!-- notionvc: 459cfed3-7874-44f9-b1cd-a5abf4a66e1b --><\/p>\n<h2>Airflow\u306e\u57fa\u672c\u6982\u5ff5<!-- notionvc: e2fb5c31-164e-4d68-ab10-78342a6feb38 --><\/h2>\n<p><strong>Airflow\u3068\u306f\uff1f<\/strong><\/p>\n<p>Airflow\u3067\u306f\u3001\u51e6\u7406\u306e\u6d41\u308c\u3092DAG\uff08Directed Acyclic Graph\uff09\u3068\u3057\u3066\u5b9a\u7fa9\u3057\u307e\u3059\u3002<\/p>\n<p>DAG\u3068\u306f<\/p>\n<p>\u300c\u3069\u306e\u51e6\u7406\u3092\u3001\u3069\u306e\u9806\u756a\u3067\u5b9f\u884c\u3059\u308b\u304b\u300d<\/p>\n<p>\u3092\u8868\u73fe\u3057\u305f\u30ef\u30fc\u30af\u30d5\u30ed\u30fc\u3067\u3059\u3002<\/p>\n<p>\u4f8b\u3048\u3070\u3001<\/p>\n<p>CSV\u53d6\u5f97<br \/>\u2193<br \/>\u30c7\u30fc\u30bf\u52a0\u5de5<br \/>\u2193<br \/>Snowflake\u3078\u30ed\u30fc\u30c9<br \/>\u2193<br \/>dbt\u5b9f\u884c<br \/>\u2193<br \/>\u54c1\u8cea\u30c1\u30a7\u30c3\u30af<br \/>\u2193<br \/>Slack\u901a\u77e5<\/p>\n<p>\u3053\u306e\u3088\u3046\u306a\u4e00\u9023\u306e\u6d41\u308c\u3092Python\u30b3\u30fc\u30c9\u3060\u3051\u3067\u7ba1\u7406\u3067\u304d\u307e\u3059\u3002<\/p>\n<p><!-- notionvc: 05dd3fe8-eece-4422-880f-85b0a16546df --><\/p>\n<h2>Airflow\u304c\u91cd\u8981\u8996\u3055\u308c\u308b\u7406\u7531<!-- notionvc: 33c89b31-54a4-4873-942e-8a20cfb6ad83 --><\/h2>\n<p>\u8fd1\u5e74\u3001\u4f01\u696d\u3067\u306f\u6271\u3046\u30c7\u30fc\u30bf\u91cf\u304c\u6025\u5897\u3057\u3066\u3044\u307e\u3059\u3002<\/p>\n<p>\u3055\u3089\u306b\u3001<\/p>\n<p>\u30fbSaaS<br \/>\u30fb\u30c7\u30fc\u30bf\u30ec\u30a4\u30af<br \/>\u30fbAI<br \/>\u30fbBI<br \/>\u30fbFeature Store<\/p>\n<p>\u306a\u3069\u30b7\u30b9\u30c6\u30e0\u304c\u8907\u96d1\u5316\u3057\u3066\u3044\u307e\u3059\u3002<\/p>\n<p>Airflow\u306f\u3001\u305d\u308c\u3089\u3092\u4e00\u5143\u7ba1\u7406\u3067\u304d\u308b\u70b9\u304c\u9ad8\u304f\u8a55\u4fa1\u3055\u308c\u3066\u3044\u307e\u3059\u3002<\/p>\n<p>\u4e3b\u306a\u7279\u5fb4\u306f\u4ee5\u4e0b\u306e\u3068\u304a\u308a\u3067\u3059\u3002<\/p>\n<p>\u30fb\u30ef\u30fc\u30af\u30d5\u30ed\u30fc\u5168\u4f53\u3092\u30b3\u30fc\u30c9\u3067\u7ba1\u7406\u3067\u304d\u308b<br \/>\u30fb\u4f9d\u5b58\u95a2\u4fc2\u3092\u53ef\u8996\u5316\u3067\u304d\u308b<br \/>\u30fb\u30a8\u30e9\u30fc\u6642\u306e\u30ea\u30c8\u30e9\u30a4\u304c\u5bb9\u6613<br \/>\u30fb\u30b9\u30b1\u30b8\u30e5\u30fc\u30eb\u5b9f\u884c\u30fb\u30a4\u30d9\u30f3\u30c8\u5b9f\u884c\u306e\u4e21\u65b9\u306b\u5bfe\u5fdc<br \/>\u30fb\u6570\u767e\u301c\u6570\u5343\u306e\u30bf\u30b9\u30af\u3092\u5b89\u5b9a\u904b\u7528\u3067\u304d\u308b<\/p>\n<p><!-- notionvc: 46ad461d-7510-460d-81c9-ba0ab3d3f759 --><\/p>\n<h2>Airflow\u306e\u30d7\u30ed\u30bb\u30b9\u3068\u4ed5\u7d44\u307f<!-- notionvc: 9b0d5813-57a4-439e-aa1a-fd3da59c0133 --><br \/><!-- notionvc: e2fb5c31-164e-4d68-ab10-78342a6feb38 --><\/h2>\n<p>Airflow\u3067\u306f\u4e00\u822c\u7684\u306b\u6b21\u306e\u6d41\u308c\u3067\u51e6\u7406\u3092\u69cb\u7bc9\u3057\u307e\u3059\u3002<\/p>\n<p>1.DAG\u3092\u4f5c\u6210\u3059\u308b<br \/>2.Task\u3092\u5b9a\u7fa9\u3059\u308b<br \/>3.Task\u9593\u306e\u4f9d\u5b58\u95a2\u4fc2\u3092\u8a2d\u5b9a\u3059\u308b<br \/>4.Scheduler\u304c\u5b9f\u884c\u30bf\u30a4\u30df\u30f3\u30b0\u3092\u7ba1\u7406\u3059\u308b<br \/>5.Executor\u304c\u5404Task\u3092\u5b9f\u884c\u3059\u308b<br \/>6.\u30ed\u30b0\u30fb\u30e1\u30c8\u30ea\u30af\u30b9\u30fb\u30a2\u30e9\u30fc\u30c8\u3092\u76e3\u8996\u3059\u308b<\/p>\n<p>\u73fe\u5728\u3067\u306fExecutor\u3082\u7528\u9014\u306b\u5fdc\u3058\u3066\u9078\u629e\u3067\u304d\u307e\u3059\u3002<\/p>\n<p>\u30fbLocal Executor<br \/>\u30fbCelery Executor<br \/>\u30fbKubernetes Executor<br \/>\u30fbCelery + Kubernetes Hybrid<\/p>\n<p>\u7279\u306bKubernetes Executor\u306f\u30af\u30e9\u30a6\u30c9\u30cd\u30a4\u30c6\u30a3\u30d6\u74b0\u5883\u3068\u306e\u76f8\u6027\u304c\u826f\u304f\u3001\u591a\u304f\u306e\u4f01\u696d\u3067\u63a1\u7528\u3055\u308c\u3066\u3044\u307e\u3059\u3002<!-- notionvc: f868f180-11f6-4787-ade2-fdb9e087e4ef --><\/p>\n<h2>Airflow 3.x\u3067\u6ce8\u76ee\u3055\u308c\u308b\u65b0\u6a5f\u80fd<!-- notionvc: 99b5bb2a-df6a-4715-a2db-8fd459ad62d4 --><br \/><!-- notionvc: e2fb5c31-164e-4d68-ab10-78342a6feb38 --><\/h2>\n<p><strong>Dataset\uff08\u30c7\u30fc\u30bf\u30a2\u30bb\u30c3\u30c8\uff09\u30d9\u30fc\u30b9\u306e\u5b9f\u884c<\/strong><!-- notionvc: c94d29dd-9d38-4b64-824f-cafe0e062991 --><\/p>\n<p>\u5f93\u6765\u306f<\/p>\n<p>\u300c\u6bce\u65e59\u6642\u300d<\/p>\n<p>\u306e\u3088\u3046\u306a\u6642\u9593\u30d9\u30fc\u30b9\u306e\u5b9f\u884c\u304c\u4e2d\u5fc3\u3067\u3057\u305f\u3002<\/p>\n<p>\u73fe\u5728\u3067\u306f<\/p>\n<p>\u300c\u30c6\u30fc\u30d6\u30eb\u66f4\u65b0\u5f8c\u300d<\/p>\n<p>\u300c\u30d5\u30a1\u30a4\u30eb\u30a2\u30c3\u30d7\u30ed\u30fc\u30c9\u5f8c\u300d<\/p>\n<p>\u306a\u3069\u3001\u30c7\u30fc\u30bf\u66f4\u65b0\u3092\u30c8\u30ea\u30ac\u30fc\u306bDAG\u3092\u958b\u59cb\u3067\u304d\u307e\u3059\u3002<\/p>\n<p>\u3053\u308c\u306b\u3088\u308a\u4e0d\u8981\u306a\u30d0\u30c3\u30c1\u51e6\u7406\u304c\u6e1b\u308a\u3001\u3088\u308a\u52b9\u7387\u7684\u306a\u30c7\u30fc\u30bf\u30d1\u30a4\u30d7\u30e9\u30a4\u30f3\u3092\u69cb\u7bc9\u3067\u304d\u307e\u3059\u3002<!-- notionvc: 2d1c063c-453e-4d00-9e4e-5a5354ab7efa --><\/p>\n<p><strong>Event-driven Scheduling<\/strong><\/p>\n<p>Airflow\u306f\u30a4\u30d9\u30f3\u30c8\u99c6\u52d5\u578b\u306e\u30aa\u30fc\u30b1\u30b9\u30c8\u30ec\u30fc\u30b7\u30e7\u30f3\u3092\u5f37\u5316\u3057\u3066\u3044\u307e\u3059\u3002<\/p>\n<p>\u4f8b\u3048\u3070\u3001<\/p>\n<p>\u30fbS3\u3078\u306e\u30d5\u30a1\u30a4\u30eb\u914d\u7f6e<br \/>\u30fbKafka\u30a4\u30d9\u30f3\u30c8<br \/>\u30fbAPI\u901a\u77e5<br \/>\u30fbdbt\u5b8c\u4e86<br \/>\u30fbDatabricks Job\u7d42\u4e86<\/p>\n<p>\u306a\u3069\u3092\u30c8\u30ea\u30ac\u30fc\u306b\u30ef\u30fc\u30af\u30d5\u30ed\u30fc\u3092\u958b\u59cb\u3067\u304d\u307e\u3059\u3002<\/p>\n<p><strong>UI\u306e\u6539\u5584<\/strong><\/p>\n<p>\u8fd1\u5e74\u306eUI\u306f\u5927\u5e45\u306b\u6539\u5584\u3055\u308c\u3001<\/p>\n<p>\u30fbDAG\u4f9d\u5b58\u95a2\u4fc2<br \/>\u30fb\u5b9f\u884c\u72b6\u6cc1<br \/>\u30fb\u30bf\u30b9\u30af\u30ed\u30b0<br \/>\u30fb\u518d\u5b9f\u884c<br \/>\u30fb\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u4f9d\u5b58\u95a2\u4fc2<\/p>\n<p>\u306a\u3069\u304c\u4ee5\u524d\u3088\u308a\u3082\u898b\u3084\u3059\u304f\u306a\u3063\u3066\u3044\u307e\u3059\u3002<\/p>\n<p>\u904b\u7528\u62c5\u5f53\u8005\u306e\u8ca0\u8377\u8efd\u6e1b\u306b\u3082\u3064\u306a\u304c\u3063\u3066\u3044\u307e\u3059\u3002<\/p>\n<p><!-- notionvc: 9d6bfefd-7f07-4e1a-b802-9ed465727a43 --><\/p>\n<p><!-- notionvc: 0ff03416-bcce-47c0-970a-2ef2da5edbca --><\/p>\n<h2>Airflow\u306e\u6d3b\u7528\u5206\u91ce<!-- notionvc: 9e9c345c-63d1-4f5c-b8b2-35ec07db7d38 --><br \/><!-- notionvc: e2fb5c31-164e-4d68-ab10-78342a6feb38 --><\/h2>\n<p><strong>ETL\u30fbELT\u30d1\u30a4\u30d7\u30e9\u30a4\u30f3<\/strong><!-- notionvc: 84d8ff4e-132b-4890-a074-1199e2f573db --><\/p>\n<p>\u3082\u3063\u3068\u3082\u4ee3\u8868\u7684\u306a\u7528\u9014\u3067\u3059\u3002<\/p>\n<p>\u4f8b\u3048\u3070\u3001<\/p>\n<p>\u30fbAPI\u53d6\u5f97<br \/>\u30fb\u30c7\u30fc\u30bf\u52a0\u5de5<br \/>\u30fbSnowflake\u30ed\u30fc\u30c9<br \/>\u30fbBigQuery\u30ed\u30fc\u30c9<br \/>\u30fb\u30c7\u30fc\u30bf\u54c1\u8cea\u30c1\u30a7\u30c3\u30af<\/p>\n<p>\u306a\u3069\u3092\u81ea\u52d5\u5316\u3067\u304d\u307e\u3059\u3002<!-- notionvc: 3ed3c852-385d-4bda-8cd3-917da2501ba2 --><\/p>\n<p><strong>AI\u30fbMLOps<\/strong><\/p>\n<p>\u8fd1\u5e74\u3082\u3063\u3068\u3082\u5229\u7528\u304c\u5897\u3048\u3066\u3044\u308b\u5206\u91ce\u3067\u3059\u3002<\/p>\n<p>Airflow\u3067\u306f<\/p>\n<p>\u30fb\u30c7\u30fc\u30bf\u53ce\u96c6<br \/>\u30fb\u7279\u5fb4\u91cf\u751f\u6210<br \/>\u30fb\u30e2\u30c7\u30eb\u5b66\u7fd2<br \/>\u30fb\u8a55\u4fa1<br \/>\u30fb\u30e2\u30c7\u30eb\u767b\u9332<br \/>\u30fb\u63a8\u8ad6\u30d0\u30c3\u30c1<\/p>\n<p>\u307e\u3067\u4e00\u9023\u306eML\u30d1\u30a4\u30d7\u30e9\u30a4\u30f3\u3092\u69cb\u7bc9\u3067\u304d\u307e\u3059\u3002<\/p>\n<p>Vertex AI\u3084Databricks\u3001MLflow\u306a\u3069\u3068\u3082\u9023\u643a\u53ef\u80fd\u3067\u3059\u3002<\/p>\n<p><strong>Generative AI\u30fbLLMOps<\/strong><\/p>\n<p>2026\u5e74\u3067\u306f\u751f\u6210AI\u57fa\u76e4\u3067\u3082Airflow\u304c\u6d3b\u8e8d\u3057\u3066\u3044\u307e\u3059\u3002<\/p>\n<p>\u4f8b\u3048\u3070\u3001<\/p>\n<p>\u30fbRAG\u7528\u30c7\u30fc\u30bf\u66f4\u65b0<br \/>\u30fb\u30d9\u30af\u30c8\u30ebDB\u66f4\u65b0<br \/>\u30fbEmbedding\u751f\u6210<br \/>\u30fb\u30c9\u30ad\u30e5\u30e1\u30f3\u30c8\u540c\u671f<br \/>\u30fbLLM\u8a55\u4fa1<br \/>\u30fbAgent\u30ef\u30fc\u30af\u30d5\u30ed\u30fc\u5b9f\u884c<\/p>\n<p>\u306a\u3069\u3092\u81ea\u52d5\u5316\u3067\u304d\u307e\u3059\u3002<\/p>\n<p>LangGraph\u3084AI\u30a8\u30fc\u30b8\u30a7\u30f3\u30c8\u57fa\u76e4\u3068\u7d44\u307f\u5408\u308f\u305b\u3066\u904b\u7528\u3059\u308b\u4f01\u696d\u3082\u5897\u3048\u3066\u3044\u307e\u3059\u3002<\/p>\n<p><!-- notionvc: a6f6a15d-df85-4171-9865-f29e30440a1a --><\/p>\n<p><strong>\u30af\u30e9\u30a6\u30c9\u30c7\u30fc\u30bf\u57fa\u76e4<\/strong><\/p>\n<p>Airflow\u306f\u4e3b\u8981\u30af\u30e9\u30a6\u30c9\u30b5\u30fc\u30d3\u30b9\u3068\u306e\u89aa\u548c\u6027\u3082\u975e\u5e38\u306b\u9ad8\u304f\u3001<\/p>\n<p>\u30fbAmazon MWAA<br \/>\u30fbGoogle Cloud Composer<br \/>\u30fbAzure Kubernetes Service<br \/>\u30fbDatabricks Workflows<br \/>\u30fbSnowflake<br \/>\u30fbBigQuery<br \/>\u30fbRedshift<\/p>\n<p>\u306a\u3069\u3068\u7d44\u307f\u5408\u308f\u305b\u3066\u5229\u7528\u3055\u308c\u3066\u3044\u307e\u3059\u3002<\/p>\n<p><!-- notionvc: 30430481-9e34-4b2f-84c9-366293afccb1 --><\/p>\n<p><!-- notionvc: 67937810-ee44-401d-b414-b04f7985dfff --><\/p>\n<h2>Airflow\u306e\u30e1\u30ea\u30c3\u30c8\u3068\u30c7\u30e1\u30ea\u30c3\u30c8<\/h2>\n<p><b>Airflow\u306e\u30e1\u30ea\u30c3\u30c8<\/b><\/p>\n<p><strong>\u30fb<span data-token-index=\"0\" class=\"notion-enable-hover\">Python\u3060\u3051\u3067\u30ef\u30fc\u30af\u30d5\u30ed\u30fc\u3092\u69cb\u7bc9\u3067\u304d\u308b<\/span><!-- notionvc: dfff4bdc-a2df-4ebe-beb2-17316fdaac95 --><br \/><\/strong>\u65b0\u3057\u3044DSL\u3092\u899a\u3048\u308b\u5fc5\u8981\u304c\u306a\u304f\u3001Python\u306e\u77e5\u8b58\u3092\u305d\u306e\u307e\u307e\u6d3b\u304b\u305b\u307e\u3059\u3002<!-- notionvc: 164b7c4e-7004-4692-ae5d-df8253f0f82c --><\/p>\n<p><strong>\u30fb<span data-token-index=\"0\" class=\"notion-enable-hover\">\u8c4a\u5bcc\u306aProvider<\/span><!-- notionvc: ef69d915-bb78-42dc-b1a1-1745099aadca --><br \/><\/strong>AWS\u3001Google Cloud\u3001Azure\u3001Snowflake\u3001Databricks\u3001Slack\u306a\u3069\u3001\u591a\u6570\u306eProvider\u304c\u7528\u610f\u3055\u308c\u3066\u304a\u308a\u3001\u5916\u90e8\u30b5\u30fc\u30d3\u30b9\u3068\u306e\u9023\u643a\u304c\u5bb9\u6613\u3067\u3059\u3002<!-- notionvc: 776768a7-48ee-4ab4-a089-ff7bdc309dbe --><\/p>\n<p><strong>\u30fb<span data-token-index=\"0\" class=\"notion-enable-hover\">\u9ad8\u3044\u62e1\u5f35\u6027<\/span><!-- notionvc: 50f10002-d319-453a-acc8-d5b21a1a231d --><br \/><\/strong>\u30ab\u30b9\u30bf\u30e0Operator\u3084Hook\u3092\u4f5c\u6210\u3067\u304d\u308b\u305f\u3081\u3001\u81ea\u793e\u30b7\u30b9\u30c6\u30e0\u3068\u306e\u9023\u643a\u3082\u67d4\u8edf\u3067\u3059\u3002<!-- notionvc: f3a15899-f02c-4415-8d82-800d3352f675 --><\/p>\n<p><strong>\u30fb<span data-token-index=\"0\" class=\"notion-enable-hover\">OSS\u30b3\u30df\u30e5\u30cb\u30c6\u30a3\u304c\u6d3b\u767a<\/span><!-- notionvc: b15035c0-df39-4692-b75b-d52f171a281d --><br \/><\/strong>\u4e16\u754c\u4e2d\u306e\u4f01\u696d\u3067\u5229\u7528\u3055\u308c\u3066\u3044\u308b\u305f\u3081\u3001\u30a2\u30c3\u30d7\u30c7\u30fc\u30c8\u3084\u60c5\u5831\u304c\u8c4a\u5bcc\u3067\u3059\u3002<!-- notionvc: b382a9bb-0dd7-4a6f-b43e-f5322dd2b0e6 --><\/p>\n<p><!-- notionvc: 2468fd3b-0203-4cae-a990-109fcf2a437c --><\/p>\n<p><b>Airflow\u306e\u30c7\u30e1\u30ea\u30c3\u30c8<\/b><\/p>\n<p><strong>\u30fb<span data-token-index=\"0\" class=\"notion-enable-hover\">\u521d\u671f\u8a2d\u8a08\u304c\u91cd\u8981<\/span><!-- notionvc: df384ebb-710e-4d3a-aa6c-36498e913feb --><br \/><\/strong>DAG\u8a2d\u8a08\u3092\u8aa4\u308b\u3068\u4fdd\u5b88\u6027\u304c\u4f4e\u4e0b\u3057\u3001\u904b\u7528\u8ca0\u8377\u304c\u5897\u52a0\u3057\u307e\u3059\u3002<!-- notionvc: dc68d9e1-9113-48e5-811e-2a375cce2f8c --><br \/><!-- notionvc: 164b7c4e-7004-4692-ae5d-df8253f0f82c --><\/p>\n<p><strong>\u30fb<span data-token-index=\"0\" class=\"notion-enable-hover\">\u30ea\u30a2\u30eb\u30bf\u30a4\u30e0\u51e6\u7406\u306f\u5f97\u610f\u3067\u306f\u306a\u3044<\/span><!-- notionvc: 4fb4f2e2-0fed-45f3-bfa1-ddcc300688df --><br \/><\/strong>Airflow\u306f\u30a4\u30d9\u30f3\u30c8\u99c6\u52d5\u3092\u5f37\u5316\u3057\u3066\u3044\u307e\u3059\u304c\u3001\u30b9\u30c8\u30ea\u30fc\u30e0\u51e6\u7406\u81ea\u4f53\u3092\u62c5\u3046\u30c4\u30fc\u30eb\u3067\u306f\u3042\u308a\u307e\u305b\u3093\u3002Kafka\u3084Apache Flink\u306a\u3069\u3068\u7d44\u307f\u5408\u308f\u305b\u308b\u306e\u304c\u4e00\u822c\u7684\u3067\u3059\u3002<!-- notionvc: 697bf850-4513-4893-9926-b903e8dbc854 --><br \/><!-- notionvc: 776768a7-48ee-4ab4-a089-ff7bdc309dbe --><\/p>\n<p><strong>\u30fb<span data-token-index=\"0\" class=\"notion-enable-hover\">\u5927\u898f\u6a21\u74b0\u5883\u3067\u306f\u904b\u7528\u30ce\u30a6\u30cf\u30a6\u304c\u5fc5\u8981<\/span><!-- notionvc: b334edd4-3675-4697-8c06-1ae6179a87b8 --><br \/><\/strong>\u6570\u5343\u301c\u6570\u4e07\u306e\u30bf\u30b9\u30af\u3092\u6271\u3046\u5834\u5408\u306f\u3001Scheduler\u3084Executor\u306e\u30c1\u30e5\u30fc\u30cb\u30f3\u30b0\u3001\u30e1\u30bf\u30c7\u30fc\u30bfDB\u306e\u6700\u9069\u5316\u306a\u3069\u304c\u91cd\u8981\u306b\u306a\u308a\u307e\u3059\u3002<!-- notionvc: 68e79f5c-756e-472f-b2ba-497458c5fe84 --><\/p>\n<p><!-- notionvc: 24a43c7f-cad2-4d9e-a1ae-f5ef8644d093 --><\/p>\n<h2>Airflow\u3068\u4ed6\u30c4\u30fc\u30eb\u306e\u9055\u3044<!-- notionvc: 2fcde327-ca89-4a98-bd23-9d425263504d --><br \/><!-- notionvc: e2fb5c31-164e-4d68-ab10-78342a6feb38 --><\/h2>\n<table id=\"3ad9bc0d-8e00-805f-84ca-edc8d8e6e3cf\" class=\"simple-table\">\n<thead class=\"simple-table-header\">\n<tr id=\"3ad9bc0d-8e00-804a-aafd-f966952ec1c2\" dir=\"ltr\">\n<th id=\"J?dS\" class=\"simple-table-header-color simple-table-header\">\u30c4\u30fc\u30eb<\/th>\n<th id=\"^=Nu\" class=\"simple-table-header-color simple-table-header\">\u4e3b\u306a\u7528\u9014<\/th>\n<th id=\"OB{W\" class=\"simple-table-header-color simple-table-header\">\u7279\u5fb4<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr id=\"3ad9bc0d-8e00-80f8-891d-feb9ef9cd144\" dir=\"ltr\">\n<td id=\"J?dS\" class=\"\">Airflow<\/td>\n<td id=\"^=Nu\" class=\"\">\u30ef\u30fc\u30af\u30d5\u30ed\u30fc\u7ba1\u7406<\/td>\n<td id=\"OB{W\" class=\"\">Python\u3067\u67d4\u8edf\u306bDAG\u3092\u8a18\u8ff0\u3067\u304d\u308b<\/td>\n<\/tr>\n<tr id=\"3ad9bc0d-8e00-8061-899c-da6703725091\" dir=\"ltr\">\n<td id=\"J?dS\" class=\"\">dbt<\/td>\n<td id=\"^=Nu\" class=\"\">SQL\u5909\u63db<\/td>\n<td id=\"OB{W\" class=\"\">\u30c7\u30fc\u30bf\u5909\u63db\u30fb\u30e2\u30c7\u30ea\u30f3\u30b0\u306b\u7279\u5316<\/td>\n<\/tr>\n<tr id=\"3ad9bc0d-8e00-80b7-a563-fdde8234d969\" dir=\"ltr\">\n<td id=\"J?dS\" class=\"\">Dagster<\/td>\n<td id=\"^=Nu\" class=\"\">\u30c7\u30fc\u30bf\u30aa\u30fc\u30b1\u30b9\u30c8\u30ec\u30fc\u30b7\u30e7\u30f3<\/td>\n<td id=\"OB{W\" class=\"\">\u30c7\u30fc\u30bf\u8cc7\u7523\uff08Asset\uff09\u4e2d\u5fc3\u306e\u8a2d\u8a08<\/td>\n<\/tr>\n<tr id=\"3ad9bc0d-8e00-8031-bcb5-f8319f22b807\" dir=\"ltr\">\n<td id=\"J?dS\" class=\"\">Prefect<\/td>\n<td id=\"^=Nu\" class=\"\">\u30ef\u30fc\u30af\u30d5\u30ed\u30fc\u7ba1\u7406<\/td>\n<td id=\"OB{W\" class=\"\">\u958b\u767a\u8005\u4f53\u9a13\u3092\u91cd\u8996\u3057\u30af\u30e9\u30a6\u30c9\u9023\u643a\u304c\u5bb9\u6613<\/td>\n<\/tr>\n<tr id=\"3ad9bc0d-8e00-8056-b6c6-e47680328f4a\" dir=\"ltr\">\n<td id=\"J?dS\" class=\"\">Databricks Workflows<\/td>\n<td id=\"^=Nu\" class=\"\">Databricks\u30b8\u30e7\u30d6\u7ba1\u7406<\/td>\n<td id=\"OB{W\" class=\"\">Databricks\u74b0\u5883\u3068\u306e\u7d71\u5408\u306b\u6700\u9069<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><br \/><!-- notionvc: 84d8ff4e-132b-4890-a074-1199e2f573db --><\/p>\n<p><span style=\"font-size: revert;\">\u8fd1\u5e74\u306f\u300cAirflow\u304bdbt\u304b\u300d\u3067\u306f\u306a\u304f\u3001Airflow\u3067dbt\u3084Databricks\u30b8\u30e7\u30d6\u3092\u30aa\u30fc\u30b1\u30b9\u30c8\u30ec\u30fc\u30b7\u30e7\u30f3\u3059\u308b\u69cb\u6210\u304c\u4e00\u822c\u7684\u306b\u306a\u3063\u3066\u3044\u307e\u3059\u3002<\/span><\/p>\n<h2>Airflow\u306e\u5c06\u6765\u5c55\u671b<\/h2>\n<p>Airflow\u306f\u4eca\u5f8c\u3082\u3001\u30c7\u30fc\u30bf\u57fa\u76e4\u30fbAI\u57fa\u76e4\u3092\u6a2a\u65ad\u7684\u306b\u7ba1\u7406\u3059\u308b\u30d7\u30e9\u30c3\u30c8\u30d5\u30a9\u30fc\u30e0\u3068\u3057\u3066\u9032\u5316\u3059\u308b\u3068\u8003\u3048\u3089\u308c\u307e\u3059\u3002<\/p>\n<p>\u4eca\u5f8c\u671f\u5f85\u3055\u308c\u308b\u30dd\u30a4\u30f3\u30c8\u306f\u6b21\u306e\u3068\u304a\u308a\u3067\u3059\u3002<\/p>\n<p>\u30fb\u30a4\u30d9\u30f3\u30c8\u99c6\u52d5\u578b\u30ef\u30fc\u30af\u30d5\u30ed\u30fc\u306e\u3055\u3089\u306a\u308b\u5f37\u5316<br \/>\u30fbAI\u30a8\u30fc\u30b8\u30a7\u30f3\u30c8\u3068\u306e\u7d71\u5408<br \/>\u30fbOpenLineage\u306b\u3088\u308b\u30c7\u30fc\u30bf\u30ea\u30cd\u30fc\u30b8\u7ba1\u7406\u306e\u6a19\u6e96\u5316<br \/>\u30fb\u30af\u30e9\u30a6\u30c9\u30cd\u30a4\u30c6\u30a3\u30d6\u904b\u7528\u306e\u9ad8\u5ea6\u5316<br \/>\u30fb\u30c7\u30fc\u30bf\u30a2\u30bb\u30c3\u30c8\u4e2d\u5fc3\u306e\u30aa\u30fc\u30b1\u30b9\u30c8\u30ec\u30fc\u30b7\u30e7\u30f3\u306e\u9032\u5316<\/p>\n<p>AI\u6642\u4ee3\u306b\u304a\u3044\u3066\u3082\u3001\u8907\u96d1\u5316\u3059\u308b\u30c7\u30fc\u30bf\u30d1\u30a4\u30d7\u30e9\u30a4\u30f3\u3092\u5b89\u5b9a\u7684\u306b\u7ba1\u7406\u3059\u308b\u5f79\u5272\u306f\u3001\u307e\u3059\u307e\u3059\u91cd\u8981\u306b\u306a\u308b\u3067\u3057\u3087\u3046\u3002<!-- notionvc: a5309aa0-4279-4756-b9c6-6fc4cec04ed2 --><\/p>\n<p><!-- notionvc: 01702f55-eecd-4b75-87a2-821ebc4b2360 --><\/p>\n<h2>\u307e\u3068\u3081<\/h2>\n<p>Apache Airflow\u306f\u3001\u5358\u306a\u308b\u30d0\u30c3\u30c1\u51e6\u7406\u30c4\u30fc\u30eb\u3067\u306f\u306a\u304f\u3001\u73fe\u4ee3\u306e\u30c7\u30fc\u30bf\u30d7\u30e9\u30c3\u30c8\u30d5\u30a9\u30fc\u30e0\u5168\u4f53\u3092\u7d71\u62ec\u3059\u308b\u30ef\u30fc\u30af\u30d5\u30ed\u30fc\u30aa\u30fc\u30b1\u30b9\u30c8\u30ec\u30fc\u30b7\u30e7\u30f3\u57fa\u76e4\u3078\u3068\u9032\u5316\u3057\u3066\u3044\u307e\u3059\u3002<\/p>\n<p>Airflow 3.x\u3067\u306f\u3001\u30c7\u30fc\u30bf\u30a2\u30bb\u30c3\u30c8\u7ba1\u7406\u3084\u30a4\u30d9\u30f3\u30c8\u99c6\u52d5\u578b\u30b9\u30b1\u30b8\u30e5\u30fc\u30ea\u30f3\u30b0\u3001\u30af\u30e9\u30a6\u30c9\u30cd\u30a4\u30c6\u30a3\u30d6\u74b0\u5883\u3068\u306e\u7d71\u5408\u304c\u3055\u3089\u306b\u5f37\u5316\u3055\u308c\u3001ETL\u30fbELT\u3060\u3051\u3067\u306a\u304f\u3001MLOps\u3084LLMOps\u3001\u751f\u6210AI\u57fa\u76e4\u306e\u904b\u7528\u306b\u3082\u6b20\u304b\u305b\u306a\u3044\u5b58\u5728\u3068\u306a\u308a\u307e\u3057\u305f\u3002<\/p>\n<p>\u4e00\u65b9\u3067\u3001dbt\u3084Databricks Workflows\u3001Dagster\u3001Prefect\u306a\u3069\u76ee\u7684\u306b\u5fdc\u3058\u305f\u30c4\u30fc\u30eb\u3068\u306e\u9069\u5207\u306a\u4f7f\u3044\u5206\u3051\u3082\u91cd\u8981\u3067\u3059\u3002\u81ea\u793e\u306e\u30c7\u30fc\u30bf\u57fa\u76e4\u3084AI\u57fa\u76e4\u306e\u69cb\u6210\u3092\u8e0f\u307e\u3048\u3001Airflow\u3092\u300c\u30aa\u30fc\u30b1\u30b9\u30c8\u30ec\u30fc\u30b7\u30e7\u30f3\u306e\u53f8\u4ee4\u5854\u300d\u3068\u3057\u3066\u4f4d\u7f6e\u3065\u3051\u308b\u3053\u3068\u3067\u3001\u62e1\u5f35\u6027\u3068\u904b\u7528\u52b9\u7387\u3092\u4e21\u7acb\u3057\u305f\u30e2\u30c0\u30f3\u306a\u30c7\u30fc\u30bf\u30d1\u30a4\u30d7\u30e9\u30a4\u30f3\u3092\u5b9f\u73fe\u3067\u304d\u308b\u3067\u3057\u3087\u3046\u3002<!-- notionvc: 1e7d9132-563d-4e3c-a161-12aa6f7ab478 --><!-- notionvc: bf6057a0-81cf-4585-b6db-6e9dae5fb792 --><\/p>","protected":false},"excerpt":{"rendered":"<p>Apache Airflow\u3068\u306f\u4f55\u304b\u30922026\u5e74\u6700\u65b0\u60c5\u5831\u3092\u3082\u3068\u306b\u5fb9\u5e95\u89e3\u8aac\u3002Airflow 3.x\u306e\u65b0\u6a5f\u80fd\u3001DAG\u306e\u4ed5\u7d44\u307f\u3001AI\u30fb\u30c7\u30fc\u30bf\u57fa\u76e4\u3067\u306e\u6d3b\u7528\u4e8b\u4f8b\u3001Databricks\u30fbSnowflake\u30fbdbt\u3068\u306e\u9023\u643a\u3001\u30e1\u30ea\u30c3\u30c8\u30fb [&hellip;]<\/p>\n","protected":false},"author":22,"featured_media":6222,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","swell_btn_cv_data":"","footnotes":"","_wp_rev_ctl_limit":""},"categories":[1249],"tags":[558,59],"class_list":["post-6329","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-knowledge","tag-airflow","tag-data-driven"],"_links":{"self":[{"href":"https:\/\/since2020.jp\/media\/wp-json\/wp\/v2\/posts\/6329","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/since2020.jp\/media\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/since2020.jp\/media\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/since2020.jp\/media\/wp-json\/wp\/v2\/users\/22"}],"replies":[{"embeddable":true,"href":"https:\/\/since2020.jp\/media\/wp-json\/wp\/v2\/comments?post=6329"}],"version-history":[{"count":1,"href":"https:\/\/since2020.jp\/media\/wp-json\/wp\/v2\/posts\/6329\/revisions"}],"predecessor-version":[{"id":9244,"href":"https:\/\/since2020.jp\/media\/wp-json\/wp\/v2\/posts\/6329\/revisions\/9244"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/since2020.jp\/media\/wp-json\/wp\/v2\/media\/6222"}],"wp:attachment":[{"href":"https:\/\/since2020.jp\/media\/wp-json\/wp\/v2\/media?parent=6329"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/since2020.jp\/media\/wp-json\/wp\/v2\/categories?post=6329"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/since2020.jp\/media\/wp-json\/wp\/v2\/tags?post=6329"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}