{"id":7200,"date":"2025-09-30T15:37:37","date_gmt":"2025-09-30T06:37:37","guid":{"rendered":"https:\/\/blog.since2020.jp\/?p=7200"},"modified":"2026-09-29T17:17:26","modified_gmt":"2026-09-29T08:17:26","slug":"sparksql_glossary","status":"publish","type":"post","link":"https:\/\/since2020.jp\/media\/sparksql_glossary\/","title":{"rendered":"SparkSQL\u3092\u300c\u3044\u307e\u300d\u4f7f\u3044\u3053\u306a\u3059\u2015\u2015Spark 4.2\u6642\u4ee3\u306e\u8a2d\u8a08\u601d\u60f3\u30fb\u5b9f\u8df5\u30fb\u6700\u65b0\u30c8\u30ec\u30f3\u30c9"},"content":{"rendered":"\n

SparkSQL\u306e\u57fa\u672c\u6982\u5ff5\u304b\u3089Catalyst\u3001AQE\u3001Spark Connect\u3001ANSI SQL\u3001Unity Catalog\u307e\u3067\u30012026\u5e74\u306eSpark 4.2\u6642\u4ee3\u306b\u62bc\u3055\u3048\u308b\u3079\u304d\u8a2d\u8a08\u601d\u60f3\u3068\u5b9f\u8df5\u7684\u306a\u30c1\u30e5\u30fc\u30cb\u30f3\u30b0\u65b9\u6cd5\u3092\u89e3\u8aac\u3002\u30ec\u30a4\u30af\u30cf\u30a6\u30b9\u3001ETL\u3001BI\u3001\u6a5f\u68b0\u5b66\u7fd2\u3001\u30b9\u30c8\u30ea\u30fc\u30df\u30f3\u30b0\u307e\u3067\u3001SQL\u3092\u4e2d\u5fc3\u306b\u5927\u898f\u6a21\u30c7\u30fc\u30bf\u51e6\u7406\u3092\u8a2d\u8a08\u3059\u308b\u305f\u3081\u306e\u8003\u3048\u65b9\u3092\u6574\u7406\u3057\u307e\u3059\u3002<\/p>\n\n\n

SparkSQL\u306f\u300cSQL\u3067Spark\u3092\u4f7f\u3046\u300d\u3060\u3051\u3067\u306f\u306a\u3044<\/h2>\n

Apache Spark\u306eSQL\u6a5f\u80fd\u3092\u6307\u3059SparkSQL\u306f\u3001\u5358\u306b\u5927\u91cf\u30c7\u30fc\u30bf\u3078SQL\u3092\u767a\u884c\u3059\u308b\u305f\u3081\u306e\u4ed5\u7d44\u307f\u3067\u306f\u3042\u308a\u307e\u305b\u3093\u3002SQL\u3001DataFrame\u3001Dataset\u3068\u3044\u3046\u5ba3\u8a00\u7684\u306aAPI\u3067\u300c\u4f55\u3092\u8a08\u7b97\u3057\u305f\u3044\u304b\u300d\u3092\u8a18\u8ff0\u3057\u3001\u305d\u306e\u51e6\u7406\u65b9\u6cd5\u3092Spark\u5074\u304c\u6700\u9069\u5316\u3059\u308b\u3053\u3068\u306b\u672c\u8cea\u304c\u3042\u308a\u307e\u3059\u3002<\/p>\n

\u73fe\u5728\u306eSpark\u306f\u30012026\u5e747\u6708\u306b4.2.0\u304c\u516c\u958b\u3055\u308c\u30014.x\u7cfb\u304c\u672c\u683c\u7684\u306b\u9032\u5c55\u3057\u3066\u3044\u307e\u3059\u30024.1\u3067\u306fSQL Scripting\u304cGA\u3068\u306a\u308a\u3001VARIANT\u578b\u3001\u518d\u5e30CTE\u3001\u8ffd\u52a0\u306e\u8fd1\u4f3c\u96c6\u8a08\u6a5f\u80fd\u306a\u3069\u304c\u5f37\u5316\u3055\u308c\u305f\u307b\u304b\u3001Spark Declarative Pipelines\u3084Structured Streaming\u306eReal-Time Mode\u3082\u767b\u5834\u3057\u307e\u3057\u305f\u3002\u3064\u307e\u308aSparkSQL\u306f\u3001\u5f93\u6765\u306e\u300c\u5927\u898f\u6a21\u30d0\u30c3\u30c1SQL\u300d\u304b\u3089\u3001\u30c7\u30fc\u30bf\u30d1\u30a4\u30d7\u30e9\u30a4\u30f3\u3001\u534a\u69cb\u9020\u5316\u30c7\u30fc\u30bf\u3001\u30b9\u30c8\u30ea\u30fc\u30df\u30f3\u30b0\u307e\u3067\u3092\u6271\u3046\u5b9f\u884c\u57fa\u76e4\u3078\u5e83\u304c\u3063\u3066\u3044\u307e\u3059\u3002<\/p>\n

\u3053\u3053\u3067\u91cd\u8981\u306a\u306e\u306f\u3001\u6700\u65b0\u6a5f\u80fd\u3092\u899a\u3048\u308b\u3053\u3068\u3088\u308a\u3082\u3001SQL\u3092\u3069\u3046\u66f8\u3051\u3070Spark\u306e\u6700\u9069\u5316\u80fd\u529b\u3092\u5f15\u304d\u51fa\u305b\u308b\u304b<\/strong>\u3092\u7406\u89e3\u3059\u308b\u3053\u3068\u3067\u3059\u3002<\/p>\n

<\/p>\n

Catalyst\u3068AQE\u3092\u7406\u89e3\u3059\u308b\u3068SparkSQL\u304c\u898b\u3048\u3066\u304f\u308b<\/h2>\n

SparkSQL\u306e\u5f37\u3055\u306f\u3001\u30af\u30a8\u30ea\u3092\u305d\u306e\u307e\u307e\u5b9f\u884c\u3057\u306a\u3044\u70b9\u306b\u3042\u308a\u307e\u3059\u3002SQL\u3084DataFrame\u306e\u51e6\u7406\u306f\u8ad6\u7406\u8a08\u753b\u3068\u3057\u3066\u8868\u73fe\u3055\u308c\u3001Analyzer\u306b\u3088\u308b\u540d\u524d\u3084\u578b\u306e\u89e3\u6c7a\u3001Catalyst Optimizer\u306b\u3088\u308b\u8ad6\u7406\u6700\u9069\u5316\u3001\u7269\u7406\u8a08\u753b\u306e\u9078\u629e\u3092\u7d4c\u3066\u3001\u6700\u7d42\u7684\u306a\u5b9f\u884c\u51e6\u7406\u3078\u5909\u63db\u3055\u308c\u307e\u3059\u3002<\/p>\n

\u305f\u3068\u3048\u3070\u4e0d\u8981\u306a\u5217\u3092\u8aad\u307f\u8fbc\u307e\u306a\u3044\u3001\u30d5\u30a3\u30eb\u30bf\u3092\u53ef\u80fd\u306a\u9650\u308a\u30c7\u30fc\u30bf\u30bd\u30fc\u30b9\u5074\u3078\u62bc\u3057\u8fbc\u3080\u3001\u9069\u5207\u306a\u7d50\u5408\u65b9\u5f0f\u3092\u9078\u3076\u3068\u3044\u3063\u305f\u51e6\u7406\u306f\u3001\u30e6\u30fc\u30b6\u30fc\u304c\u3059\u3079\u3066\u624b\u4f5c\u696d\u3067\u6307\u5b9a\u3059\u308b\u5fc5\u8981\u304c\u3042\u308a\u307e\u305b\u3093\u3002<\/p>\n

\u3055\u3089\u306b\u73fe\u5728\u306eSpark\u3067\u306f\u3001**Adaptive Query Execution\uff08AQE\uff09**\u304c\u91cd\u8981\u3067\u3059\u3002AQE\u306f\u5b9f\u884c\u4e2d\u306b\u5f97\u3089\u308c\u305f\u7d71\u8a08\u60c5\u5831\u3092\u4f7f\u3063\u3066\u8a08\u753b\u3092\u518d\u6700\u9069\u5316\u3057\u307e\u3059\u3002\u30b7\u30e3\u30c3\u30d5\u30eb\u5f8c\u306e\u30d1\u30fc\u30c6\u30a3\u30b7\u30e7\u30f3\u7d71\u5408\u3001\u30c7\u30fc\u30bf\u30b9\u30ad\u30e5\u30fc\u306e\u5206\u5272\u3001\u5b9f\u884c\u6642\u30b5\u30a4\u30ba\u306b\u5fdc\u3058\u305f\u7d50\u5408\u65b9\u5f0f\u306e\u5909\u66f4\u306a\u3069\u306b\u5bfe\u5fdc\u3057\u3001AQE\u81ea\u4f53\u306fSpark 3.2\u4ee5\u964d\u30c7\u30d5\u30a9\u30eb\u30c8\u3067\u6709\u52b9\u3067\u3059\u3002<\/p>\n

\u305d\u306e\u305f\u3081\u3001SparkSQL\u306e\u30c1\u30e5\u30fc\u30cb\u30f3\u30b0\u3067\u306f\u300c\u8a2d\u5b9a\u5024\u3092\u3072\u305f\u3059\u3089\u5909\u66f4\u3059\u308b\u300d\u3088\u308a\u3001\u307e\u305aEXPLAIN<\/code>\u3084Spark UI\u3067\u5b9f\u969b\u306b\u3069\u306e\u5b9f\u884c\u8a08\u753b\u304c\u9078\u3070\u308c\u305f\u306e\u304b<\/strong>\u3092\u898b\u308b\u3053\u3068\u304c\u91cd\u8981\u3067\u3059\u3002<\/p>\n

<\/p>\n\n

Spark 4.x\u3067\u7279\u306b\u91cd\u8981\u306a\u5909\u5316<\/strong><\/p>\n\n

ANSI SQL\u304c\u6a19\u6e96\u306b\u306a\u3063\u305f<\/strong><\/p>\n

Spark 4.0\u3067\u306fspark.sql.ansi.enabled<\/code>\u304c\u30c7\u30d5\u30a9\u30eb\u30c8\u3067true\u306b\u306a\u308a\u307e\u3057\u305f\u3002\u7121\u52b9\u306a\u6f14\u7b97\u306a\u3069\u306b\u5bfe\u3057\u3066NULL\u3092\u8fd4\u3057\u3066\u51e6\u7406\u3092\u7d99\u7d9a\u3059\u308b\u306e\u3067\u306f\u306a\u304f\u3001\u30a8\u30e9\u30fc\u3068\u3057\u3066\u6271\u3046ANSI\u6e96\u62e0\u306e\u6319\u52d5\u304c\u6a19\u6e96\u306b\u306a\u3063\u3066\u3044\u307e\u3059\u3002<\/p>\n

\u3053\u308c\u306f\u300c\u53b3\u3057\u304f\u306a\u3063\u305f\u300d\u3068\u3044\u3046\u3060\u3051\u3067\u306f\u3042\u308a\u307e\u305b\u3093\u3002\u30c7\u30fc\u30bf\u54c1\u8cea\u4e0a\u306e\u554f\u984c\u3092\u65e9\u671f\u306b\u767a\u898b\u3057\u3084\u3059\u304f\u306a\u308b\u305f\u3081\u3001ETL\u3084\u5206\u6790\u57fa\u76e4\u306e\u4fe1\u983c\u6027\u3092\u9ad8\u3081\u308b\u65b9\u5411\u306e\u5909\u66f4\u3067\u3059\u3002<\/p>\n

\u4e00\u65b9\u3001Spark 3.x\u304b\u30894.x\u3078\u79fb\u884c\u3059\u308b\u5834\u5408\u3001\u6697\u9ed9\u306e\u578b\u5909\u63db\u3084\u30aa\u30fc\u30d0\u30fc\u30d5\u30ed\u30fc\u306a\u3069\u3092\u524d\u63d0\u306b\u3057\u305f\u65e2\u5b58SQL\u304c\u5931\u6557\u3059\u308b\u53ef\u80fd\u6027\u304c\u3042\u308a\u307e\u3059\u3002\u79fb\u884c\u3067\u306f\u5358\u7d14\u306a\u30d0\u30fc\u30b8\u30e7\u30f3\u30a2\u30c3\u30d7\u3067\u306f\u306a\u304f\u3001\u578b\u3001NULL\u3001\u65e5\u4ed8\u3001\u6570\u5024\u5909\u63db\u3092\u542b\u3081\u305f\u56de\u5e30\u30c6\u30b9\u30c8\u304c\u5fc5\u8981\u3067\u3059\u3002<\/p>\n

SQL\u305d\u306e\u3082\u306e\u304c\u9ad8\u6a5f\u80fd\u5316\u3057\u3066\u3044\u308b<\/strong><\/p>\n

Spark 4.1\u3067\u306fSQL Scripting\u304cGA\u3068\u306a\u308a\u3001\u5909\u6570\u3084\u5236\u5fa1\u69cb\u6587\u306a\u3069\u3092\u5229\u7528\u3057\u305f\u3001\u3088\u308a\u624b\u7d9a\u304d\u7684\u306aSQL\u51e6\u7406\u3092\u6b63\u5f0f\u306b\u69cb\u7bc9\u3067\u304d\u308b\u3088\u3046\u306b\u306a\u308a\u307e\u3057\u305f\u3002<\/p>\n

\u3055\u3089\u306bVARIANT\u578b\u3082GA\u3068\u306a\u308a\u3001JSON\u306a\u3069\u306e\u534a\u69cb\u9020\u5316\u30c7\u30fc\u30bf\u3092\u6271\u3046\u5834\u9762\u3067SQL\u306e\u8868\u73fe\u529b\u304c\u9ad8\u307e\u3063\u3066\u3044\u307e\u3059\u3002\u518d\u5e30CTE\u3084\u591a\u6570\u306e\u7d44\u307f\u8fbc\u307f\u95a2\u6570\u3082\u8ffd\u52a0\u3055\u308c\u3001\u5f93\u6765\u306a\u3089Python\u3084\u5916\u90e8\u51e6\u7406\u3078\u9003\u304c\u3057\u3066\u3044\u305f\u30ed\u30b8\u30c3\u30af\u3092SQL\u5074\u3078\u5bc4\u305b\u3089\u308c\u308b\u30b1\u30fc\u30b9\u304c\u5897\u3048\u3066\u3044\u307e\u3059\u3002<\/p>\n

\u3053\u308c\u306f\u300cPython\u3092\u4f7f\u308f\u306a\u304f\u3066\u3088\u3044\u300d\u3068\u3044\u3046\u610f\u5473\u3067\u306f\u3042\u308a\u307e\u305b\u3093\u3002\u3080\u3057\u308d\u3001SQL\u3067\u8868\u73fe\u3067\u304d\u308b\u51e6\u7406\u306fSQL\u306b\u5bc4\u305b\u3001Spark\u306e\u6700\u9069\u5316\u5bfe\u8c61\u3068\u3057\u3066\u6271\u3046<\/strong>\u3068\u3044\u3046\u8a2d\u8a08\u5224\u65ad\u304c\u3057\u3084\u3059\u304f\u306a\u3063\u305f\u3068\u8003\u3048\u308b\u3079\u304d\u3067\u3057\u3087\u3046\u3002<\/p>\n

Spark Connect\u304c\u5909\u3048\u308b\u30a2\u30d7\u30ea\u30b1\u30fc\u30b7\u30e7\u30f3\u8a2d\u8a08<\/strong><\/h2>\n

Spark Connect\u306f\u3001Spark\u30af\u30e9\u30a4\u30a2\u30f3\u30c8\u3068Spark\u30b5\u30fc\u30d0\u30fc\u3092\u5206\u96e2\u3059\u308b\u4ed5\u7d44\u307f\u3067\u3059\u3002\u30af\u30e9\u30a4\u30a2\u30f3\u30c8\u306fDataFrame\u64cd\u4f5c\u3092\u672a\u89e3\u6c7a\u306e\u8ad6\u7406\u8a08\u753b\u3068\u3057\u3066Protocol Buffers\u3067\u30b7\u30ea\u30a2\u30e9\u30a4\u30ba\u3057\u3001gRPC\u7d4c\u7531\u3067\u30b5\u30fc\u30d0\u30fc\u3078\u9001\u308a\u307e\u3059\u3002\u7d50\u679c\u306fApache Arrow\u306e\u30d0\u30c3\u30c1\u3068\u3057\u3066\u8fd4\u5374\u3055\u308c\u307e\u3059\u3002<\/p>\n

\u5f93\u6765\u578b\u306eSpark\u3067\u306f\u3001\u30a2\u30d7\u30ea\u30b1\u30fc\u30b7\u30e7\u30f3\u3068Spark Driver\u306e\u95a2\u4fc2\u304c\u5bc6\u63a5\u3067\u3057\u305f\u3002Connect\u3067\u306f\u30af\u30e9\u30a4\u30a2\u30f3\u30c8\u3092\u5206\u96e2\u3067\u304d\u308b\u305f\u3081\u3001IDE\u3001Web\u30a2\u30d7\u30ea\u30b1\u30fc\u30b7\u30e7\u30f3\u3001\u30c7\u30fc\u30bf\u30b5\u30fc\u30d3\u30b9\u306a\u3069\u304b\u3089\u30ea\u30e2\u30fc\u30c8Spark\u3092\u5229\u7528\u3057\u3084\u3059\u304f\u306a\u308a\u307e\u3059\u3002<\/p>\n

\u305f\u3060\u3057\u3001\u5b8c\u5168\u4e92\u63db\u3067\u306f\u3042\u308a\u307e\u305b\u3093\u3002SparkContext\u3084RDD\u306a\u3069\u3001Driver\u5185\u90e8\u306b\u4f9d\u5b58\u3059\u308bAPI\u306fSpark Connect\u3067\u306f\u5229\u7528\u3067\u304d\u307e\u305b\u3093\u3002\u65e2\u5b58\u30a2\u30d7\u30ea\u30b1\u30fc\u30b7\u30e7\u30f3\u3092\u79fb\u884c\u3059\u308b\u5834\u5408\u306f\u3001\u5229\u7528API\u304cConnect\u5bfe\u5fdc\u304b\u3092\u4e8b\u524d\u306b\u78ba\u8a8d\u3059\u308b\u5fc5\u8981\u304c\u3042\u308a\u307e\u3059\u3002<\/p>\n

\u3064\u307e\u308aSpark Connect\u306e\u4fa1\u5024\u306f\u5358\u306a\u308b\u300c\u30ea\u30e2\u30fc\u30c8\u63a5\u7d9a\u300d\u3067\u306f\u306a\u304f\u3001Spark\u3092\u30a2\u30d7\u30ea\u30b1\u30fc\u30b7\u30e7\u30f3\u304b\u3089\u5229\u7528\u3059\u308b\u969b\u306e\u5883\u754c\u3092\u660e\u78ba\u306b\u3059\u308b\u3053\u3068<\/strong>\u306b\u3042\u308a\u307e\u3059\u3002<\/p>\n

<\/p>\n

\u5b9f\u52d9\u3067\u6700\u521d\u306b\u898b\u308b\u3079\u304d\u306fI\/O\u3068\u30c7\u30fc\u30bf\u914d\u7f6e<\/strong><\/h2>\n

SparkSQL\u3092\u9ad8\u901f\u5316\u3059\u308b\u3068\u304d\u3001\u3044\u304d\u306a\u308ashuffle partitions<\/code>\u3084\u30ad\u30e3\u30c3\u30b7\u30e5\u8a2d\u5b9a\u3092\u5909\u66f4\u3059\u308b\u306e\u306f\u5f97\u7b56\u3067\u306f\u3042\u308a\u307e\u305b\u3093\u3002<\/p>\n

\u6700\u521d\u306b\u898b\u308b\u3079\u304d\u306a\u306e\u306f\u3001\u3069\u308c\u3060\u3051\u306e\u30c7\u30fc\u30bf\u3092\u8aad\u3093\u3067\u3044\u308b\u304b\u3067\u3059\u3002<\/p>\n

Parquet\u306a\u3069\u306e\u5217\u6307\u5411\u30d5\u30a9\u30fc\u30de\u30c3\u30c8\u3092\u5229\u7528\u3057\u3001\u5fc5\u8981\u306a\u5217\u3060\u3051\u3092\u53d6\u5f97\u3067\u304d\u308c\u3070\u3001I\/O\u3092\u5927\u5e45\u306b\u6e1b\u3089\u305b\u307e\u3059\u3002\u3055\u3089\u306b\u30d5\u30a3\u30eb\u30bf\u6761\u4ef6\u304c\u30c7\u30fc\u30bf\u30bd\u30fc\u30b9\u5074\u307e\u3067\u62bc\u3057\u8fbc\u307e\u308c\u308b\u3053\u3068\u3067\u3001\u8aad\u307f\u8fbc\u3080\u30c7\u30fc\u30bf\u91cf\u3092\u524a\u6e1b\u3067\u304d\u307e\u3059\u3002<\/p>\n

\u6b21\u306b\u91cd\u8981\u306a\u306e\u304c\u30d1\u30fc\u30c6\u30a3\u30b7\u30e7\u30f3\u8a2d\u8a08\u3067\u3059\u3002\u65e5\u4ed8\u3084\u5730\u57df\u306a\u3069\u3001\u5b9f\u969b\u306e\u691c\u7d22\u6761\u4ef6\u3068\u5408\u308f\u306a\u3044\u30d1\u30fc\u30c6\u30a3\u30b7\u30e7\u30f3\u3092\u5927\u91cf\u306b\u4f5c\u308b\u3068\u3001\u304b\u3048\u3063\u3066\u7ba1\u7406\u30b3\u30b9\u30c8\u304c\u5897\u3048\u307e\u3059\u3002\u9ad8\u30ab\u30fc\u30c7\u30a3\u30ca\u30ea\u30c6\u30a3\u306e\u5217\u3092\u7121\u6761\u4ef6\u306b\u30d1\u30fc\u30c6\u30a3\u30b7\u30e7\u30f3\u30ad\u30fc\u306b\u3059\u308b\u306e\u3082\u907f\u3051\u308b\u3079\u304d\u3067\u3059\u3002<\/p>\n

Spark 4.2\u306e\u6027\u80fd\u30c1\u30e5\u30fc\u30cb\u30f3\u30b0\u30ac\u30a4\u30c9\u3067\u3082\u3001\u30ad\u30e3\u30c3\u30b7\u30e5\u3001\u30d1\u30fc\u30c6\u30a3\u30b7\u30e7\u30f3\u3001\u7d71\u8a08\u3001\u7d50\u5408\u65b9\u5f0f\u3001AQE\u304c\u4e3b\u8981\u306a\u30c1\u30e5\u30fc\u30cb\u30f3\u30b0\u9818\u57df\u3068\u3057\u3066\u6574\u7406\u3055\u308c\u3066\u3044\u307e\u3059\u3002<\/p>\n

<\/p>\n

\u300c\u7d71\u8a08\u3092\u4e0e\u3048\u308b\u300d\u3068\u3044\u3046\u767a\u60f3<\/h2>\n

Spark\u306eOptimizer\u306f\u3001\u30c6\u30fc\u30d6\u30eb\u3084\u5217\u306e\u898f\u6a21\u3092\u77e5\u3089\u306a\u3051\u308c\u3070\u6700\u9069\u306a\u5b9f\u884c\u8a08\u753b\u3092\u9078\u3073\u306b\u304f\u304f\u306a\u308a\u307e\u3059\u3002<\/p>\n

\u305d\u306e\u305f\u3081\u5927\u898f\u6a21\u306a\u30c7\u30fc\u30bf\u57fa\u76e4\u3067\u306f\u3001\u7d71\u8a08\u60c5\u5831\u3092\u9069\u5207\u306b\u7ba1\u7406\u3059\u308b\u3053\u3068\u304c\u91cd\u8981\u3067\u3059\u3002EXPLAIN COST<\/code>\u306a\u3069\u3092\u5229\u7528\u3059\u308c\u3070\u3001Optimizer\u304c\u3069\u306e\u3088\u3046\u306a\u30b3\u30b9\u30c8\u898b\u7a4d\u3082\u308a\u3092\u884c\u3063\u3066\u3044\u308b\u304b\u3092\u78ba\u8a8d\u3067\u304d\u307e\u3059\u3002\u307e\u305f\u3001\u5b9f\u884c\u5f8c\u306b\u306fSpark UI\u3067runtime statistics\u3092\u78ba\u8a8d\u3067\u304d\u307e\u3059\u3002<\/p>\n

\u7279\u306bJOIN\u3067\u306f\u3001\u5de8\u5927\u30c6\u30fc\u30d6\u30eb\u540c\u58eb\u3092\u7121\u6761\u4ef6\u306b\u30b7\u30e3\u30c3\u30d5\u30eb\u3055\u305b\u308b\u306e\u3067\u306f\u306a\u304f\u3001\u5c0f\u3055\u306a\u30c6\u30fc\u30d6\u30eb\u3092Broadcast\u3067\u304d\u306a\u3044\u304b\u3001\u5b9f\u884c\u6642\u306bAQE\u304c\u3069\u306e\u7d50\u5408\u65b9\u5f0f\u3078\u5909\u66f4\u3057\u305f\u304b\u3092\u78ba\u8a8d\u3057\u307e\u3059\u3002<\/p>\n

\u91cd\u8981\u306a\u306e\u306f\u300cBROADCAST\u3092\u8a2d\u5b9a\u3059\u308c\u3070\u901f\u3044\u300d\u3068\u3044\u3046\u5358\u7d14\u306a\u8a71\u3067\u306f\u3042\u308a\u307e\u305b\u3093\u3002\u30c7\u30fc\u30bf\u30b5\u30a4\u30ba\u3001\u7d71\u8a08\u3001\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u3001\u30b9\u30ad\u30e5\u30fc\u3092\u542b\u3081\u3066\u5b9f\u884c\u8a08\u753b\u3092\u898b\u308b<\/strong>\u3053\u3068\u304c\u672c\u8cea\u3067\u3059\u3002<\/p>\n

<\/p>\n

Unity Catalog\u6642\u4ee3\u306eSparkSQL<\/h2>\n

Databricks\u74b0\u5883\u3067\u306f\u3001SQL\u306e\u6027\u80fd\u3060\u3051\u3067\u306a\u304f\u300c\u8ab0\u304c\u3069\u306e\u30c7\u30fc\u30bf\u3078\u30a2\u30af\u30bb\u30b9\u3067\u304d\u308b\u304b\u300d\u307e\u3067\u8a2d\u8a08\u5bfe\u8c61\u306b\u306a\u308a\u307e\u3059\u3002<\/p>\n

\u73fe\u5728\u306eUnity Catalog\u306f\u3001\u30c7\u30fc\u30bf\u3068AI\u8cc7\u7523\u306e\u7d71\u5408\u30ac\u30d0\u30ca\u30f3\u30b9\u5c64\u3068\u3057\u3066\u4f4d\u7f6e\u3065\u3051\u3089\u308c\u3001\u30a2\u30af\u30bb\u30b9\u5236\u5fa1\u3001\u30ea\u30cd\u30fc\u30b8\u3001\u76e3\u67fb\u3001\u30c7\u30fc\u30bf\u767a\u898b\u3001\u54c1\u8cea\u76e3\u8996\u306a\u3069\u3092\u63d0\u4f9b\u3057\u3066\u3044\u307e\u3059\u3002\u30c6\u30fc\u30d6\u30eb\u3084\u30d3\u30e5\u30fc\u306a\u3069\u306fcatalog.schema.object<\/code>\u3068\u3044\u30463\u968e\u5c64\u306e\u540d\u524d\u7a7a\u9593\u3067\u7ba1\u7406\u3067\u304d\u307e\u3059\u3002<\/p>\n

\u3055\u3089\u306b\u73fe\u5728\u306f\u3001\u5f93\u6765\u306e\u5358\u7d14\u306aGRANT\u3060\u3051\u3067\u306a\u304f\u3001\u5c5e\u6027\u30d9\u30fc\u30b9\u306e\u30a2\u30af\u30bb\u30b9\u5236\u5fa1\u3084\u884c\u30d5\u30a3\u30eb\u30bf\u3001\u5217\u30de\u30b9\u30af\u306a\u3069\u3001\u3088\u308a\u7d30\u7c92\u5ea6\u306e\u5236\u5fa1\u3082\u5229\u7528\u3067\u304d\u307e\u3059\u3002<\/p>\n

\u3057\u305f\u304c\u3063\u3066\u3001\u4f01\u696d\u306eSparkSQL\u8a2d\u8a08\u3067\u306f\u3001<\/p>\n

SQL\u3092\u66f8\u304f \u2192 \u30c6\u30fc\u30d6\u30eb\u3092\u4f5c\u308b \u2192 \u6a29\u9650\u3092\u8a2d\u5b9a\u3059\u308b<\/strong><\/p>\n

\u3068\u3044\u3046\u500b\u5225\u4f5c\u696d\u3067\u306f\u306a\u304f\u3001<\/p>\n

\u30c7\u30fc\u30bf\u3092\u767a\u898b\u3059\u308b \u2192 \u6240\u6709\u8005\u3092\u5b9a\u7fa9\u3059\u308b \u2192 \u6a29\u9650\u3092\u8a2d\u8a08\u3059\u308b \u2192 SQL\u3067\u5229\u7528\u3059\u308b \u2192 \u30ea\u30cd\u30fc\u30b8\u3068\u76e3\u67fb\u3067\u8ffd\u8de1\u3059\u308b<\/strong><\/p>\n

\u3068\u3044\u3046\u4e00\u9023\u306e\u30e9\u30a4\u30d5\u30b5\u30a4\u30af\u30eb\u3067\u8003\u3048\u308b\u3053\u3068\u304c\u91cd\u8981\u306b\u306a\u3063\u3066\u3044\u307e\u3059\u3002<\/p>\n

<\/p>\n

\u30b9\u30c8\u30ea\u30fc\u30df\u30f3\u30b0\u3068AI\u6642\u4ee3\u306eSparkSQL<\/h2>\n

SparkSQL\u306e\u9069\u7528\u7bc4\u56f2\u306f\u3001\u30d0\u30c3\u30c1ETL\u3060\u3051\u3067\u306f\u3042\u308a\u307e\u305b\u3093\u3002<\/p>\n

Structured Streaming\u306b\u3088\u3063\u3066\u3001SQL\u3084DataFrame\u3092\u5229\u7528\u3057\u305f\u7d99\u7d9a\u7684\u306a\u30c7\u30fc\u30bf\u51e6\u7406\u304c\u53ef\u80fd\u3067\u3059\u3002\u3055\u3089\u306bSpark 4.1\u3067\u306fReal-Time Mode\u304c\u6b63\u5f0f\u306b\u30b5\u30dd\u30fc\u30c8\u3055\u308c\u3001\u30b9\u30c6\u30fc\u30c8\u30ec\u30b9\u51e6\u7406\u3067\u306f\u30b5\u30d6\u79d2\u3001\u6761\u4ef6\u306b\u3088\u3063\u3066\u306f\u4e00\u6841\u30df\u30ea\u79d2\u30ec\u30d9\u30eb\u306e\u4f4e\u30ec\u30a4\u30c6\u30f3\u30b7\u51e6\u7406\u3092\u76ee\u6307\u305b\u308b\u3088\u3046\u306b\u306a\u308a\u307e\u3057\u305f\u3002<\/p>\n

\u307e\u305f\u3001\u6a5f\u68b0\u5b66\u7fd2\u3067\u306fSQL\u306b\u3088\u308b\u7279\u5fb4\u91cf\u751f\u6210\u3001Python\u306b\u3088\u308b\u30e2\u30c7\u30eb\u51e6\u7406\u3001\u30b9\u30c8\u30ea\u30fc\u30df\u30f3\u30b0\u306b\u3088\u308b\u7279\u5fb4\u91cf\u66f4\u65b0\u3092\u540c\u4e00\u57fa\u76e4\u3067\u7d44\u307f\u5408\u308f\u305b\u308b\u69cb\u6210\u3082\u8003\u3048\u3089\u308c\u307e\u3059\u3002<\/p>\n

\u751f\u6210AI\u306e\u666e\u53ca\u306b\u3088\u3063\u3066\u30c7\u30fc\u30bf\u51e6\u7406\u57fa\u76e4\u306b\u3082\u30ea\u30a2\u30eb\u30bf\u30a4\u30e0\u6027\u3068\u67d4\u8edf\u6027\u304c\u6c42\u3081\u3089\u308c\u308b\u73fe\u5728\u3001SparkSQL\u306f\u300c\u5206\u6790\u8005\u304c\u4f7f\u3046SQL\u30a8\u30f3\u30b8\u30f3\u300d\u304b\u3089\u3001AI\u30fbBI\u30fbETL\u30fb\u30b9\u30c8\u30ea\u30fc\u30df\u30f3\u30b0\u3092\u652f\u3048\u308b\u5171\u901a\u30c7\u30fc\u30bf\u51e6\u7406\u5c64<\/strong>\u3078\u5f79\u5272\u3092\u5e83\u3052\u3066\u3044\u307e\u3059\u3002<\/p>\n

\u3088\u304f\u3042\u308b\u30a2\u30f3\u30c1\u30d1\u30bf\u30fc\u30f3
<\/h2>\n

\u4ee3\u8868\u7684\u306a\u306e\u304c\u300c\u3068\u308a\u3042\u3048\u305acache\u3059\u308b\u300d\u8a2d\u8a08\u3067\u3059\u3002\u30ad\u30e3\u30c3\u30b7\u30e5\u306f\u518d\u5229\u7528\u3055\u308c\u308b\u30c7\u30fc\u30bf\u306b\u306f\u6709\u52b9\u3067\u3059\u304c\u3001\u5de8\u5927\u306a\u30c7\u30fc\u30bf\u3092\u4e00\u5ea6\u3057\u304b\u4f7f\u308f\u306a\u3044\u51e6\u7406\u3067\u306f\u30e1\u30e2\u30ea\u3092\u5727\u8feb\u3057\u3001\u9006\u52b9\u679c\u306b\u306a\u308b\u3053\u3068\u304c\u3042\u308a\u307e\u3059\u3002Spark SQL\u306e\u30ad\u30e3\u30c3\u30b7\u30e5\u306f\u5217\u6307\u5411\u5f62\u5f0f\u3067\u4fdd\u6301\u3055\u308c\u307e\u3059\u304c\u3001\u30ef\u30fc\u30af\u30ed\u30fc\u30c9\u306b\u5fdc\u3058\u3066\u5229\u7528\u3059\u308b\u5fc5\u8981\u304c\u3042\u308a\u307e\u3059\u3002<\/p>\n

\u6b21\u306b\u3001\u5c0f\u30d5\u30a1\u30a4\u30eb\u554f\u984c\u304c\u3042\u308a\u307e\u3059\u3002\u5927\u91cf\u306e\u5c0f\u3055\u306a\u30d5\u30a1\u30a4\u30eb\u306f\u30d5\u30a1\u30a4\u30eb\u4e00\u89a7\u51e6\u7406\u3084\u30bf\u30b9\u30af\u8d77\u52d5\u3001\u30e1\u30bf\u30c7\u30fc\u30bf\u51e6\u7406\u306e\u30aa\u30fc\u30d0\u30fc\u30d8\u30c3\u30c9\u3092\u5897\u3084\u3057\u307e\u3059\u3002\u51fa\u529b\u30d1\u30fc\u30c6\u30a3\u30b7\u30e7\u30f3\u6570\u3084\u66f8\u304d\u8fbc\u307f\u65b9\u5f0f\u3092\u898b\u76f4\u3057\u3001\u5fc5\u8981\u306b\u5fdc\u3058\u3066COALESCE<\/code>\u3084REPARTITION<\/code>\u306a\u3069\u3092\u4f7f\u3044\u307e\u3059\u3002<\/p>\n

\u305d\u3057\u3066\u6700\u5927\u306e\u30a2\u30f3\u30c1\u30d1\u30bf\u30fc\u30f3\u306f\u3001\u8a2d\u5b9a\u5024\u3060\u3051\u3092\u5909\u66f4\u3057\u3066\u6027\u80fd\u6539\u5584\u3057\u305f\u3064\u3082\u308a\u306b\u306a\u308b\u3053\u3068<\/strong>\u3067\u3059\u3002<\/p>\n

\u6027\u80fd\u554f\u984c\u304c\u8d77\u304d\u305f\u3089\u3001<\/p>\n

\u300c\u3069\u306e\u30c7\u30fc\u30bf\u3092\u8aad\u3093\u3067\u3044\u308b\u304b\u300d<\/p>\n

\u300c\u3069\u306eJOIN\u304c\u30b7\u30e3\u30c3\u30d5\u30eb\u3092\u767a\u751f\u3055\u305b\u3066\u3044\u308b\u304b\u300d<\/p>\n

\u300c\u30d1\u30fc\u30c6\u30a3\u30b7\u30e7\u30f3\u306b\u504f\u308a\u304c\u306a\u3044\u304b\u300d<\/p>\n

\u300cAQE\u304c\u3069\u3046\u5224\u65ad\u3057\u305f\u304b\u300d<\/p>\n

\u300c\u5b9f\u969b\u306e\u5b9f\u884c\u6642\u9593\u306f\u3069\u306eStage\u306b\u6d88\u8cbb\u3055\u308c\u3066\u3044\u308b\u304b\u300d<\/p>\n

\u3092\u9806\u756a\u306b\u78ba\u8a8d\u3057\u307e\u3059\u3002<\/p>\n

\u307e\u3068\u3081\uff1a2026\u5e74\u306eSparkSQL\u3092\u4f7f\u3044\u3053\u306a\u3059\u305f\u3081\u306e\u8003\u3048\u65b9<\/h2>\n

\u3053\u308c\u304b\u3089\u306eSparkSQL\u3067\u91cd\u8981\u306a\u306e\u306f\u3001SQL\u6587\u6cd5\u306e\u6697\u8a18\u3067\u306f\u3042\u308a\u307e\u305b\u3093\u3002<\/p>\n

\u7b2c\u4e00\u306b\u3001\u5ba3\u8a00\u7684\u306b\u66f8\u3044\u3066Optimizer\u3078\u4ed5\u4e8b\u3092\u6e21\u3059\u3053\u3068<\/strong>\u3002\u7b2c\u4e8c\u306b\u3001EXPLAIN\u3068Spark UI\u3092\u4f7f\u3063\u3066\u5b9f\u884c\u8a08\u753b\u3092\u8aad\u3080\u3053\u3068<\/strong>\u3002\u7b2c\u4e09\u306b\u3001\u30c7\u30fc\u30bf\u914d\u7f6e\u3001\u7d71\u8a08\u3001\u30d1\u30fc\u30c6\u30a3\u30b7\u30e7\u30f3\u3001JOIN\u3092\u7269\u7406\u8a2d\u8a08\u3068\u3057\u3066\u8003\u3048\u308b\u3053\u3068<\/strong>\u3002\u305d\u3057\u3066\u7b2c\u56db\u306b\u3001Unity Catalog\u306a\u3069\u306e\u30ac\u30d0\u30ca\u30f3\u30b9\u307e\u3067\u542b\u3081\u3066\u30c7\u30fc\u30bf\u57fa\u76e4\u3092\u8a2d\u8a08\u3059\u308b\u3053\u3068<\/strong>\u3067\u3059\u3002<\/p>\n

Spark 4.x\u3067\u306fANSI SQL\u3001SQL Scripting\u3001VARIANT\u3001Spark Connect\u3001Declarative Pipelines\u3001Real-Time Mode\u306a\u3069\u3001SparkSQL\u3092\u53d6\u308a\u5dfb\u304f\u6a5f\u80fd\u304c\u6025\u901f\u306b\u5e83\u304c\u3063\u3066\u3044\u307e\u3059\u3002<\/p>\n

\u3057\u304b\u3057\u3001\u57fa\u790e\u304c\u4e0d\u8981\u306b\u306a\u3063\u305f\u308f\u3051\u3067\u306f\u3042\u308a\u307e\u305b\u3093\u3002\u3080\u3057\u308d\u6a5f\u80fd\u304c\u5897\u3048\u305f\u304b\u3089\u3053\u305d\u3001**\u300c\u4f55\u3092Spark\u306b\u4efb\u305b\u3001\u4f55\u3092\u30c7\u30fc\u30bf\u8a2d\u8a08\u5074\u3067\u89e3\u6c7a\u3059\u308b\u304b\u300d**\u3068\u3044\u3046\u5883\u754c\u3092\u7406\u89e3\u3059\u308b\u3053\u3068\u304c\u91cd\u8981\u306b\u306a\u3063\u3066\u3044\u307e\u3059\u3002<\/p>\n

SparkSQL\u3092\u4f7f\u3044\u3053\u306a\u3059\u3068\u306f\u3001SQL\u3092\u901f\u304f\u66f8\u304f\u3053\u3068\u3067\u306f\u3042\u308a\u307e\u305b\u3093\u3002<\/p>\n

SQL\u3068\u3044\u3046\u5ba3\u8a00\u304b\u3089\u3001Spark\u304c\u3069\u306e\u3088\u3046\u306a\u8a08\u753b\u3092\u4f5c\u308a\u3001\u3069\u306e\u30c7\u30fc\u30bf\u3092\u8aad\u307f\u3001\u3069\u3053\u3067\u30b7\u30e3\u30c3\u30d5\u30eb\u3057\u3001\u3069\u306e\u3088\u3046\u306b\u5b9f\u884c\u3059\u308b\u306e\u304b\u3092\u8aac\u660e\u3067\u304d\u308b\u3053\u3068\u3002<\/strong><\/p>\n

\u305d\u3053\u307e\u3067\u7406\u89e3\u3067\u304d\u308c\u3070\u3001\u30c7\u30fc\u30bf\u91cf\u304c\u5897\u3048\u305f\u308a\u3001\u30d0\u30c3\u30c1\u304b\u3089\u30b9\u30c8\u30ea\u30fc\u30df\u30f3\u30b0\u3078\u79fb\u884c\u3057\u305f\u308a\u3001AI\u3084BI\u3068\u7d71\u5408\u3057\u305f\u308a\u3057\u3066\u3082\u3001\u5834\u5f53\u305f\u308a\u7684\u306a\u8a2d\u5b9a\u5909\u66f4\u3067\u306f\u306a\u304f\u3001\u8a2d\u8a08\u3068\u3057\u3066\u6027\u80fd\u3068\u4fe1\u983c\u6027\u3092\u6539\u5584\u3067\u304d\u308b\u3088\u3046\u306b\u306a\u308a\u307e\u3059\u3002<\/p>\n

SparkSQL\u306e\u73fe\u5728\u5730\u306f\u3001\u300cSQL\u3067\u30d3\u30c3\u30b0\u30c7\u30fc\u30bf\u3092\u51e6\u7406\u3059\u308b\u6280\u8853\u300d\u304b\u3089\u3001\u5ba3\u8a00\u7684\u306a\u30c7\u30fc\u30bf\u51e6\u7406\u3001\u9069\u5fdc\u7684\u306a\u6700\u9069\u5316\u3001\u30ea\u30a2\u30eb\u30bf\u30a4\u30e0\u51e6\u7406\u3001\u305d\u3057\u3066\u30ac\u30d0\u30ca\u30f3\u30b9\u3092\u4e00\u4f53\u5316\u3059\u308b\u30c7\u30fc\u30bf\u57fa\u76e4\u6280\u8853<\/strong>\u3078\u9032\u5316\u3057\u305f\u3068\u3053\u308d\u306b\u3042\u308a\u307e\u3059\u3002<\/p>\n

<\/p>","protected":false},"excerpt":{"rendered":"

SparkSQL\u306e\u57fa\u672c\u6982\u5ff5\u304b\u3089Catalyst\u3001AQE\u3001Spark Connect\u3001ANSI SQL\u3001Unity Catalog\u307e\u3067\u30012026\u5e74\u306eSpark 4.2\u6642\u4ee3\u306b\u62bc\u3055\u3048\u308b\u3079\u304d\u8a2d\u8a08\u601d\u60f3\u3068\u5b9f\u8df5\u7684\u306a\u30c1\u30e5\u30fc\u30cb\u30f3\u30b0\u65b9\u6cd5\u3092\u89e3 […]<\/p>\n","protected":false},"author":22,"featured_media":7204,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","swell_btn_cv_data":"","footnotes":"","_wp_rev_ctl_limit":""},"categories":[1249],"tags":[1150,1149,1151,1148,811],"class_list":["post-7200","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-knowledge","tag-ansi","tag-catalyst","tag-sparkconnect","tag-sparksql","tag-811"],"_links":{"self":[{"href":"https:\/\/since2020.jp\/media\/wp-json\/wp\/v2\/posts\/7200","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/since2020.jp\/media\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/since2020.jp\/media\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/since2020.jp\/media\/wp-json\/wp\/v2\/users\/22"}],"replies":[{"embeddable":true,"href":"https:\/\/since2020.jp\/media\/wp-json\/wp\/v2\/comments?post=7200"}],"version-history":[{"count":1,"href":"https:\/\/since2020.jp\/media\/wp-json\/wp\/v2\/posts\/7200\/revisions"}],"predecessor-version":[{"id":9825,"href":"https:\/\/since2020.jp\/media\/wp-json\/wp\/v2\/posts\/7200\/revisions\/9825"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/since2020.jp\/media\/wp-json\/wp\/v2\/media\/7204"}],"wp:attachment":[{"href":"https:\/\/since2020.jp\/media\/wp-json\/wp\/v2\/media?parent=7200"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/since2020.jp\/media\/wp-json\/wp\/v2\/categories?post=7200"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/since2020.jp\/media\/wp-json\/wp\/v2\/tags?post=7200"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}