{"id":7205,"date":"2025-09-30T15:38:43","date_gmt":"2025-09-30T06:38:43","guid":{"rendered":"https:\/\/blog.since2020.jp\/?p=7205"},"modified":"2026-09-29T17:31:16","modified_gmt":"2026-09-29T08:31:16","slug":"pyspark_glossary","status":"publish","type":"post","link":"https:\/\/since2020.jp\/media\/pyspark_glossary\/","title":{"rendered":"PySpark\u5b8c\u5168\u30ed\u30fc\u30c9\u30de\u30c3\u30d7\u2500\u2500pandas\u30e6\u30fc\u30b6\u30fc\u304b\u3089\u672c\u756a\u904b\u7528\u3001\u30b9\u30c8\u30ea\u30fc\u30df\u30f3\u30b0\u307e\u3067"},"content":{"rendered":"\n<p>PySpark\u306e\u57fa\u790e\u304b\u3089DataFrame\u3001pandas API on Spark\u3001ETL\u3001\u6a5f\u68b0\u5b66\u7fd2\u524d\u51e6\u7406\u3001Structured Streaming\u3001\u672c\u756a\u904b\u7528\u307e\u3067\u3092\u4f53\u7cfb\u7684\u306b\u89e3\u8aac\u3002Spark 4.x\u306e\u6700\u65b0\u6a5f\u80fd\u3084VARIANT\u3001Python UDF\u30fbUDTF\u3001Kubernetes\u3001Delta Lake\u306e\u9032\u5316\u3092\u8e0f\u307e\u3048\u3001pandas\u30e6\u30fc\u30b6\u30fc\u304c\u5206\u6563\u51e6\u7406\u3092\u5b9f\u52d9\u3078\u5c0e\u5165\u3059\u308b\u305f\u3081\u306e\u8a2d\u8a08\u539f\u5247\u3068\u30ed\u30fc\u30c9\u30de\u30c3\u30d7\u3092\u7d39\u4ecb\u3057\u307e\u3059\u3002<\/p>\n\n\n<h2>\u306a\u305c\u4eca\u3001PySpark\u3092\u5b66\u3076\u306e\u304b<!-- notionvc: 8d787440-769c-4818-83e4-c434a7d07a5f --><\/h2>\n<p>PySpark\u306f\u3001Apache Spark\u3092Python\u304b\u3089\u5229\u7528\u3059\u308b\u305f\u3081\u306eAPI\u3067\u3059\u3002\u5927\u91cf\u30c7\u30fc\u30bf\u3092\u8907\u6570\u306e\u30b3\u30f3\u30d4\u30e5\u30fc\u30bf\u30fc\u306b\u5206\u6563\u3057\u3066\u51e6\u7406\u3067\u304d\u308b\u305f\u3081\u3001\u30ed\u30b0\u5206\u6790\u3001ETL\u3001\u30c7\u30fc\u30bf\u30ec\u30a4\u30af\u30cf\u30a6\u30b9\u3001\u6a5f\u68b0\u5b66\u7fd2\u306e\u524d\u51e6\u7406\u3001\u30b9\u30c8\u30ea\u30fc\u30df\u30f3\u30b0\u51e6\u7406\u306a\u3069\u3067\u5229\u7528\u3055\u308c\u3066\u3044\u307e\u3059\u3002<\/p>\n<p>\u73fe\u5728\u306ePySpark\u3092\u7406\u89e3\u3059\u308b\u3046\u3048\u3067\u91cd\u8981\u306a\u306e\u304c\u3001Apache Spark 4.x\u3078\u306e\u79fb\u884c\u3067\u3059\u3002Spark 4.0\u3067\u306fPython Data Source API\u3001Python UDTF\u3001VARIANT\u578b\u3001Structured Streaming\u306e\u72b6\u614b\u7ba1\u7406\u6a5f\u80fd\u306a\u3069\u304c\u8ffd\u52a0\u3055\u308c\u3001PySpark\u306f\u5358\u306a\u308b\u300cSpark\u3092Python\u304b\u3089\u64cd\u4f5c\u3059\u308b\u305f\u3081\u306e\u8584\u3044API\u300d\u304b\u3089\u3001Python\u4e2d\u5fc3\u306e\u30c7\u30fc\u30bf\u30a8\u30f3\u30b8\u30cb\u30a2\u30ea\u30f3\u30b0\u57fa\u76e4\u3078\u3055\u3089\u306b\u9032\u5316\u3057\u307e\u3057\u305f\u3002<\/p>\n<p>\u3055\u3089\u306bSpark 4.1\u3067\u306f\u3001\u5ba3\u8a00\u7684\u306a\u30c7\u30fc\u30bf\u30d1\u30a4\u30d7\u30e9\u30a4\u30f3\u3001\u4f4e\u9045\u5ef6\u30b9\u30c8\u30ea\u30fc\u30df\u30f3\u30b0\u3001Arrow\u3092\u6d3b\u7528\u3057\u305fPython\u51e6\u7406\u306a\u3069\u304c\u5f37\u5316\u3055\u308c\u3066\u3044\u307e\u3059\u3002\u3064\u307e\u308a2026\u5e74\u73fe\u5728\u306ePySpark\u306f\u3001\u300c\u5927\u91cf\u30c7\u30fc\u30bf\u3092\u30d0\u30c3\u30c1\u51e6\u7406\u3059\u308b\u6280\u8853\u300d\u3060\u3051\u3067\u306f\u306a\u304f\u3001\u30d0\u30c3\u30c1\u304b\u3089\u30ea\u30a2\u30eb\u30bf\u30a4\u30e0\u307e\u3067\u3092\u540c\u3058\u8003\u3048\u65b9\u3067\u8a2d\u8a08\u3059\u308b\u305f\u3081\u306e\u7dcf\u5408\u7684\u306a\u30c7\u30fc\u30bf\u51e6\u7406\u57fa\u76e4\u306b\u306a\u3063\u3066\u3044\u307e\u3059\u3002<!-- notionvc: 4b83b64c-6d99-44a0-ab42-95bfc0472205 --><\/p>\n<p><!-- notionvc: d1219caf-489e-4ecc-8404-08d5f19c4d4f --><\/p>\n<h2>PySpark\u306e\u57fa\u672c\u306fDataFrame\u3068Spark SQL<!-- notionvc: ba2211d2-0856-4915-a64f-abfc30601e82 --><\/h2>\n<p>PySpark\u3092\u5b66\u3076\u3068\u304d\u3001\u6700\u521d\u304b\u3089RDD\u3092\u6df1\u304f\u7406\u89e3\u3059\u308b\u5fc5\u8981\u306f\u3042\u308a\u307e\u305b\u3093\u3002\u73fe\u5728\u306e\u5b9f\u52d9\u3067\u306f\u3001\u307e\u305aDataFrame\u3068Spark SQL\u3092\u7406\u89e3\u3059\u308b\u3053\u3068\u304c\u91cd\u8981\u3067\u3059\u3002<\/p>\n<p>DataFrame\u306f\u8868\u5f62\u5f0f\u306e\u30c7\u30fc\u30bf\u3092\u6271\u3046API\u3067\u3001<code>select<\/code>\u3001<code>filter<\/code>\u3001<code>groupBy<\/code>\u3001<code>join<\/code>\u3001<code>withColumn<\/code>\u306a\u3069\u3092\u7d44\u307f\u5408\u308f\u305b\u3066\u51e6\u7406\u3057\u307e\u3059\u3002SQL\u3092\u4f7f\u3048\u308b\u30c1\u30fc\u30e0\u3067\u3042\u308c\u3070\u3001Spark SQL\u3068DataFrame API\u3092\u4f7f\u3044\u5206\u3051\u306a\u304c\u3089\u540c\u3058\u51e6\u7406\u57fa\u76e4\u3092\u5171\u6709\u3067\u304d\u307e\u3059\u3002<\/p>\n<p>\u3053\u3053\u3067\u91cd\u8981\u306a\u306e\u306f\u3001PySpark\u306e\u30b3\u30fc\u30c9\u304c\u305d\u306e\u307e\u307e1\u884c\u305a\u3064Python\u3067\u5b9f\u884c\u3055\u308c\u308b\u308f\u3051\u3067\u306f\u306a\u3044\u3053\u3068\u3067\u3059\u3002Spark\u306f\u51e6\u7406\u5185\u5bb9\u3092\u8ad6\u7406\u30fb\u7269\u7406\u8a08\u753b\u3078\u5909\u63db\u3057\u3001Catalyst Optimizer\u3084Adaptive Query Execution\uff08AQE\uff09\u306a\u3069\u3092\u5229\u7528\u3057\u3066\u5b9f\u884c\u65b9\u6cd5\u3092\u6700\u9069\u5316\u3057\u307e\u3059\u3002<\/p>\n<p>\u305d\u306e\u305f\u3081\u3001PySpark\u3067\u306f\u300cPython\u3068\u3057\u3066\u77ed\u304f\u66f8\u3051\u308b\u304b\u300d\u3060\u3051\u3067\u306a\u304f\u3001\u300cSpark\u304c\u52b9\u7387\u7684\u306a\u5b9f\u884c\u8a08\u753b\u3092\u4f5c\u308c\u308b\u304b\u300d\u3068\u3044\u3046\u8996\u70b9\u304c\u91cd\u8981\u306b\u306a\u308a\u307e\u3059\u3002<!-- notionvc: c4fe5f9d-3921-4f3c-8000-d6b824679517 --><!-- notionvc: 4334161c-bfdc-471a-a485-5449610d03c8 --><\/p>\n<h2>pandas\u30e6\u30fc\u30b6\u30fc\u306a\u3089\u3001\u307e\u305apandas API on Spark<!-- notionvc: e0f47f2a-1a8b-47f2-a9ab-fbc4249a7324 --><\/h2>\n<p>pandas\u306b\u6163\u308c\u3066\u3044\u308b\u4eba\u304cPySpark\u3078\u79fb\u884c\u3059\u308b\u5834\u5408\u3001pandas API on Spark\u306f\u6709\u529b\u306a\u5165\u53e3\u3067\u3059\u3002<\/p>\n<p>pandas\u306b\u8fd1\u3044\u8a18\u6cd5\u3067\u5206\u6563\u30c7\u30fc\u30bf\u3092\u6271\u3048\u308b\u305f\u3081\u3001\u65e2\u5b58\u306e\u5206\u6790\u30b3\u30fc\u30c9\u3092\u6bb5\u968e\u7684\u306bSpark\u3078\u79fb\u884c\u3067\u304d\u307e\u3059\u3002\u305f\u3060\u3057\u3001pandas\u3068\u5b8c\u5168\u306b\u540c\u3058\u3067\u306f\u3042\u308a\u307e\u305b\u3093\u3002\u51e6\u7406\u306b\u3088\u3063\u3066\u306f\u30c7\u30fc\u30bf\u3092\u30c9\u30e9\u30a4\u30d0\u30fc\u3078\u96c6\u3081\u308b\u5fc5\u8981\u304c\u3042\u308a\u3001<code>to_numpy()<\/code>\u306a\u3069\u306f\u5927\u91cf\u30c7\u30fc\u30bf\u306b\u5bfe\u3057\u3066\u5b89\u6613\u306b\u4f7f\u7528\u3059\u308b\u3068\u30e1\u30e2\u30ea\u4e0d\u8db3\u306b\u3064\u306a\u304c\u308a\u307e\u3059\u3002<\/p>\n<p>\u3057\u305f\u304c\u3063\u3066\u3001\u300cpandas API on Spark\u306a\u3089pandas\u306e\u30b3\u30fc\u30c9\u3092\u305d\u306e\u307e\u307e\u5de8\u5927\u30c7\u30fc\u30bf\u3078\u9069\u7528\u3067\u304d\u308b\u300d\u3068\u8003\u3048\u308b\u306e\u3067\u306f\u306a\u304f\u3001pandas\u306e\u64cd\u4f5c\u611f\u3092\u7dad\u6301\u3057\u306a\u304c\u3089\u3001\u5206\u6563\u51e6\u7406\u306e\u8003\u3048\u65b9\u3092\u8eab\u306b\u3064\u3051\u308b\u305f\u3081\u306e\u6a4b\u6e21\u3057\u3068\u3057\u3066\u5229\u7528\u3059\u308b\u306e\u304c\u9069\u5207\u3067\u3059\u3002<\/p>\n<p>\u5b66\u7fd2\u9806\u5e8f\u3068\u3057\u3066\u306f\u3001pandas\u3067\u30c7\u30fc\u30bf\u64cd\u4f5c\u306e\u57fa\u672c\u3092\u78ba\u8a8d\u3057\u3001pandas API on Spark\u3067\u5206\u6563\u51e6\u7406\u3092\u4f53\u9a13\u3057\u3001\u305d\u306e\u5f8c\u306bPySpark DataFrame\u3068Spark SQL\u3078\u9032\u3080\u3068\u7406\u89e3\u3057\u3084\u3059\u304f\u306a\u308a\u307e\u3059\u3002<\/p>\n<h2>2026\u5e74\u7248\u3067\u91cd\u8981\u306aPySpark\u8a2d\u8a08\u539f\u5247<!-- notionvc: f13bf4ca-6748-4532-bceb-1b38fedc1864 --><\/h2>\n<p>\u6700\u521d\u306b\u899a\u3048\u3066\u304a\u304d\u305f\u3044\u539f\u5247\u306f\u3001\u300cPython\u3067\u5168\u90e8\u51e6\u7406\u3057\u306a\u3044\u300d\u3053\u3068\u3067\u3059\u3002<\/p>\n<p>Spark\u306b\u306f\u591a\u6570\u306e\u7d44\u307f\u8fbc\u307f\u95a2\u6570\u304c\u7528\u610f\u3055\u308c\u3066\u3044\u308b\u305f\u3081\u3001\u6587\u5b57\u5217\u51e6\u7406\u3001\u65e5\u4ed8\u51e6\u7406\u3001\u96c6\u8a08\u3001\u914d\u5217\u30fbMap\u64cd\u4f5c\u306a\u3069\u306f\u53ef\u80fd\u306a\u9650\u308aSpark SQL\u3084DataFrame\u306e\u7d44\u307f\u8fbc\u307f\u95a2\u6570\u3067\u5b9f\u88c5\u3057\u307e\u3059\u3002<\/p>\n<p>Python UDF\u306f\u4fbf\u5229\u3067\u3059\u304c\u3001Python\u3068Spark\u306e\u5b9f\u884c\u30a8\u30f3\u30b8\u30f3\u306e\u9593\u3067\u30c7\u30fc\u30bf\u3092\u53d7\u3051\u6e21\u3059\u30b3\u30b9\u30c8\u304c\u767a\u751f\u3057\u307e\u3059\u3002\u305d\u306e\u305f\u3081\u3001\u7d44\u307f\u8fbc\u307f\u95a2\u6570\u3067\u8868\u73fe\u3067\u304d\u308b\u51e6\u7406\u307e\u3067UDF\u306b\u3057\u3066\u3057\u307e\u3046\u3068\u3001\u6700\u9069\u5316\u306e\u4f59\u5730\u3092\u5931\u3046\u3053\u3068\u304c\u3042\u308a\u307e\u3059\u3002<\/p>\n<p>\u4e00\u65b9\u3001\u73fe\u5728\u306eSpark\u3067\u306fPython\u5074\u306e\u51e6\u7406\u3082\u9032\u5316\u3057\u3066\u3044\u307e\u3059\u3002Python UDTF\u3084Arrow\u3092\u5229\u7528\u3057\u305f\u51e6\u7406\u306a\u3069\u304c\u5f37\u5316\u3055\u308c\u3066\u304a\u308a\u3001\u300cUDF\u306f\u7d76\u5bfe\u306b\u4f7f\u3063\u3066\u306f\u3044\u3051\u306a\u3044\u300d\u3068\u3044\u3046\u8003\u3048\u65b9\u3082\u9069\u5207\u3067\u306f\u3042\u308a\u307e\u305b\u3093\u3002\u91cd\u8981\u306a\u306e\u306f\u3001\u30cd\u30a4\u30c6\u30a3\u30d6\u95a2\u6570\u3001SQL\u3001Arrow\u7cfbAPI\u3001Python UDF\u306e\u3069\u308c\u3092\u9078\u3076\u3068\u51e6\u7406\u306e\u610f\u5473\u3068\u6027\u80fd\u3092\u4e21\u7acb\u3067\u304d\u308b\u304b\u3092\u5224\u65ad\u3059\u308b\u3053\u3068\u3067\u3059\u3002<\/p>\n<p>\u3082\u3046\u4e00\u3064\u91cd\u8981\u306a\u306e\u304cJOIN\u3067\u3059\u3002\u5927\u898f\u6a21\u30c7\u30fc\u30bf\u540c\u58eb\u306eJOIN\u3067\u306f\u30b7\u30e3\u30c3\u30d5\u30eb\u304c\u767a\u751f\u3057\u3001\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u8ee2\u9001\u3084\u30c7\u30a3\u30b9\u30afI\/O\u304c\u30dc\u30c8\u30eb\u30cd\u30c3\u30af\u306b\u306a\u308a\u307e\u3059\u3002\u5c0f\u3055\u3044\u30de\u30b9\u30bf\u30fc\u30c6\u30fc\u30d6\u30eb\u306a\u3089\u30d6\u30ed\u30fc\u30c9\u30ad\u30e3\u30b9\u30c8JOIN\u3092\u691c\u8a0e\u3057\u3001\u5de8\u5927\u30c6\u30fc\u30d6\u30eb\u3067\u306f\u7d50\u5408\u30ad\u30fc\u306e\u504f\u308a\u3001\u3064\u307e\u308a\u30c7\u30fc\u30bf\u30b9\u30ad\u30e5\u30fc\u3092\u78ba\u8a8d\u3057\u307e\u3059\u3002<\/p>\n<p>\u300cJOIN\u304c\u9045\u3044\u304b\u3089\u30af\u30e9\u30b9\u30bf\u3092\u5927\u304d\u304f\u3059\u308b\u300d\u3068\u3044\u3046\u5bfe\u51e6\u3060\u3051\u3067\u306f\u306a\u304f\u3001JOIN\u524d\u306b\u30d5\u30a3\u30eb\u30bf\u3059\u308b\u3001\u4e8b\u524d\u96c6\u7d04\u3059\u308b\u3001\u4e0d\u8981\u306a\u5217\u3092\u524a\u308b\u3068\u3044\u3063\u305f\u30c7\u30fc\u30bf\u51e6\u7406\u305d\u306e\u3082\u306e\u306e\u8a2d\u8a08\u3092\u898b\u76f4\u3059\u3053\u3068\u304c\u91cd\u8981\u3067\u3059\u3002<!-- notionvc: 23cc9524-a83f-486c-a4a7-277de932ca1b --><\/p>\n<p><!-- notionvc: 0bcffc92-b9e3-4c4d-8e5c-a02b2b620ddb --><\/p>\n<h2>ETL\u3068\u30ec\u30a4\u30af\u30cf\u30a6\u30b9\u3067\u306f\u300c\u30c7\u30fc\u30bf\u306e\u7f6e\u304d\u65b9\u300d\u304c\u6027\u80fd\u3092\u6c7a\u3081\u308b<!-- notionvc: 20c61574-1f0b-4f26-b429-0f30d8bd81be --><\/h2>\n<p>PySpark\u3092\u672c\u756a\u5229\u7528\u3059\u308b\u5834\u5408\u3001\u30b3\u30fc\u30c9\u3060\u3051\u3067\u306a\u304f\u30c7\u30fc\u30bf\u306e\u7269\u7406\u8a2d\u8a08\u304c\u91cd\u8981\u306b\u306a\u308a\u307e\u3059\u3002<\/p>\n<p>\u7279\u306b\u554f\u984c\u306b\u306a\u308a\u3084\u3059\u3044\u306e\u304c\u3001\u5c0f\u3055\u306a\u30d5\u30a1\u30a4\u30eb\u306e\u5927\u91cf\u767a\u751f\u3067\u3059\u3002\u6570KB\u304b\u3089\u6570MB\u7a0b\u5ea6\u306e\u30d5\u30a1\u30a4\u30eb\u304c\u5927\u91cf\u306b\u5b58\u5728\u3059\u308b\u3068\u3001\u30d5\u30a1\u30a4\u30eb\u5217\u6319\u3084\u8aad\u307f\u8fbc\u307f\u306e\u30aa\u30fc\u30d0\u30fc\u30d8\u30c3\u30c9\u304c\u5897\u3048\u3001\u51e6\u7406\u6027\u80fd\u304c\u4f4e\u4e0b\u3057\u307e\u3059\u3002<\/p>\n<p>\u305d\u306e\u305f\u3081\u3001<code>repartition<\/code>\u3084<code>coalesce<\/code>\u3092\u9069\u5207\u306b\u5229\u7528\u3057\u3001\u51fa\u529b\u30d5\u30a1\u30a4\u30eb\u6570\u3068\u30b5\u30a4\u30ba\u3092\u610f\u8b58\u3057\u307e\u3059\u3002\u30d1\u30fc\u30c6\u30a3\u30b7\u30e7\u30f3\u3092\u5897\u3084\u305b\u3070\u5fc5\u305a\u901f\u304f\u306a\u308b\u308f\u3051\u3067\u306f\u306a\u304f\u3001\u5c11\u306a\u3059\u304e\u3066\u3082\u591a\u3059\u304e\u3066\u3082\u554f\u984c\u306b\u306a\u308a\u307e\u3059\u3002<\/p>\n<p>\u307e\u305f\u3001\u30c7\u30fc\u30bf\u30ec\u30a4\u30af\u30cf\u30a6\u30b9\u3067\u306fParquet\u306a\u3069\u306e\u5217\u6307\u5411\u30d5\u30a9\u30fc\u30de\u30c3\u30c8\u306b\u52a0\u3048\u3001Delta Lake\u306e\u3088\u3046\u306a\u30c6\u30fc\u30d6\u30eb\u5f62\u5f0f\u3092\u7d44\u307f\u5408\u308f\u305b\u308b\u69cb\u6210\u304c\u4e00\u822c\u7684\u3067\u3059\u3002<\/p>\n<p>Delta Lake\u306f\u73fe\u57284.x\u7cfb\u307e\u3067\u9032\u5316\u3057\u3066\u304a\u308a\u3001UniForm\u306b\u3088\u3063\u3066Iceberg\u3084Hudi\u30af\u30e9\u30a4\u30a2\u30f3\u30c8\u3068\u306e\u76f8\u4e92\u904b\u7528\u6027\u3092\u610f\u8b58\u3057\u305f\u8a2d\u8a08\u3082\u53ef\u80fd\u3067\u3059\u30022026\u5e74\u306b\u306fDelta Lake 4.2\u3082\u767b\u5834\u3057\u3066\u304a\u308a\u3001\u30ab\u30bf\u30ed\u30b0\u7ba1\u7406\u578b\u30c6\u30fc\u30d6\u30eb\u3084\u30b9\u30c8\u30ea\u30fc\u30df\u30f3\u30b0\u3001Kernel\u306a\u3069\u306e\u9818\u57df\u3078\u6a5f\u80fd\u304c\u62e1\u5f35\u3055\u308c\u3066\u3044\u307e\u3059\u3002<\/p>\n<p>\u3064\u307e\u308a\u3001\u300cSpark\u3067\u51e6\u7406\u3059\u308b\u300d\u3060\u3051\u3067\u306f\u306a\u304f\u3001\u300c\u3069\u306e\u5f62\u5f0f\u3067\u4fdd\u5b58\u3057\u3001\u3069\u306e\u30a8\u30f3\u30b8\u30f3\u304b\u3089\u8aad\u3080\u306e\u304b\u300d\u307e\u3067\u542b\u3081\u3066\u30c7\u30fc\u30bf\u57fa\u76e4\u3092\u8a2d\u8a08\u3059\u308b\u3053\u3068\u304c\u91cd\u8981\u3067\u3059\u3002<!-- notionvc: 8cbd5425-748a-4d8a-89ec-56a81402e8e1 --><\/p>\n<p><!-- notionvc: 755a1cb4-0068-4567-8508-f93b379d252d --><\/p>\n<h2>\u6a5f\u68b0\u5b66\u7fd2\u3067\u306f\u300cSpark\u3067\u3059\u3079\u3066\u5b66\u7fd2\u3059\u308b\u300d\u5fc5\u8981\u306f\u306a\u3044<!-- notionvc: f61abd19-8f0f-4ecc-9d12-1ed94c4fd4d6 --><\/h2>\n<p>\u6a5f\u68b0\u5b66\u7fd2\u7528\u9014\u3067\u306f\u3001PySpark\u3092\u5b66\u7fd2\u30a2\u30eb\u30b4\u30ea\u30ba\u30e0\u305d\u306e\u3082\u306e\u306b\u4f7f\u3046\u3088\u308a\u3082\u3001\u5927\u898f\u6a21\u306a\u524d\u51e6\u7406\u3084\u7279\u5fb4\u91cf\u751f\u6210\u306b\u4f7f\u3046\u30b1\u30fc\u30b9\u304c\u591a\u304f\u3042\u308a\u307e\u3059\u3002<\/p>\n<p>\u305f\u3068\u3048\u3070\u3001\u6570\u5e74\u5206\u306e\u8cfc\u8cb7\u5c65\u6b74\u3068Web\u30a2\u30af\u30bb\u30b9\u30ed\u30b0\u3092\u7d50\u5408\u3057\u3001\u6b20\u640d\u51e6\u7406\u3001\u30ab\u30c6\u30b4\u30ea\u5909\u63db\u3001\u6642\u7cfb\u5217\u96c6\u8a08\u3092\u884c\u3063\u3066\u5b66\u7fd2\u30c7\u30fc\u30bf\u3092\u4f5c\u308b\u51e6\u7406\u306fSpark\u3068\u76f8\u6027\u304c\u3088\u3044\u9818\u57df\u3067\u3059\u3002<\/p>\n<p>\u4e00\u65b9\u3001\u30e2\u30c7\u30eb\u5b66\u7fd2\u306b\u3064\u3044\u3066\u306f\u3001\u30c7\u30fc\u30bf\u30b5\u30a4\u30ba\u3084\u30a2\u30eb\u30b4\u30ea\u30ba\u30e0\u306b\u3088\u3063\u3066LightGBM\u3001XGBoost\u3001PyTorch\u3001TensorFlow\u306a\u3069\u306e\u5c02\u7528\u30d5\u30ec\u30fc\u30e0\u30ef\u30fc\u30af\u3092\u4f7f\u3046\u65b9\u6cd5\u304c\u3042\u308a\u307e\u3059\u3002<\/p>\n<p>\u3053\u3053\u3067\u3082\u91cd\u8981\u306a\u306e\u306f\u5f79\u5272\u5206\u62c5\u3067\u3059\u3002\u300c\u3059\u3079\u3066\u3092Spark\u3067\u5b9f\u884c\u3059\u308b\u300d\u306e\u3067\u306f\u306a\u304f\u3001\u300c\u5206\u6563\u51e6\u7406\u304c\u5fc5\u8981\u306a\u5de5\u7a0b\u3092Spark\u306b\u4efb\u305b\u308b\u300d\u3068\u3044\u3046\u767a\u60f3\u304c\u3001\u5b9f\u52d9\u3067\u306f\u6271\u3044\u3084\u3059\u3044\u8a2d\u8a08\u306b\u306a\u308a\u307e\u3059\u3002<\/p>\n<p><!-- notionvc: 4be5dde0-0f77-4b7e-bc04-c31c0c7a8ddb --><\/p>\n<p><!-- notionvc: 4e48015e-bcc6-4ee2-8ef7-7ad25ad3a383 --><\/p>\n<h2>Structured Streaming\u3067\u30d0\u30c3\u30c1\u3068\u30ea\u30a2\u30eb\u30bf\u30a4\u30e0\u3092\u3064\u306a\u3050<!-- notionvc: 4b7e4993-6f83-480d-8595-5ca4d2fa6146 --><\/h2>\n<p>PySpark\u306e\u5fdc\u7528\u7bc4\u56f2\u3092\u5e83\u3052\u308b\u306a\u3089\u3001Structured Streaming\u306f\u907f\u3051\u3066\u901a\u308c\u307e\u305b\u3093\u3002<\/p>\n<p>Structured Streaming\u3067\u306f\u3001\u30b9\u30c8\u30ea\u30fc\u30df\u30f3\u30b0\u30c7\u30fc\u30bf\u3092DataFrame\u306b\u8fd1\u3044\u8003\u3048\u65b9\u3067\u51e6\u7406\u3067\u304d\u307e\u3059\u3002\u30bb\u30f3\u30b5\u30fc\u30c7\u30fc\u30bf\u3001\u30a2\u30af\u30bb\u30b9\u30ed\u30b0\u3001IoT\u30a4\u30d9\u30f3\u30c8\u3001\u53d6\u5f15\u30c7\u30fc\u30bf\u306a\u3069\u3092\u7d99\u7d9a\u7684\u306b\u51e6\u7406\u3057\u306a\u304c\u3089\u3001\u30d0\u30c3\u30c1\u51e6\u7406\u3068\u4f3c\u305fAPI\u3092\u5229\u7528\u3067\u304d\u307e\u3059\u3002<\/p>\n<p>\u305f\u3060\u3057\u3001\u30ea\u30a2\u30eb\u30bf\u30a4\u30e0\u51e6\u7406\u3067\u306f\u300c\u30c7\u30fc\u30bf\u304c\u3044\u3064\u5230\u7740\u3059\u308b\u304b\u300d\u304c\u91cd\u8981\u3067\u3059\u3002\u30a4\u30d9\u30f3\u30c8\u6642\u523b\u3068\u5230\u7740\u6642\u523b\u306f\u4e00\u81f4\u3057\u306a\u3044\u305f\u3081\u3001\u30a6\u30a9\u30fc\u30bf\u30fc\u30de\u30fc\u30af\u3092\u5229\u7528\u3057\u3066\u9045\u5ef6\u30c7\u30fc\u30bf\u3092\u3069\u3053\u307e\u3067\u8a31\u5bb9\u3059\u308b\u304b\u3092\u8a2d\u8a08\u3057\u307e\u3059\u3002<\/p>\n<p>Spark 4.x\u3067\u306f\u72b6\u614b\u7ba1\u7406\u3082\u5927\u304d\u304f\u9032\u5316\u3057\u307e\u3057\u305f\u3002TransformWithState\u306a\u3069\u306b\u3088\u3063\u3066\u3001\u72b6\u614b\u3092\u6301\u3064\u30b9\u30c8\u30ea\u30fc\u30df\u30f3\u30b0\u51e6\u7406\u3092\u3088\u308a\u67d4\u8edf\u306b\u8a18\u8ff0\u3067\u304d\u308b\u3088\u3046\u306b\u306a\u3063\u3066\u3044\u307e\u3059\u3002<\/p>\n<p>\u3055\u3089\u306bSpark 4.1\u3067\u306fReal-Time Mode\u3082\u5c0e\u5165\u3055\u308c\u3001\u7279\u5b9a\u306e\u30b9\u30c6\u30fc\u30c8\u30ec\u30b9\u51e6\u7406\u3067\u306f\u975e\u5e38\u306b\u4f4e\u3044\u30ec\u30a4\u30c6\u30f3\u30b7\u3092\u72d9\u3048\u308b\u3088\u3046\u306b\u306a\u308a\u307e\u3057\u305f\u3002<\/p>\n<p>\u305f\u3060\u3057\u3001\u3059\u3079\u3066\u306e\u51e6\u7406\u3092\u30ea\u30a2\u30eb\u30bf\u30a4\u30e0\u5316\u3059\u308b\u5fc5\u8981\u306f\u3042\u308a\u307e\u305b\u3093\u3002\u79d2\u5358\u4f4d\u306e\u5fdc\u7b54\u304c\u4e0d\u8981\u306a\u3089\u3001\u30de\u30a4\u30af\u30ed\u30d0\u30c3\u30c1\u65b9\u5f0f\u306e\u307b\u3046\u304c\u904b\u7528\u3084\u30b3\u30b9\u30c8\u306e\u9762\u3067\u5408\u7406\u7684\u306a\u5834\u5408\u304c\u3042\u308a\u307e\u3059\u3002\u30ea\u30a2\u30eb\u30bf\u30a4\u30e0\u5316\u306f\u6280\u8853\u8981\u4ef6\u3067\u306f\u306a\u304f\u3001\u30d3\u30b8\u30cd\u30b9\u8981\u4ef6\u304b\u3089\u9006\u7b97\u3059\u308b\u3053\u3068\u304c\u91cd\u8981\u3067\u3059\u3002<!-- notionvc: 2ca57707-1312-4e6d-b2cb-1e314a974763 --><\/p>\n<p><!-- notionvc: 8aeb6529-20ee-45bf-9a9f-4def1721178b --><\/p>\n<h2>Spark 4.x\u3067\u7279\u306b\u6ce8\u76ee\u3057\u305f\u3044\u6a5f\u80fd<!-- notionvc: 1f20bcd1-049a-4576-a08d-b8e7dcdd9544 --><\/h2>\n<p>Spark 4.0\u3067\u306fVARIANT\u578b\u304c\u5c0e\u5165\u3055\u308c\u3001JSON\u306a\u3069\u306e\u534a\u69cb\u9020\u5316\u30c7\u30fc\u30bf\u3092\u6271\u3044\u3084\u3059\u304f\u306a\u308a\u307e\u3057\u305f\u3002\u30ed\u30b0\u3084\u30a4\u30d9\u30f3\u30c8\u30c7\u30fc\u30bf\u3067\u306f\u30b9\u30ad\u30fc\u30de\u304c\u4e00\u5b9a\u3057\u306a\u3044\u30b1\u30fc\u30b9\u304c\u3042\u308b\u305f\u3081\u3001\u8907\u96d1\u306aJSON\u3092\u3059\u3079\u3066\u6587\u5b57\u5217\u3068\u3057\u3066\u51e6\u7406\u3059\u308b\u3088\u308a\u3001\u534a\u69cb\u9020\u5316\u30c7\u30fc\u30bf\u3068\u3057\u3066\u6271\u3048\u308b\u3053\u3068\u306b\u306f\u5927\u304d\u306a\u610f\u5473\u304c\u3042\u308a\u307e\u3059\u3002<\/p>\n<p>\u307e\u305f\u3001Python Data Source API\u3084Python UDTF\u306b\u3088\u3063\u3066\u3001Python\u304b\u3089Spark\u306e\u62e1\u5f35\u30dd\u30a4\u30f3\u30c8\u3078\u30a2\u30af\u30bb\u30b9\u3059\u308b\u9078\u629e\u80a2\u3082\u5897\u3048\u307e\u3057\u305f\u3002<\/p>\n<p>\u3055\u3089\u306bSpark 4.0\u3067\u306fANSI SQL\u30e2\u30fc\u30c9\u304c\u30c7\u30d5\u30a9\u30eb\u30c8\u3067\u6709\u52b9\u306b\u306a\u3063\u3066\u3044\u307e\u3059\u30023.x\u7cfb\u304b\u3089\u79fb\u884c\u3059\u308b\u5834\u5408\u3001\u6697\u9ed9\u7684\u306a\u578b\u5909\u63db\u306a\u3069\u306e\u6319\u52d5\u304c\u5909\u308f\u308b\u53ef\u80fd\u6027\u304c\u3042\u308b\u305f\u3081\u3001\u65e2\u5b58\u30b8\u30e7\u30d6\u306e\u79fb\u884c\u30c6\u30b9\u30c8\u306f\u6b20\u304b\u305b\u307e\u305b\u3093\u3002<\/p>\n<p>Spark 4.1\u3067\u306f\u3001Spark Declarative Pipelines\u306b\u3088\u3063\u3066\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u3068\u30af\u30a8\u30ea\u3092\u5ba3\u8a00\u7684\u306b\u5b9a\u7fa9\u3059\u308b\u65b9\u5411\u6027\u3082\u5f37\u307e\u308a\u307e\u3057\u305f\u3002Spark\u306f\u5358\u306a\u308b\u5b9f\u884c\u30a8\u30f3\u30b8\u30f3\u304b\u3089\u3001\u30c7\u30fc\u30bf\u30d1\u30a4\u30d7\u30e9\u30a4\u30f3\u305d\u306e\u3082\u306e\u3092\u7ba1\u7406\u3059\u308b\u57fa\u76e4\u3078\u3068\u9818\u57df\u3092\u5e83\u3052\u3066\u3044\u307e\u3059\u3002<\/p>\n<h2>Kubernetes\u3068\u672c\u756a\u904b\u7528<!-- notionvc: a9067a04-76c3-451b-ba15-3a4672bf0347 --><br \/><!-- notionvc: 1f20bcd1-049a-4576-a08d-b8e7dcdd9544 --><\/h2>\n<p>\u672c\u756a\u74b0\u5883\u3067\u306f\u3001\u30b3\u30fc\u30c9\u3092\u66f8\u3051\u308b\u3060\u3051\u3067\u306f\u4e0d\u5341\u5206\u3067\u3059\u3002<\/p>\n<p>Spark\u306fStandalone\u3001YARN\u3001Kubernetes\u306a\u3069\u8907\u6570\u306e\u5b9f\u884c\u74b0\u5883\u306b\u5bfe\u5fdc\u3057\u3066\u304a\u308a\u3001\u30af\u30e9\u30a6\u30c9\u30cd\u30a4\u30c6\u30a3\u30d6\u306a\u57fa\u76e4\u3067\u306fKubernetes\u4e0a\u3067\u30c9\u30e9\u30a4\u30d0\u30fc\u3068\u30a8\u30b0\u30bc\u30ad\u30e5\u30fc\u30bf\u30fc\u3092Pod\u3068\u3057\u3066\u7ba1\u7406\u3059\u308b\u69cb\u6210\u3082\u9078\u629e\u3067\u304d\u307e\u3059\u3002<\/p>\n<p>\u305f\u3060\u3057\u3001Kubernetes\u3092\u4f7f\u3048\u3070\u81ea\u52d5\u7684\u306b\u904b\u7528\u304c\u7c21\u5358\u306b\u306a\u308b\u308f\u3051\u3067\u306f\u3042\u308a\u307e\u305b\u3093\u3002\u30b3\u30f3\u30c6\u30ca\u30a4\u30e1\u30fc\u30b8\u3001\u4f9d\u5b58\u30e9\u30a4\u30d6\u30e9\u30ea\u3001\u30b5\u30fc\u30d3\u30b9\u30a2\u30ab\u30a6\u30f3\u30c8\u3001\u30ea\u30bd\u30fc\u30b9\u5236\u9650\u3001\u30ed\u30b0\u3001\u518d\u5b9f\u884c\u3001\u30c1\u30a7\u30c3\u30af\u30dd\u30a4\u30f3\u30c8\u3001\u76e3\u8996\u306a\u3069\u3092\u8a2d\u8a08\u3059\u308b\u5fc5\u8981\u304c\u3042\u308a\u307e\u3059\u3002<\/p>\n<p>\u672c\u756a\u3067\u306f\u7279\u306b\u3001\u51e6\u7406\u6642\u9593\u3060\u3051\u3067\u306f\u306a\u304f\u3001\u5165\u529b\u30c7\u30fc\u30bf\u91cf\u3001\u30b7\u30e3\u30c3\u30d5\u30eb\u91cf\u3001\u30e1\u30e2\u30ea\u4f7f\u7528\u91cf\u3001\u5931\u6557\u7387\u3001\u518d\u5b9f\u884c\u6642\u9593\u3001\u30af\u30e9\u30a6\u30c9\u30b3\u30b9\u30c8\u307e\u3067\u89b3\u6e2c\u3059\u308b\u3053\u3068\u304c\u91cd\u8981\u3067\u3059\u3002<!-- notionvc: 310f4011-8e00-40ca-af6e-b9647031927f --><\/p>\n<h2>\u3088\u304f\u3042\u308b\u5931\u6557\u30d1\u30bf\u30fc\u30f3<!-- notionvc: 8410c7c5-14ac-461e-8cc0-a92944469221 --><br \/><!-- notionvc: 1f20bcd1-049a-4576-a08d-b8e7dcdd9544 --><\/h2>\n<p>PySpark\u3067\u983b\u767a\u3059\u308b\u554f\u984c\u306b\u306f\u5171\u901a\u70b9\u304c\u3042\u308a\u307e\u3059\u3002<\/p>\n<p>\u4ee3\u8868\u7684\u306a\u306e\u304c\u3001<code>collect()<\/code>\u306b\u3088\u3063\u3066\u5927\u91cf\u30c7\u30fc\u30bf\u3092\u30c9\u30e9\u30a4\u30d0\u30fc\u3078\u96c6\u3081\u3066\u3057\u307e\u3046\u3053\u3068\u3067\u3059\u3002Spark\u306f\u5206\u6563\u51e6\u7406\u3057\u3066\u3044\u3066\u3082\u3001\u6700\u5f8c\u306b\u3059\u3079\u3066\u3092Python\u30d7\u30ed\u30bb\u30b9\u3078\u623b\u305b\u3070\u5206\u6563\u51e6\u7406\u306e\u30e1\u30ea\u30c3\u30c8\u304c\u5931\u308f\u308c\u307e\u3059\u3002<\/p>\n<p>\u6b21\u306b\u3001UDF\u306e\u4e71\u7528\u3001\u904e\u5270\u306arepartition\u3001\u5de8\u5927JOIN\u3001\u30c7\u30fc\u30bf\u30b9\u30ad\u30e5\u30fc\u3001\u5c0f\u3055\u306a\u30d5\u30a1\u30a4\u30eb\u306e\u5927\u91cf\u751f\u6210\u304c\u3042\u308a\u307e\u3059\u3002<\/p>\n<p>\u307e\u305f\u3001\u30ad\u30e3\u30c3\u30b7\u30e5\u3082\u4e07\u80fd\u3067\u306f\u3042\u308a\u307e\u305b\u3093\u3002\u4f55\u5ea6\u3082\u518d\u5229\u7528\u3059\u308b\u30c7\u30fc\u30bf\u306a\u3089\u6709\u52b9\u3067\u3059\u304c\u3001\u4e00\u5ea6\u3057\u304b\u4f7f\u308f\u306a\u3044\u30c7\u30fc\u30bf\u3092\u7121\u8a08\u753b\u306b\u30ad\u30e3\u30c3\u30b7\u30e5\u3059\u308b\u3068\u3001\u30e1\u30e2\u30ea\u3084\u30b9\u30c8\u30ec\u30fc\u30b8\u3092\u5727\u8feb\u3057\u307e\u3059\u3002<\/p>\n<p>\u3057\u305f\u304c\u3063\u3066\u3001\u6027\u80fd\u6539\u5584\u3067\u306f\u300c\u3068\u308a\u3042\u3048\u305a\u30ad\u30e3\u30c3\u30b7\u30e5\u300d\u300c\u3068\u308a\u3042\u3048\u305a\u30af\u30e9\u30b9\u30bf\u5897\u5f37\u300d\u3067\u306f\u306a\u304f\u3001Spark UI\u3084\u5b9f\u884c\u8a08\u753b\u3092\u78ba\u8a8d\u3057\u3001\u3069\u3053\u3067\u6642\u9593\u3068\u30ea\u30bd\u30fc\u30b9\u3092\u6d88\u8cbb\u3057\u3066\u3044\u308b\u306e\u304b\u3092\u7279\u5b9a\u3059\u308b\u3053\u3068\u304c\u57fa\u672c\u3067\u3059\u3002<!-- notionvc: 876008a7-3ce9-4b5c-a337-b2e78ddd2e95 --><\/p>\n<h2>PySpark\u306e\u5b66\u7fd2\u30ed\u30fc\u30c9\u30de\u30c3\u30d7<!-- notionvc: 065b6085-d2b4-432b-a79c-68cffdda5060 --><br \/><!-- notionvc: 1f20bcd1-049a-4576-a08d-b8e7dcdd9544 --><\/h2>\n<p>pandas\u30e6\u30fc\u30b6\u30fc\u306a\u3089\u3001\u6700\u521d\u304b\u3089Spark\u306e\u5185\u90e8\u5b9f\u88c5\u3092\u3059\u3079\u3066\u7406\u89e3\u3059\u308b\u5fc5\u8981\u306f\u3042\u308a\u307e\u305b\u3093\u3002<\/p>\n<p>\u307e\u305apandas\u3067DataFrame\u64cd\u4f5c\u3092\u5fa9\u7fd2\u3057\u3001\u6b21\u306bpandas API on Spark\u3067\u5206\u6563\u30c7\u30fc\u30bf\u51e6\u7406\u3092\u4f53\u9a13\u3057\u307e\u3059\u3002\u305d\u306e\u5f8c\u3001PySpark DataFrame\u3001Spark SQL\u3001JOIN\u3001\u96c6\u8a08\u3001\u30a6\u30a3\u30f3\u30c9\u30a6\u95a2\u6570\u3078\u9032\u307f\u307e\u3059\u3002<\/p>\n<p>\u5b9f\u52d9\u3078\u9032\u3080\u6bb5\u968e\u3067\u306f\u3001Parquet\u3001Delta Lake\u3001\u30d1\u30fc\u30c6\u30a3\u30b7\u30e7\u30f3\u8a2d\u8a08\u3001Spark UI\u3001\u5b9f\u884c\u8a08\u753b\u3001\u30b7\u30e3\u30c3\u30d5\u30eb\u3001AQE\u3092\u5b66\u3073\u307e\u3059\u3002<\/p>\n<p>\u3055\u3089\u306b\u30c7\u30fc\u30bf\u57fa\u76e4\u3092\u62c5\u5f53\u3059\u308b\u306a\u3089Structured Streaming\u3001\u30a6\u30a9\u30fc\u30bf\u30fc\u30de\u30fc\u30af\u3001\u72b6\u614b\u7ba1\u7406\u3001\u30c1\u30a7\u30c3\u30af\u30dd\u30a4\u30f3\u30c8\u3001Kubernetes\u306a\u3069\u3078\u5e83\u3052\u307e\u3059\u3002<\/p>\n<p>\u3053\u306e\u9806\u756a\u306a\u3089\u3001\u300cPython\u3092\u66f8\u3051\u308b\u300d\u304b\u3089\u300c\u5206\u6563\u51e6\u7406\u3092\u8a2d\u8a08\u3067\u304d\u308b\u300d\u3078\u6bb5\u968e\u7684\u306b\u6210\u9577\u3067\u304d\u307e\u3059\u3002<!-- notionvc: 439f9bc7-d2cb-4b88-8ffc-cccdb735ea66 --><\/p>\n<p><!-- notionvc: fb3a8709-27ae-47bd-abc3-0f2a12e1e6bd --><\/p>\n<p><!-- notionvc: 8c9439ce-d4b2-4a98-8b2c-8c95920fcfa6 --><\/p>\n<h2>\u307e\u3068\u3081\u2500\u2500PySpark\u306f\u300cPython\u7248Spark\u300d\u304b\u3089\u30c7\u30fc\u30bf\u57fa\u76e4\u3078<!-- notionvc: 80ad3a6c-07c4-4606-8062-8a85a39c1c28 --><\/h2>\n<p>PySpark\u3092\u5358\u7d14\u306b\u300c\u5927\u91cf\u30c7\u30fc\u30bf\u3092\u51e6\u7406\u3059\u308bPython\u30e9\u30a4\u30d6\u30e9\u30ea\u300d\u3068\u7406\u89e3\u3059\u308b\u3068\u3001\u305d\u306e\u4fa1\u5024\u3092\u5341\u5206\u306b\u6d3b\u304b\u305b\u307e\u305b\u3093\u3002<\/p>\n<p>\u672c\u8cea\u306f\u3001DataFrame\u3068SQL\u3092\u4e2d\u5fc3\u306b\u3001\u30d0\u30c3\u30c1ETL\u3001\u30ec\u30a4\u30af\u30cf\u30a6\u30b9\u3001\u6a5f\u68b0\u5b66\u7fd2\u524d\u51e6\u7406\u3001\u30b9\u30c8\u30ea\u30fc\u30df\u30f3\u30b0\u307e\u3067\u3092\u540c\u3058\u5b9f\u884c\u30e2\u30c7\u30eb\u3067\u6271\u3048\u308b\u3053\u3068\u306b\u3042\u308a\u307e\u3059\u3002<\/p>\n<p>2026\u5e74\u73fe\u5728\u306fSpark 4.x\u306b\u3088\u3063\u3066\u3001VARIANT\u3001Python Data Source\u3001Python UDTF\u3001\u72b6\u614b\u7ba1\u7406\u3001\u4f4e\u9045\u5ef6\u30b9\u30c8\u30ea\u30fc\u30df\u30f3\u30b0\u3001\u5ba3\u8a00\u7684\u30d1\u30a4\u30d7\u30e9\u30a4\u30f3\u306a\u3069\u304c\u9032\u5316\u3057\u3066\u3044\u307e\u3059\u3002<\/p>\n<p>\u4e00\u65b9\u3067\u3001\u6700\u65b0\u6a5f\u80fd\u3092\u4f7f\u3046\u3053\u3068\u81ea\u4f53\u304c\u76ee\u7684\u3067\u306f\u3042\u308a\u307e\u305b\u3093\u3002\u91cd\u8981\u306a\u306e\u306f\u3001\u30c7\u30fc\u30bf\u91cf\u3001\u30ec\u30a4\u30c6\u30f3\u30b7\u3001\u30b3\u30b9\u30c8\u3001\u904b\u7528\u4f53\u5236\u306b\u5fdc\u3058\u3066\u9069\u5207\u306a\u51e6\u7406\u65b9\u5f0f\u3092\u9078\u3076\u3053\u3068\u3067\u3059\u3002<\/p>\n<p>pandas\u304b\u3089\u59cb\u3081\u308b\u306a\u3089\u3001\u307e\u305aDataFrame\u64cd\u4f5c\u3092\u7406\u89e3\u3057\u3001pandas API on Spark\u3067\u5206\u6563\u51e6\u7406\u3078\u79fb\u884c\u3057\u3001Spark SQL\u3068\u5b9f\u884c\u8a08\u753b\u3092\u5b66\u3073\u3001\u6700\u5f8c\u306b\u30b9\u30c8\u30ea\u30fc\u30df\u30f3\u30b0\u3084\u672c\u756a\u904b\u7528\u3078\u9032\u3080\u3002\u3053\u306e\u300c\u5c0f\u3055\u304f\u59cb\u3081\u3066\u3001\u5206\u6563\u51e6\u7406\u306e\u8a2d\u8a08\u80fd\u529b\u3092\u7a4d\u307f\u4e0a\u3052\u308b\u300d\u3053\u3068\u3053\u305d\u3001PySpark\u3092\u5b9f\u52d9\u3067\u4f7f\u3044\u3053\u306a\u3059\u6700\u77ed\u30eb\u30fc\u30c8\u3067\u3059\u3002<!-- notionvc: c2672818-6978-446d-a7ef-adfae983c9ce --><\/p>\n<p><!-- notionvc: 2f812547-1116-4dda-afaa-19b622dd8fbc --><\/p>","protected":false},"excerpt":{"rendered":"<p>PySpark\u306e\u57fa\u790e\u304b\u3089DataFrame\u3001pandas API on Spark\u3001ETL\u3001\u6a5f\u68b0\u5b66\u7fd2\u524d\u51e6\u7406\u3001Structured Streaming\u3001\u672c\u756a\u904b\u7528\u307e\u3067\u3092\u4f53\u7cfb\u7684\u306b\u89e3\u8aac\u3002Spark 4.x\u306e\u6700\u65b0\u6a5f\u80fd\u3084VARIAN [&hellip;]<\/p>\n","protected":false},"author":22,"featured_media":7203,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","swell_btn_cv_data":"","footnotes":"","_wp_rev_ctl_limit":""},"categories":[1249],"tags":[1152,985,1153,674,848],"class_list":["post-7205","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-knowledge","tag-apache-spark","tag-pyspark","tag-structured-streaming","tag-674","tag-848"],"_links":{"self":[{"href":"https:\/\/since2020.jp\/media\/wp-json\/wp\/v2\/posts\/7205","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/since2020.jp\/media\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/since2020.jp\/media\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/since2020.jp\/media\/wp-json\/wp\/v2\/users\/22"}],"replies":[{"embeddable":true,"href":"https:\/\/since2020.jp\/media\/wp-json\/wp\/v2\/comments?post=7205"}],"version-history":[{"count":1,"href":"https:\/\/since2020.jp\/media\/wp-json\/wp\/v2\/posts\/7205\/revisions"}],"predecessor-version":[{"id":9831,"href":"https:\/\/since2020.jp\/media\/wp-json\/wp\/v2\/posts\/7205\/revisions\/9831"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/since2020.jp\/media\/wp-json\/wp\/v2\/media\/7203"}],"wp:attachment":[{"href":"https:\/\/since2020.jp\/media\/wp-json\/wp\/v2\/media?parent=7205"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/since2020.jp\/media\/wp-json\/wp\/v2\/categories?post=7205"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/since2020.jp\/media\/wp-json\/wp\/v2\/tags?post=7205"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}