반응형
Notice
Recent Posts
Recent Comments
Link
«   2025/07   »
일 월 화 수 목 금 토
1 2 3 4 5
6 7 8 9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30 31
Tags more
Archives
Today
Total
관리 메뉴
728x90
반응형

목록2025/07/01 (2)

지구정복

[Spark] org.apache.spark.SparkException: Cannot broadcast the table that is larger than 8.0 GiB: 11.3 GiB.

spark 3.4.1 내부적으로 브로드캐스트 조인시에 브로드캐스트될 테이블이 너무 클 경우 아래 에러가 발생한다.Traceback (most recent call last): File "", line 1, in File "/usr/my/current/spark3-client/python/pyspark/sql/session.py", line 1440, in sql return DataFrame(self._jsparkSession.sql(sqlQuery, litArgs), self) File "..
데이터 엔지니어링 정복/Spark 2025. 7. 1. 11:11
[Iceberg] org.apache.iceberg.exceptions.ValidationException: Manifest is missing

iceberg 1.3.1spark 3.4.1hive 3.1.3 rewrite_manifest call procedure사용하다가 아래 에러 발생.25/07/01 00:09:18 INFO BaseMetastoreTableOperations: Refreshing table metadata from new version: hdfs://nameservice1/user/hive/warehouse/iceberg_test_db.db/my_test2/metadata/647318-e501c610-fe41-4a9a-bed0-7f10deec9c2b.metadata.json Traceback (most recent call last): File "/home/airflow_dags/src/pyspark/iceberg/Iceb..
데이터 엔지니어링 정복/Iceberg 2025. 7. 1. 11:09
Prev 1 Next
반응형