지구정복

-환경Spark 3.4.1Iceberg 1.3.1 Hive 3.1.3 현재 Iceberg catalog는 Hive metastore로 사용중이다. 아래와 같이 Spark-sql로 Iceberg table을 생성하고 Drop하려니깐 에러가 발생했다.CREATE TABLE iceberg_test_db.test_tbl ( data STRING, log_timestamp TIMESTAMP)USING icebergPARTITIONED BY (days(log_timestamp))TBLPROPERTIES ( 'read.parquet.vectorization.enabled' = 'true', 'write.metadata.delete-after-commit.enabled' = 'true', 'wri..

CHAPTER 6 Apache SparkConfigurationConfiguring Apache Iceberg and SparkConfiguring via the CLIAs a first step, you’ll need to specify the required packages to be installed and used with the Spark session. To do so, Spark provides the --packages option, which allows Spark to easily download the specified Maven-based packages and  its dependencies to add them to the classpath of your application. ..

CHAPTER 5 Iceberg Catalogs Requirements of an Iceberg CatalogIceberg provides a catalog interface that requires the implementation of a set of functions, primarily ones to list existing tables, create tables, drop tables, check whether a table exists, and rename tables. Hive Metastore, AWS Glue, and a filesystem catalog (Hadoop). with a filesystem as the catalog, there’s a file called version-hi..

CHAPTER 4 Optimizing the Performance of Iceberg TablesCompaction When you are querying yourApache Iceberg tables, you need to open and scan each file and then close the filewhen you’re done. The more files you have to scan for a query, the greater the costthese file operations will put on your query. it is possible to run intothe “small files problem,” where too many small files have an impact o..