지구정복

Spark3로 managed table create하는데 아래와 같은 에러 발생 spark-sql> CREATE TABLE spark_catalog.db1_test.example_hive1 (         >     id INT,         >     name STRING,         >     age INT         > )         > STORED AS parquet;  ErrorCaused by: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Table db1_test.example_hive1 failed strict managed table checks due to the following reas..

airflow db upgrade error발생하면  airflow db upgrade 진행 만약 위 명령어 에러나면 아래 airflow python package 설치해준다./usr/airflow/.pyenv/versions/venv/bin/pip install sqlalchemy==1.4.49/usr/airflow/.pyenv/versions/venv/bin/pip install apache-airflow-providers-common-sql 그리고 다시 airflow db upgrade 진행

CHAPTER 6 Apache SparkConfigurationConfiguring Apache Iceberg and SparkConfiguring via the CLIAs a first step, you’ll need to specify the required packages to be installed and used with the Spark session. To do so, Spark provides the --packages option, which allows Spark to easily download the specified Maven-based packages and  its dependencies to add them to the classpath of your application. ..

CHAPTER 5 Iceberg Catalogs Requirements of an Iceberg CatalogIceberg provides a catalog interface that requires the implementation of a set of functions, primarily ones to list existing tables, create tables, drop tables, check whether a table exists, and rename tables. Hive Metastore, AWS Glue, and a filesystem catalog (Hadoop). with a filesystem as the catalog, there’s a file called version-hi..

CHAPTER 4 Optimizing the Performance of Iceberg TablesCompaction When you are querying yourApache Iceberg tables, you need to open and scan each file and then close the filewhen you’re done. The more files you have to scan for a query, the greater the costthese file operations will put on your query. it is possible to run intothe “small files problem,” where too many small files have an impact o..