| 일 | 월 | 화 | 수 | 목 | 금 | 토 |
|---|---|---|---|---|---|---|
| 1 | 2 | 3 | 4 | 5 | ||
| 6 | 7 | 8 | 9 | 10 | 11 | 12 |
| 13 | 14 | 15 | 16 | 17 | 18 | 19 |
| 20 | 21 | 22 | 23 | 24 | 25 | 26 |
| 27 | 28 | 29 | 30 |
- Data Engineering
- Iceberg
- 코엑스
- 코엑스맛집
- dfs
- 용인맛집
- Data Engineer
- 코테
- hadoop
- 프로그래머스
- Trino
- BigData
- 파이썬
- HIVE
- 코딩
- 여행
- 자바
- 양평
- bigdata engineering
- 개발
- apache iceberg
- bigdata engineer
- 영어
- java
- BFS
- 삼성역맛집
- 코딩테스트
- 알고리즘
- 백준
- 맛집
- Today
- Total
목록Data Engineering (6)
지구정복
[Iceberg] Iceberg Guide Book Summary | CHAPTER 6. Apache Spark
CHAPTER 6 Apache SparkConfigurationConfiguring Apache Iceberg and SparkConfiguring via the CLIAs a first step, you’ll need to specify the required packages to be installed and used with the Spark session. To do so, Spark provides the --packages option, which allows Spark to easily download the specified Maven-based packages and its dependencies to add them to the classpath of your application. ..
[Iceberg] Iceberg Guide Book Summary | CHAPTER 4. Optimizing the Performance of Iceberg Tables
CHAPTER 4 Optimizing the Performance of Iceberg TablesCompaction When you are querying yourApache Iceberg tables, you need to open and scan each file and then close the filewhen you’re done. The more files you have to scan for a query, the greater the costthese file operations will put on your query. it is possible to run intothe “small files problem,” where too many small files have an impact o..
[Iceberg] 기본 내용 정리 | Basic concepts using Trino
참고icberg guidebookhttps://trino.io/docs/current/connector/iceberg.html#metadata-tablesTrino 402버전 문서: https://trino.io/docs/402/overview/use-cases.html 0. PE/DE로서 Iceberg 사용시 어떤 점들을 중점적으로 알아야 할까? -Trino/Spark3 설치법-Iceberg 셋팅법(Trino와 Spark3에서 사용가능하도록 셋팅하기)-Ranger 권한관리방법(현재는 Trino만 유저에 한해서만 작동하지만 LDAP연동시에는 그룹 권한은 미작동 / Spark3도 되지만 권한적용이 원활히 되진 않음 / 그룹 및 유저권한관리)-Iceberg의 기본 개념 및 사용방법 1. Iceberg ..
[Iceberg] Iceberg Guide Book Summary | Chaper 11. Apache Iceberg in Production
CHAPTER 11 Apache Iceberg in ProductionIn this chapter, we will discuss many of the tools that can be used to help monitor and maintain Apache Iceberg tables in production. Apache Iceberg Metadata TablesSELECT * FROM catalog.table.history AS OF VERSION 1059035530770364194 These tables are generated from the metadata across Apache Iceberg metadata files at query time. The history Metadata Table..