Tej zaum nws yuav nyuaj me ntsis los xav txog tag nrho cov kev pabcuam muaj thiab cov kev xaiv vaj tsev thaum xav txog cov ntaub ntawv platforms.
Ib lub lag luam cov ntaub ntawv platform feem ntau muaj cov ntaub ntawv warehouses, cov ntaub ntawv qauv, cov ntaub ntawv pas dej, thiab cov ntawv ceeb toom, txhua tus muaj lub hom phiaj tshwj xeeb thiab cov kev txawj xav tau. Nyob rau hauv sib piv, ib tug tshiab tsim hu ua cov ntaub ntawv lakehouse tau tshwm sim nyob rau hauv ob peb xyoos dhau los.
Lub versatility ntawm cov ntaub ntawv pas dej thiab cov ntaub ntawv warehouse tswj cov ntaub ntawv yog ua ke nyob rau hauv ib tug revolutionary cov ntaub ntawv cia architecture hu ua "cov ntaub ntawv lakehouse."
Peb yuav tshuaj xyuas cov ntaub ntawv lakehouse hauv qhov tob hauv cov ntawv no, suav nrog nws cov khoom, nta, architecture, thiab lwm yam.
Data Lakehouse yog dab tsi?
Raws li lub npe cuam tshuam, cov ntaub ntawv lakehouse yog ib hom tshiab ntawm cov ntaub ntawv architecture uas sib txuas cov ntaub ntawv pas dej nrog cov ntaub ntawv warehouse los daws qhov tsis txaus ntawm txhua qhov sib cais.
Nyob rau hauv qhov tseeb, lub lakehouse system siv pheej yig cia khaws cov ntaub ntawv loj heev hauv lawv cov ntaub ntawv qub, zoo li cov ntaub ntawv pas dej. Ntxiv cov txheej txheem metadata nyob rau sab saum toj ntawm lub khw tseem muab cov qauv ntaub ntawv thiab txhawb nqa cov ntaub ntawv tswj cov cuab yeej zoo li cov uas pom hauv cov ntaub ntawv khaws cia.
Nws khaws cov ntaub ntawv loj heev ntawm cov txheej txheem, cov txheej txheem ib nrab, thiab cov ntaub ntawv tsis tsim nyog uas lawv tau txais los ntawm cov kev lag luam sib txawv, cov tshuab, thiab cov khoom siv siv thoob plaws hauv lawv lub koom haum.
Feem ntau ntawm lub sijhawm, cov ntaub ntawv pas dej siv cov khoom siv qis qis nrog cov ntaub ntawv thov programming interface (API) los khaws cov ntaub ntawv hauv cov ntaub ntawv qhib, cov ntaub ntawv dav dav.
Qhov no ua rau nws muaj peev xwm rau ntau pab neeg nkag mus rau tag nrho cov tuam txhab cov ntaub ntawv los ntawm ib qho system rau ntau yam kev pib, xws li cov ntaub ntawv tshawb fawb, tshuab kev kawm, thiab kev txawj ntse ua lag luam.
nta
- Tsawg-nqi cia. Cov ntaub ntawv lakehouse yuav tsum muaj peev xwm khaws cov ntaub ntawv hauv cov khoom pheej yig cia, xws li Google huab Cia, Azure Blob Storage, Amazon Simple Storage Service, lossis ib txwm siv ORC lossis Parquet.
- Muaj peev xwm rau cov ntaub ntawv optimization: Cov ntaub ntawv layout optimization, caching, thiab indexing yog ob peb yam piv txwv ntawm yuav ua li cas cov ntaub ntawv lakehouse yuav tsum tau optimize cov ntaub ntawv thaum tswj cov ntaub ntawv tus thawj hom.
- Ib txheej ntawm kev hloov pauv metadata: Nyob rau sab saum toj ntawm qhov tseem ceeb ntawm tus nqi qis, qhov no ua rau muaj peev xwm tswj xyuas cov ntaub ntawv tseem ceeb rau cov ntaub ntawv khaws cia kev ua haujlwm.
- Kev them nyiaj yug rau Declarative DataFrame API: Feem ntau ntawm AI cov cuab yeej siv tau DataFrames los khaws cov ntaub ntawv raw khoom. Kev them nyiaj yug rau Declarative DataFrame API nce lub peev xwm los txhim kho cov ntaub ntawv nthuav tawm thiab cov qauv hauv kev teb rau cov ntaub ntawv tshwj xeeb lossis kev ua haujlwm AI.
- Kev them nyiaj yug rau ACID kev lag luam: Lub ntsiab lus ACID, uas sawv cev rau atomicity, sib xws, kev sib cais, thiab kav ntev, yog ib qho tseem ceeb hauv kev txhais cov kev lag luam thiab ua kom cov ntaub ntawv sib xws thiab kev cia siab ntawm cov ntaub ntawv. Xws li kev lag luam yav dhau los tsuas yog ua tau hauv cov ntaub ntawv warehouses, tab sis cov lakehouse muaj kev xaiv los siv lawv nrog cov ntaub ntawv pas dej ib yam nkaus. Nrog rau ob peb cov ntaub ntawv kav dej nrog rau cov ntaub ntawv sib koom ua ke nyeem thiab sau, qhov no daws qhov teeb meem ntawm cov ntaub ntawv tsis zoo ntawm yav tom ntej.
Cov ntsiab lus ntawm Data Lakehouse
Lub architecture ntawm cov ntaub ntawv lakehouse tau muab faib ua ob theem tseem ceeb ntawm qib siab. Cov txheej txheem khaws cia cov ntaub ntawv tau txais yog tswj los ntawm Lakehouse platform (piv txwv li, cov ntaub ntawv pas dej).
Tsis tas yuav thauj cov ntaub ntawv mus rau hauv cov ntaub ntawv warehouse lossis hloov nws mus rau hauv cov txheej txheem tsim khoom, cov txheej txheem txheej txheem tuaj yeem nug cov ntaub ntawv hauv txheej cia ncaj qha siv ntau yam cuab yeej.
Tom qab ntawd, BI apps, nrog rau AI thiab ML thev naus laus zis, tuaj yeem siv cov ntaub ntawv. Kev lag luam ntawm cov ntaub ntawv pas dej yog muab los ntawm tus qauv tsim, tab sis vim tias txhua lub cav ua haujlwm tuaj yeem nyeem cov ntaub ntawv no, cov lag luam muaj kev ywj pheej los ua kom cov ntaub ntawv npaj nkag mus rau kev tshuaj xyuas los ntawm ntau lub tshuab. Cov txheej txheem kev ua tau zoo thiab tus nqi tuaj yeem txhim kho los ntawm kev siv cov qauv no rau kev ua thiab kev tshuaj xyuas.
Vim nws txoj kev txhawb nqa rau kev lag luam database uas ua raws li cov hauv qab no ACID (atomicity, sib xws, kev sib cais, thiab ua haujlwm ntev), cov qauv tsim kuj ua rau ntau tog nkag mus thiab sau cov ntaub ntawv ib txhij hauv lub system:
- Atomicity hais txog qhov tseeb tias tag nrho cov kev sib pauv lossis tsis muaj nws, ua tiav thaum ua tiav kev sib pauv. Nyob rau hauv cov ntaub ntawv uas tus txheej txheem cuam tshuam, qhov no yuav pab kom tsis txhob muaj cov ntaub ntawv poob lossis kev noj nyiaj txiag.
- Zoo sib xws guarantees muas tshwm sim nyob rau hauv ib tug kwv yees, zoo ib yam. Nws tuav txoj kev ncaj ncees ntawm cov ntaub ntawv los ntawm kev xyuas kom meej tias txhua cov ntaub ntawv raug cai raws li cov cai tau teev tseg.
- rho tawm xyuas kom meej tias, txog thaum nws tiav lawm, tsis muaj kev hloov pauv tuaj yeem cuam tshuam los ntawm lwm yam kev lag luam hauv qhov system. Qhov no tso cai rau ntau tog nyeem thiab sau los ntawm tib lub system ib txhij yam tsis muaj kev cuam tshuam rau ib leeg.
- durability lav tias cov kev hloov pauv rau cov ntaub ntawv hauv ib qho system tseem muaj nyob tom qab kev sib pauv tiav, txawm tias muaj qhov system tsis ua haujlwm. Txhua qhov kev hloov pauv uas tau coj los ntawm kev sib pauv yuav khaws cia rau hauv cov ntaub ntawv mus ib txhis.
Data Lakehouse Architecture
Databricks (tus tsim kho tshiab thiab tsim qauv ntawm lawv lub tswv yim Delta Lake) thiab AWS yog ob tus neeg tawm tswv yim tseem ceeb rau lub tswv yim ntawm cov ntaub ntawv lakehouse. Yog li peb yuav tsum cia siab rau lawv txoj kev paub thiab kev pom los piav qhia txog kev tsim vaj tsev ntawm cov pas dej.
Cov ntaub ntawv lakehouse feem ntau yuav muaj tsib txheej:
- Ingestion txheej
- Cia txheej
- Metadata txheej
- API txheej
- Kev siv txheej
Ingestion txheej
Lub kaw lus thawj txheej yog tswj xyuas cov ntaub ntawv los ntawm ntau qhov chaw thiab xa mus rau txheej cia. Cov txheej txheem tuaj yeem siv ntau txoj cai los txuas rau ntau qhov chaw sab hauv thiab sab nraud, suav nrog kev sib txuas ua ke thiab streaming cov ntaub ntawv ua cov peev txheej, xws li
- NoSQL databases,
- cov ntaub ntawv sib koom
- Cov ntawv thov CRM,
- cov vev xaib,
- IoT sensors,
- kev tshaj tawm,
- Software as a Service (SaaS), thiab
- Relational database tswj systems, thiab lwm yam.
Ntawm no, cov khoom xws li Apache Kafka rau cov ntaub ntawv streaming thiab Amazon Data Migration Service (Amazon DMS) rau kev xa cov ntaub ntawv los ntawm RDBMSs thiab NoSQL databases tuaj yeem ua haujlwm.
Cia txheej
Lub lakehouse architecture yog tsim los pab kom khaws ntau hom ntaub ntawv raws li cov khoom hauv khw muag khoom pheej yig, xws li AWS S3. Siv cov ntaub ntawv qhib, cov cuab yeej siv tau tuaj yeem nyeem cov khoom no ncaj qha los ntawm lub khw.
Qhov no ua rau nws muaj peev xwm rau ntau APIs thiab cov khoom siv txheej txheej kom nkag tau thiab siv tib cov ntaub ntawv. Cov txheej txheem metadata khaws cov schemas rau cov txheej txheem thiab cov txheej txheem ib nrab kom cov khoom siv tuaj yeem siv rau cov ntaub ntawv thaum lawv nyeem nws.
Lub Hadoop Distributed File System (HDFS) platform, piv txwv li, tuaj yeem siv los tsim cov kev pabcuam huab cia uas cais kev suav thiab khaws cia hauv tsev. Lakehouse yog qhov tsim nyog rau cov kev pabcuam no.
Metadata txheej
Cov txheej txheem metadata yog qhov tseem ceeb ntawm cov ntaub ntawv lakehouse uas txawv qhov tsim. Nws yog ib daim ntawv teev npe uas muaj cov ntaub ntawv metadata (cov ntaub ntawv hais txog lwm cov ntaub ntawv) rau txhua yam khoom khaws cia hauv lub pas dej thiab tso cai rau cov neeg siv los siv cov peev txheej tswj xyuas xws li:
- Ib qho kev sib raug zoo ntawm cov ntaub ntawv tau pom los ntawm kev sib koom ua ke ua tsaug rau ACID kev hloov pauv;
- caching txuag huab khoom khaws cov ntaub ntawv;
- ntxiv cov ntaub ntawv qauv indexes siv indexing kom ceev cov lus nug ua;
- siv xoom-copy cloning kom duplicate cov ntaub ntawv khoom; thiab
- khaws tej yam versions ntawm cov ntaub ntawv, thiab lwm yam, siv cov ntaub ntawv versioning.
Tsis tas li ntawd, cov txheej txheem metadata tso cai rau kev siv schema kev tswj hwm, kev siv DW schema topologies zoo li lub hnub qub / snowflake schemas, thiab kev muab cov ntaub ntawv tswj hwm thiab kev tshuaj xyuas muaj peev xwm ncaj qha rau ntawm cov ntaub ntawv pas dej, txhim kho kev ncaj ncees ntawm tag nrho cov ntaub ntawv kav dej.
Cov yam ntxwv rau schema evolution thiab kev tswj hwm muaj nyob rau hauv schema tswj. Los ntawm kev tsis lees paub cov ntawv sau uas tsis ua raws li lub rooj schema, kev tswj hwm schema ua rau cov neeg siv tswj xyuas cov ntaub ntawv kev ncaj ncees thiab zoo.
Schema evolution tso cai rau lub rooj schema tam sim no hloov kho kom haum rau cov ntaub ntawv hloov pauv. Vim muaj ib qho kev tswj xyuas ib leeg nyob rau sab saum toj ntawm lub pas dej cov ntaub ntawv, kuj tseem muaj kev nkag mus rau kev tswj thiab txheeb xyuas qhov ua tau.
API txheej
Lwm txheej txheej tseem ceeb ntawm kev tsim vaj tsev yog tam sim no tam sim no, hosting ntau APIs uas txhua tus neeg siv kawg tuaj yeem siv los ua haujlwm sai dua thiab tau txais kev txheeb cais ntau dua.
Kev siv cov metadata APIs ua rau nws yooj yim dua los txheeb xyuas thiab nkag mus rau cov ntaub ntawv xav tau rau daim ntawv thov.
Hais txog cov tsev qiv ntawv kawm tshuab, qee qhov ntawm lawv, xws li TensorFlow thiab Spark MLlib, tuaj yeem nyeem cov ntaub ntawv qhib xws li Parquet thiab ncaj qha nkag mus rau txheej txheej metadata.
Nyob rau tib lub sijhawm, DataFrame APIs muab txoj hauv kev ntau dua rau kev ua kom zoo dua qub, ua rau cov programmers npaj thiab hloov cov ntaub ntawv tawg.
Kev siv txheej
Fais fab BI, Tableau, thiab lwm yam cuab yeej thiab cov apps tau tuav hauv qab txheej noj. Nrog rau lub pas dej tsim, tag nrho cov metadata thiab tag nrho cov ntaub ntawv khaws cia hauv lub pas dej tau nkag mus rau cov neeg siv khoom.
Lub pas dej tuaj yeem siv los ntawm txhua tus neeg siv hauv ib lub tuam txhab los ua txhua yam kev ua haujlwm analytics, suav nrog tsim kev lag luam kev txawj ntse dashboards thiab khiav SQL queries thiab tshuab kev kawm ua haujlwm.
Qhov zoo ntawm Data Lakehouse
Cov koom haum tuaj yeem tsim cov ntaub ntawv lakehouse los koom ua ke lawv cov ntaub ntawv tam sim no thiab txhim kho lawv cov txheej txheem tswj cov ntaub ntawv tag nrho. Los ntawm dismantling lub silo barriers txuas ntau qhov chaw, cov ntaub ntawv lakehouse tuaj yeem hloov qhov kev xav tau rau cov kev daws teeb meem sib txawv.
Piv nrog rau cov ntaub ntawv curated, qhov kev sib koom ua ke no ua rau muaj txiaj ntsig zoo tshaj qhov kawg-rau-kawg. Qhov no muaj ntau yam zoo:
- Tsawg kev tswj hwm: Ntau dua li rho tawm cov ntaub ntawv los ntawm cov ntaub ntawv nyoos thiab npaj siv rau hauv cov ntaub ntawv khaws cia, cov ntaub ntawv lakehouse tso cai rau txhua qhov chaw txuas rau nws kom lawv cov ntaub ntawv muaj thiab teeb tsa rau kev siv.
- Ua kom muaj txiaj ntsig zoo: Cov ntaub ntawv lakehouses yog tsim los siv cov txheej txheem niaj hnub uas faib cov kev suav thiab kev khaws cia, ua kom yooj yim los nthuav kev khaws cia yam tsis muaj kev siv hluav taws xob ntxiv. Tsuas yog kev siv pheej yig cov ntaub ntawv khaws cia ua rau muaj peev xwm ua kom muaj txiaj ntsig zoo.
- Kev tswj hwm cov ntaub ntawv zoo dua: Cov ntaub ntawv lakehouses yog tsim los nrog cov qauv qhib architecture, tso cai rau kev tswj hwm kev nyab xeeb, kev ntsuas, kev siv lub luag haujlwm, thiab lwm yam tseem ceeb ntawm kev tswj xyuas. Los ntawm kev sib sau cov peev txheej thiab cov ntaub ntawv, lawv yooj yim thiab txhim khu kev tswj hwm.
- Cov qauv yooj yim: Txij li thaum qhov kev sib txuas tau txwv tsis pub dhau xyoo 1980, thaum cov ntaub ntawv warehouses tau tsim thawj zaug, cov qauv hauv cheeb tsam schema tau nquag tsim hauv kev lag luam, txawm tias cov tuam tsev. Cov ntaub ntawv lakehouses siv qhov tseeb tias ntau hom ntaub ntawv tam sim no muaj cov qauv qhib rau schema los ntawm kev noj cov ntaub ntawv ntau qhov chaw nrog cov txheej txheem sib tshooj los ua kom cov txheej txheem.
Disadvantages ntawm Data Lakehouse
Txawm tias tag nrho cov hoopla ib puag ncig cov ntaub ntawv lakehouses, nws yog ib qho tseem ceeb kom nco ntsoov tias lub tswv yim tseem tshiab heev. Nco ntsoov ntsuas qhov tsis zoo ua ntej ua tiav rau qhov kev tsim tshiab no.
- Monolithic qauv: Ib lub tsev lakehouse tag nrho cov qauv tsim muaj ntau qhov zoo, tab sis nws kuj ua rau qee qhov teeb meem. Monolithic architecture feem ntau ua rau cov kev pabcuam tsis zoo rau txhua tus neeg siv thiab tuaj yeem nruj thiab nyuaj rau kev tswj xyuas. Feem ntau, architects thiab designers nyiam ib tug ntau modular architecture uas lawv yuav customize rau ntau yam kev siv.
- Lub tshuab tseem tsis tau muaj: Lub hom phiaj zaum kawg entails ib qho tseem ceeb ntawm kev kawm tshuab thiab kev txawj ntse. Ua ntej lakehouses tuaj yeem ua tau raws li kev xav, cov thev naus laus zis no yuav tsum txhim kho ntxiv.
- Tsis muaj kev nce qib tseem ceeb tshaj cov qauv uas twb muaj lawm: Tseem muaj qhov tsis txaus ntseeg ntau npaum li cas cov tsev lakehouses yuav pab tau tiag tiag. Qee tus detractors sib cav hais tias lub pas dej-warehouse tsim ua ke nrog cov cuab yeej siv hluav taws xob tsim nyog tuaj yeem ua tau zoo sib xws.
Kev sib tw ntawm Data Lakehouse
Nws tuaj yeem nyuaj rau kev siv cov ntaub ntawv lakehouse txheej txheem. Vim yog qhov tsis sib xws ntawm nws cov khoom tivthaiv, nws tsis raug saib cov ntaub ntawv lakehouse raws li tag nrho cov qauv zoo tagnrho lossis "ib lub platform rau txhua yam," rau ib qho.
Tsis tas li ntawd, vim muaj kev nce qib ntawm cov ntaub ntawv pas dej, cov lag luam yuav tsum tau txav lawv cov ntaub ntawv khaws cia tam sim no rau lawv, tsuas yog tso siab rau kev cog lus ntawm kev vam meej uas tsis muaj txiaj ntsig kev lag luam.
Yog tias muaj teeb meem latency lossis kev tawm tsam thoob plaws hauv txoj kev hloov pauv, qhov no yuav ua rau kim, siv sijhawm, thiab tej zaum tsis zoo.
Cov neeg siv khoom lag luam yuav tsum tau txais cov thev naus laus zis tshwj xeeb, raws li qee tus neeg muag khoom uas nthuav tawm lossis cuam tshuam kev lag luam kev daws teeb meem raws li cov ntaub ntawv lakehouses. Cov no tej zaum yuav tsis ua haujlwm nrog lwm cov cuab yeej txuas nrog cov ntaub ntawv pas dej ntawm qhov nruab nrab ntawm qhov system, ntxiv rau cov teeb meem.
Tsis tas li ntawd, nws yuav nyuaj rau muab 24/7 analytics thaum khiav lag luam-tseem ceeb ua hauj lwm, uas hu rau infrastructure nrog nqi-zoo scalability.
xaus
Qhov tshiab tshaj plaws ntau yam ntawm cov chaw zov me nyuam nyob rau hauv xyoo tas los no yog cov ntaub ntawv lakehouse. Nws koom nrog ntau yam kev lag luam, xws li cov ntaub ntawv thev naus laus zis, qhib software, huab xam, thiab faib khoom raws tu qauv.
Nws tso cai rau cov lag luam hauv nruab nrab khaws txhua yam ntaub ntawv los ntawm txhua qhov chaw, ua kom yooj yim rau kev tswj hwm thiab tshuaj xyuas. Data Lakehouse yog ib lub tswv yim zoo nkauj heev.
Ib lub tuam txhab yuav muaj kev sib tw tseem ceeb yog tias nws tau nkag mus rau tag nrho-hauv-ib cov ntaub ntawv platform uas tau nrawm thiab ua haujlwm zoo li lub tsev khaws ntaub ntawv thaum tseem hloov pauv raws li cov ntaub ntawv pas dej.
Lub tswv yim tseem tab tom tsim thiab tseem tshiab. Raws li qhov tshwm sim, nws tuaj yeem siv qee lub sijhawm los txiav txim siab seb puas muaj ib yam dab tsi tuaj yeem nthuav dav.
Peb txhua tus yuav tsum xav paub txog qhov kev taw qhia uas Lakehouse architecture tab tom mus.
Sau ntawv cia Ncua