Zingakhale zovuta kulingalira zonse zomwe zilipo komanso zosankha zamamangidwe poganizira za nsanja.
Pulatifomu ya data yamabizinesi nthawi zambiri imakhala ndi malo osungiramo zidziwitso, mitundu ya data, nyanja za data, ndi malipoti, chilichonse chimakhala ndi cholinga china komanso maluso ofunikira. Mosiyana ndi izi, mapangidwe atsopano otchedwa data lakehouse atulukira m'zaka zingapo zapitazi.
Kusinthasintha kwa malo osungira zidziwitso ndi kasamalidwe ka data kosungiramo zidziwitso zikuphatikizidwa mukusintha kosungirako deta komwe kumatchedwa "data lakehouse."
Tiwona mozama zalakehouse mu positi iyi, kuphatikiza zigawo zake, mawonekedwe ake, kapangidwe kake, ndi zina.
Kodi Data Lakehouse ndi chiyani?
Monga momwe dzinalo likusonyezera, malo osungiramo data ndi mtundu watsopano wa zomangamanga zomwe zimagwirizanitsa nyanja ya data ndi malo osungiramo deta kuti athetse zofooka za aliyense payekha.
M'malo mwake, makina osungiramo nyanja amagwiritsa ntchito malo osungiramo otsika mtengo kuti asunge zambiri zamitundu yawo yoyambirira, monga nyanja za data. Kuwonjezera masanjidwe a metadata pamwamba pa sitolo kumaperekanso dongosolo la deta ndikupatsa mphamvu zida zoyendetsera deta monga zomwe zimapezeka m'malo osungiramo deta.
Imasunga ma data ochulukirapo adongosolo, osasinthika, komanso osakhazikika omwe amapeza kuchokera kumabizinesi osiyanasiyana, machitidwe, ndi zida zomwe zimagwiritsidwa ntchito m'bungwe lawo lonse.
Nthawi zambiri, nyanja za data zimagwiritsa ntchito zida zotsika mtengo zosungirako zokhala ndi mawonekedwe opangira mafayilo (API) kusungira zidziwitso m'mafayilo otseguka, amtundu wamba.
Izi zimapangitsa kuti magulu ambiri athe kupeza deta yonse ya kampani kudzera mu dongosolo limodzi lazinthu zosiyanasiyana, monga sayansi ya deta, makina kuphunzira, ndi nzeru zamabizinesi.
Mawonekedwe
- Kusungirako zotsika mtengo. Malo osungiramo zinthu zakale akuyenera kusungiramo zinthu zotsika mtengo, monga Google Cloud Kusungirako, Azure Blob Storage, Amazon Simple Storage Service, kapena mbadwa pogwiritsa ntchito ORC kapena Parquet.
- Kuthekera kwa kukhathamiritsa kwa data: Kukhathamiritsa kwa masanjidwe a data, kusungitsa, ndi kulondolera ndi zitsanzo zochepa za momwe bwalo la data liyenera kukwanitsira kukhathamiritsa deta ndikusunga mawonekedwe ake enieni.
- Chigawo cha metadata yosinthira: Pamwamba pa zosungirako zotsika mtengo, izi zimathandizira kuthekera kwa kasamalidwe ka data kofunikira pakugwirira ntchito kosungirako deta.
- Thandizo la Declarative DataFrame API: Zida zambiri za AI zitha kugwiritsa ntchito DataFrames kuti zitengenso data ya sitolo yaiwisi. Kuthandizira kwa Declarative DataFrame API kumawonjezera kuthekera kosintha mafotokozedwe ndi kapangidwe ka data poyankha sayansi ya data kapena ntchito ya AI.
- Thandizo la zochitika za ACID: Mawu oti ACID, omwe amaimira atomicity, consistency, isolation, ndi durability, ndi gawo lofunika kwambiri pofotokozera zochitika ndikuwonetsetsa kusasinthasintha ndi kudalirika kwa deta. Zochita zoterezi zinali zotheka kale m'malo osungiramo zinthu, koma lakehouse imapereka mwayi wogwiritsa ntchito ndi nyanja za data komanso. Ndi mapaipi angapo a data kuphatikizapo kuwerengera ndi kulemba nthawi imodzi, izi zimathetsa vuto la khalidwe lochepa la deta.
Zinthu za Data Lakehouse
Zomangamanga za data Lakehouse zimagawidwa m'magulu awiri akuluakulu pamlingo wapamwamba. Zosungirako zosungirako zimayang'aniridwa ndi nsanja ya Lakehouse (ie, nyanja ya data).
Popanda kufunikira kuyika deta mu malo osungiramo deta kapena kuisintha kukhala eni ake, gawo lokonzekera limatha kufunsa zomwe zili muzosungiramo mwachindunji pogwiritsa ntchito zida zingapo.
Kenako, mapulogalamu a BI, komanso matekinoloje a AI ndi ML, amatha kugwiritsa ntchito deta. Zachuma za nyanja ya data zimaperekedwa ndi kapangidwe kameneka, koma chifukwa injini iliyonse yokonza ikhoza kuwerenga deta iyi, mabizinesi ali ndi ufulu wopanga deta yokonzedwa kuti ipezeke kuti ifufuzidwe ndi machitidwe osiyanasiyana. Ntchito ya purosesa komanso mtengo wake ukhoza kupitsidwanso pogwiritsa ntchito njirayi pokonza ndi kusanthula.
Chifukwa cha kuthandizira kwake pamachitidwe a database omwe amatsatira njira zotsatirazi za ACID (atomicity, consistency, isolation, and durability), zomangamanga zimathandizanso maphwando ambiri kupeza ndi kulemba deta nthawi imodzi mkati mwa dongosolo:
- Atomiki zimatanthawuza kuti ntchito yonse kapena palibe, imatheka mukamaliza kuchita. Kukachitika kuti ndondomeko yasokonezedwa, izi zimathandiza kupewa kutaya deta kapena ziphuphu.
- Kusagwirizana zimatsimikizira kuti malonda amachitika m'njira yodziwikiratu, yosasinthika. Imasunga umphumphu wa deta poonetsetsa kuti deta iliyonse ndi yovomerezeka mogwirizana ndi malamulo okonzedweratu.
- kutchinjiriza imawonetsetsa kuti, mpaka itatha, palibe ntchito yomwe ingakhudzidwe ndi zochitika zina zilizonse mkati mwadongosolo. Izi zimathandiza maphwando ambiri kuwerenga ndi kulemba kuchokera ku dongosolo lomwelo nthawi imodzi popanda kusokonezana.
- kwake zimatsimikizira kuti kusintha kwa deta mu dongosolo kumapitirizabe kukhalapo pambuyo pa kutha kwa ntchito, ngakhale zitalephera. Zosintha zilizonse zomwe zimachitika chifukwa chamalonda zimasungidwa pafayilo mpaka kalekale.
Data Lakehouse Architecture
Databricks (woyambitsa ndi wopanga malingaliro awo a Delta Lake) ndi AWS ndiawiri akulu omwe amachirikiza lingaliro la nyumba ya data. Motero tidzadalira chidziwitso chawo ndi luntha kufotokoza kamangidwe ka nyumba za nyanja.
Dongosolo la data la Lakehouse nthawi zambiri limakhala ndi zigawo zisanu:
- Kumeza wosanjikiza
- Malo osungirako
- Metadata layer
- API wosanjikiza
- Kugwiritsa wosanjikiza
Kumeza wosanjikiza
Gawo loyamba la dongosololi limayang'anira kusonkhanitsa deta kuchokera kuzinthu zosiyanasiyana ndikuzitumiza kumalo osungirako zinthu. Wosanjikiza atha kugwiritsa ntchito ma protocol angapo kuti alumikizane ndi magwero angapo amkati ndi akunja, kuphatikiza kuphatikiza batch ndi kutsitsa luso losinthira deta, monga
- NoSQL databases,
- mafayilo amagawana
- Mapulogalamu a CRM,
- masamba,
- Masensa a IoT,
- makanema,
- Mapulogalamu a Mapulogalamu monga Service (SaaS), ndi
- machitidwe oyang'anira database, etc.
Pakadali pano, zigawo monga Apache Kafka zotsatsira deta ndi Amazon Data Migration Service (Amazon DMS) zolowetsa deta kuchokera ku RDBMSs ndi NoSQL databases zitha kugwiritsidwa ntchito.
Malo osungirako
Zomangamanga zanyumba ya nyanjayi zimapangidwira kuti zitheke kusunga mitundu yosiyanasiyana ya data ngati zinthu m'masitolo otsika mtengo, monga AWS S3. Pogwiritsa ntchito mafayilo otseguka, zida za kasitomala zimatha kuwerenga zinthu izi mwachindunji kuchokera kusitolo.
Izi zimapangitsa kuti ma API ambiri ndi magawo ogwiritsira ntchito azitha kupeza ndikugwiritsa ntchito deta yomweyo. Chosanjikiza cha metadata chimasunga ma schemas a datasets okhazikika komanso opangidwa pang'ono kotero kuti zigawozo zitha kuziyika pa data pomwe zikuwerenga.
Pulogalamu ya Hadoop Distributed File System (HDFS), mwachitsanzo, ingagwiritsidwe ntchito pomanga ntchito zosungira mitambo zomwe zimagawanitsa makompyuta ndi kusungirako malo. Lakehouse ndiyoyenera kuchita izi.
Metadata layer
Chosanjikiza cha metadata ndi gawo lofunikira kwambiri lanyumba ya data yomwe imasiyanitsa kapangidwe kake. Ndi kabukhu kakang'ono komwe kamapereka metadata (zambiri zama data ena) pazinthu zonse zosungidwa m'nyanjayi ndipo amalola ogwiritsa ntchito kugwiritsa ntchito luso la oyang'anira monga:
- Kusinthika kosasinthika kwa database kumawonedwa ndi zochitika panthawi imodzi chifukwa cha zochitika za ACID;
- caching kuti musunge mafayilo osungira zinthu zamtambo;
- kuwonjezera ma index of data structure pogwiritsa ntchito indexing kuti mufulumizitse kukonza mafunso;
- kugwiritsa ntchito ziro-copy cloning kubwereza zinthu za data; ndi
- kusunga mitundu ina ya data, ndi zina zotero, gwiritsani ntchito kumasulira kwa data.
Kuphatikiza apo, kusanjikiza kwa metadata kumathandizira kukhazikitsa kasamalidwe ka schema, kugwiritsa ntchito zida za DW schema topologies monga nyenyezi / snowflake schemas, komanso kupereka utsogoleri wa data ndikuwunika mwachindunji panyanja ya data, kupititsa patsogolo kukhulupirika kwa payipi yonse ya data.
Zomwe zimapangidwira kusinthika kwa schema ndi kukakamiza zikuphatikizidwa mu kasamalidwe ka schema. Pokana zolemba zilizonse zomwe sizikugwirizana ndi schema ya tebulo, kukakamiza kwa schema kumathandizira ogwiritsa ntchito kusunga kukhulupirika ndi mtundu wa data.
Chisinthiko cha schema chimalola schema yomwe ilipo patebulo kuti isinthidwe kuti igwirizane ndi kusintha kwa data. Chifukwa cha mawonekedwe amodzi omwe ali pamwamba pa nyanja ya data, palinso njira zowongolera komanso zowerengera.
API wosanjikiza
Chigawo china chofunikira cha zomangamanga chilipo, kuchititsa ma API angapo omwe ogwiritsa ntchito onse amatha kugwiritsa ntchito kuti agwire ntchito mwachangu komanso kupeza ziwerengero zapamwamba kwambiri.
Kugwiritsa ntchito ma metadata APIs kumapangitsa kuti zikhale zosavuta kuzindikira ndi kupeza zinthu zomwe zimafunikira pa pulogalamu yomwe yaperekedwa.
Pankhani ya malaibulale ophunzirira makina, ena mwa iwo, monga TensorFlow ndi Spark MLlib, amatha kuwerenga mafayilo otseguka ngati Parquet ndikupeza mwachindunji wosanjikiza wa metadata.
Nthawi yomweyo, DataFrame APIs imapereka mwayi wokulirapo, kupangitsa opanga mapulogalamu kukonza ndikusintha deta yobalalika.
Kugwiritsa wosanjikiza
Power BI, Tableau, ndi zida zina ndi mapulogalamu amakhala pansi pazakudya. Ndi kapangidwe ka nyanja yam'madzi, metadata yonse ndi zonse zomwe zimasungidwa m'nyanja zimatha kupezeka ndi mapulogalamu a kasitomala.
Lakehouse ikhoza kugwiritsidwa ntchito ndi ogwiritsa ntchito onse mkati mwa kampani kuti achite zamitundu yonse ntchito za analytics, kuphatikiza kupanga ma dashboard anzeru zamabizinesi ndikuyendetsa mafunso a SQL ndi ntchito zophunzirira makina.
Ubwino wa Data Lakehouse
Mabungwe atha kupanga malo osungiramo data kuti agwirizanitse nsanja yawo yamakono ndikuwongolera njira zonse zoyendetsera deta. Pochotsa zotchinga za nkhokwe zolumikiza magwero osiyanasiyana, malo osungiramo data atha kulowa m'malo kufunikira kwa mayankho apadera.
Poyerekeza ndi magwero a data omwe amasungidwa, kuphatikiza uku kumatulutsa njira yomaliza yomaliza. Izi zili ndi zabwino zingapo:
- Utsogoleri wochepa: M'malo mochotsa zidziwitso kuchokera mu data yomwe idapangidwa ndikuikonzekera kuti igwiritsidwe ntchito m'malo osungiramo zidziwitso, malo osungiramo data amalola magwero aliwonse omwe alumikizidwa ndi datayo kukhala ndi deta yawo ndikukonzekera kugwiritsidwa ntchito.
- Kuwonjezeka kwa mtengo wogwira: Malo osungiramo zinthu zakale amamangidwa pogwiritsa ntchito zipangizo zamakono zomwe zimagawaniza mawerengedwe ndi kusungirako, zomwe zimapangitsa kuti zikhale zosavuta kuwonjezera malo osungira popanda kuwonjezera mphamvu zamagetsi. Kungogwiritsa ntchito zosungirako zotsika mtengo kumapangitsa kuti scalability ikhale yotsika mtengo.
- Ulamuliro wabwino wa data: Malo osungiramo data amamangidwa ndi mamangidwe otseguka, omwe amalola kuwongolera kowonjezereka pachitetezo, ma metrics, mwayi wotengera maudindo, ndi zigawo zina zofunika zowongolera. Pogwirizanitsa zothandizira ndi magwero a deta, amachepetsa ndi kupititsa patsogolo utsogoleri.
- Mfundo zosavuta: Popeza kuti kulumikizana kunali koletsedwa kwambiri m'zaka za m'ma 1980, pamene malo osungiramo deta anayamba kupangidwa, miyezo ya schema ya m'deralo inkapangidwa kawirikawiri mkati mwa mabizinesi, ngakhale m'madipatimenti. Malo osungiramo data amagwiritsa ntchito mfundo yakuti mitundu yambiri ya deta tsopano ili ndi miyezo yotseguka ya schema mwa kulowetsa magwero ambiri a deta ndi ndondomeko yofanana yofanana kuti athetse ndondomeko.
Zoyipa za Data Lakehouse
Ngakhale ma hoopla ozungulira malo osungiramo data, ndikofunikira kukumbukira kuti lingaliro likadali latsopano kwambiri. Onetsetsani kuti mwayesa zovuta zake musanachite mokwanira ku mapangidwe atsopanowa.
- Monolithic kapangidwe: Mapangidwe a Lakehouse ophatikiza zonse amapereka zabwino zingapo, koma amadzutsanso zovuta zina. Zomangamanga za monolithic nthawi zambiri zimabweretsa kusagwira bwino ntchito kwa ogwiritsa ntchito onse ndipo zimatha kukhala zolimba komanso zovuta kuzisamalira. Nthawi zambiri, omanga ndi omanga amakonda zomanga modular zomwe amatha kuzisintha pazogwiritsa ntchito zosiyanasiyana.
- Ukadaulo sunapezekebe: cholinga chomaliza chimaphatikizapo kuphunzira pamakina ambiri ndi luntha lochita kupanga. Nyumba za m'nyanja zisanayambe kugwira ntchito monga momwe amaganizira, matekinolojewa ayenera kutukuka kwambiri.
- Palibe kupita patsogolo kwakukulu pamapangidwe omwe alipo: Padakali kukayikira kwakukulu ponena za kuchuluka kwa ndalama zomwe nyumba za m’nyanja zingapereke. Otsutsa ena amanena kuti pulani ya nyumba yosungiramo zinthu m'nyanja yolumikizidwa ndi zida zoyenera zodzichitira imagwira ntchito bwino mofananamo.
Zovuta za Data Lakehouse
Zitha kukhala zovuta kugwiritsa ntchito njira ya data Lakehouse. Chifukwa cha zovuta za zigawo zake, sikulakwa kuona malo osungiramo data ngati malo abwino ophatikizanapo kapena "pulatifomu imodzi ya chirichonse," imodzi.
Kuphatikiza apo, chifukwa chakuchulukirachulukira kwa ma data, mabizinesi akuyenera kusamutsa malo awo osungiramo data kwa iwo, kudalira lonjezo lopambana popanda phindu lililonse pazachuma.
Ngati pali vuto lililonse lachikumbutso kapena kuzimitsidwa panthawi yonseyi, izi zitha kukhala zodula, zowononga nthawi, mwinanso zosatetezeka.
Ogwiritsa ntchito mabizinesi akuyenera kutsata umisiri wapadera kwambiri, malinga ndi mavenda ena omwe amagulitsa momveka bwino kapena mosabisa ngati nkhokwe za data. Izi sizingagwire ntchito nthawi zonse ndi zida zina zolumikizidwa kunyanja ya data pakatikati pa dongosolo, ndikuwonjezera zovuta.
Kuphatikiza apo, zitha kukhala zovuta kupereka ma analytics 24/7 pomwe mukuyendetsa ntchito zovuta kwambiri zamabizinesi, zomwe zimafuna kuti zomangamanga zikhale zotsika mtengo.
Kutsiliza
Mitundu yatsopano kwambiri yama data m'zaka zaposachedwa ndi data lakehouse. Zimaphatikiza magawo osiyanasiyana, monga ukadaulo wazidziwitso, mapulogalamu otseguka, mtambo kompyuta, ndi kugawa ma protocol osungira.
Zimathandizira mabizinesi kuti azisunga pakati mitundu yonse ya data pamalo aliwonse, kumathandizira kasamalidwe ndi kusanthula mosavuta. Data Lakehouse ndi lingaliro lochititsa chidwi kwambiri.
Kampani iliyonse ingakhale ndi mpikisano wofunika kwambiri ngati ikanakhala ndi mwayi wopita ku nsanja ya data-in-one yomwe inali yachangu komanso yogwira mtima ngati malo osungiramo zinthu komanso kukhala yosinthika ngati nyanja ya data.
Lingaliro likukulabe ndipo likadali latsopano. Chifukwa chake, zingatenge nthawi kuti mudziwe ngati china chake chingakhale chofala kapena ayi.
Tonse tiyenera kukhala ndi chidwi chofuna kudziwa komwe kamangidwe ka Lakehouse akulowera.
Siyani Mumakonda