Cov Ntaub Ntawv Kev Tshawb Fawb thiab cov kws tshaj lij kev kawm tshuab suav nrog ntau cov ntaub ntawv tseem ceeb ntawm ntau hom hauv cov ntaub ntawv tshawb fawb ib txwm ua. Ntau tus qauv tau tsim nrog ntau yam kev teeb tsa thiab cov yam ntxwv, nrog rau ntau qhov kev hloov pauv ntawm qhov ntsuas ntsuas kom tau txais kev pom zoo.
Hauv qhov xwm txheej zoo li no, txhua qhov kev hloov kho cov ntaub ntawv thiab cov qauv tsim cov txheej txheem kev hloov kho yuav tsum tau saib xyuas thiab ntsuas txhawm rau txiav txim siab seb qhov ua haujlwm twg thiab qhov tsis ua haujlwm. Nws kuj tseem ceeb heev kom muaj peev xwm rov qab mus rau ib tsab ntawv dhau los thiab saib rau cov txiaj ntsig yav dhau los.
Data Version Control (DVC), uas pab tswj cov ntaub ntawv, tus qauv hauv qab, thiab khiav cov txiaj ntsig rov ua dua, yog ib qho kev siv tshuab uas ua rau peb saib xyuas txhua yam ntawm no.
Hauv cov ntawv tshaj tawm no, peb yuav saib ze rau hauv Cov Ntaub Ntawv Version Tswj, thiab cov cuab yeej zoo tshaj plaws los siv. Cia peb pib.
Data Version Control yog dab tsi?
Versioning yog yuav tsum tau rau tag nrho cov ntau lawm systems. Ib qho taw tes ntawm kev nkag mus rau cov ntaub ntawv tshiab tshaj plaws. Ib qho kev pab cuam uas feem ntau hloov kho, tshwj xeeb tshaj yog los ntawm ntau tus neeg siv tib lub sijhawm, xav tau kev tsim txoj hauv kev tshawb xyuas kom taug qab txhua qhov kev hloov pauv.
Lub version tswj system yog lub luag haujlwm los xyuas kom meej tias txhua tus hauv pab pawg nyob rau tib nplooj ntawv. Nws lav tias txhua tus neeg hauv pab pawg ua haujlwm ntawm cov ntaub ntawv tsis ntev los no thiab, qhov tseem ceeb tshaj, txhua tus neeg koom tes ntawm tib txoj haujlwm ib zaug.
Yog tias koj muaj cov cuab yeej tsim nyog, koj tuaj yeem ua tiav qhov no nrog kev siv zog tsawg!
Koj yuav muaj cov ntaub ntawv sib xws thiab cov ntaub ntawv ntxaws ntxaws ntawm tag nrho koj cov kev tshawb fawb yog tias koj siv cov ntaub ntawv muaj txiaj ntsig kev tswj hwm cov tswv yim. Cov ntaub ntawv versioning cov cuab yeej yog qhov tseem ceeb rau koj qhov kev ua haujlwm yog tias koj mob siab txog kev rov tsim dua tshiab, traceability, thiab ML qauv keeb kwm.
Lawv pab koj kom tau txais ib qho version ntawm ib yam khoom, zoo li hash ntawm dataset lossis qauv, uas koj tuaj yeem siv los txheeb xyuas thiab sib piv. Cov ntaub ntawv no feem ntau nkag mus rau hauv koj cov kev tswj hwm metadata kom lav tias koj tus qauv kev cob qhia yog versioned thiab rov ua dua.
Qhov zoo tshaj plaws Cov Ntaub Ntawv Version Tswj cov cuab yeej
Tam sim no nws yog lub sijhawm los saib cov ntaub ntawv zoo tshaj plaws version tswj cov kev daws teeb meem muaj, uas koj tuaj yeem siv los taug qab txhua feem ntawm koj cov cai.
1. Git LFS cov
Qhov project Git LFS yog siv dawb. Hauv Git, cov ntaub ntawv loj xws li cov qauv suab, cov yeeb yaj kiab, cov ntaub ntawv khaws cia, thiab cov duab tau hloov pauv nrog cov ntawv taw qhia, thiab cov ntsiab lus ntawm cov ntaub ntawv tau txais kev cawmdim ntawm cov chaw taws teeb tswj xws li GitHub.com lossis GitHub Enterprise.
Nws tso cai rau koj siv Git rau cov ntaub ntawv loj heev - txog li ob peb GB hauv qhov loj - tuav ntau dua hauv koj qhov chaw khaws cia Git siv cov chaw cia sab nraud, thiab clone thiab khaws cov ntaub ntawv loj dua sai dua. Thaum nws los txog rau kev tswj cov ntaub ntawv, qhov no yog ib qho kev daws teeb meem zoo nkauj. Txhawm rau ua haujlwm nrog Git, koj tsis tas yuav muaj cov lus txib ntxiv, lub kaw lus cia, lossis cov khoom siv.
Nws txwv qhov ntau ntawm cov ntaub ntawv koj rub tawm. Qhov no txhais tau hais tias cloning thiab retrieving loj cov ntaub ntawv los ntawm repositories yuav sai dua. Cov taw tes yog ua los ntawm cov khoom siv sib zog thiab taw tes rau LFS.
Yog li ntawd, thaum koj thawb koj cov repo mus rau hauv lub chaw cia khoom loj, nws hloov kho sai thiab siv qhov chaw tsawg dua.
pros
- Yooj yim integrates rau hauv txoj kev loj hlob workflows ntawm feem ntau cov lag luam.
- Tsis tas yuav tswj hwm txoj cai ntxiv vim nws siv tib qhov kev tso cai raws li Git repository.
cons
- Git LFS yuav tsum siv cov servers tshwj xeeb los khaws koj cov ntaub ntawv. Yog li ntawd, koj cov ntaub ntawv tshawb fawb pawg yuav raug kaw, thiab koj cov haujlwm engineering yuav nce.
- Tshwj xeeb heev, thiab yuav tsum tau siv ntau yam cuab yeej sib txawv rau cov theem tom ntej hauv cov ntaub ntawv tshawb fawb txog kev ua haujlwm.
pricing
Nws yog pub dawb rau siv rau txhua tus.
2. LakeFS
LakeFS yog ib qho kev qhib cov ntaub ntawv versioning kev daws teeb meem uas khaws cov ntaub ntawv hauv S3 lossis GCS thiab muaj Git-zoo li branching thiab ua raws li cov qauv uas ntsuas rau petabytes.
Lub tswv yim kev ua haujlwm no ua rau koj cov ntaub ntawv pas dej ACID ua raws los ntawm kev tso cai hloov pauv tshwm sim hauv cov ceg sib txawv uas tuaj yeem tsim, sib koom ua ke, thiab dov rov qab atomically thiab instantaneously.
LakeFS ua rau pab pawg los tsim cov ntaub ntawv hauv pas dej ua si uas rov ua dua, atomic, thiab versioned. Nws yog ib tug newbie rau scene, tab sis nws yog ib tug quab yuam kom reckoned nrog.
Nws siv Git-zoo li branching thiab version tswj txoj hauv kev los cuam tshuam nrog koj data pas dej, scalable mus txog Petabytes ntawm cov ntaub ntawv. Ntawm qhov ntsuas exabyte, koj tuaj yeem tshawb xyuas qhov kev tswj hwm version.
pros
- Git-zoo li kev ua haujlwm suav nrog kev txiav, cog lus, sib koom ua ke, thiab rov qab los.
- Pre-commit/merge hooks yog siv rau cov ntaub ntawv CI/CD checks.
- Muab cov yam ntxwv nyuaj xws li ACID kev hloov pauv rau huab cia yooj yim xws li S3 thiab GCS, tag nrho thaum cov ntawv tseem nyob nruab nrab.
- Rov qab hloov pauv rau cov ntaub ntawv hauv lub sijhawm tiag tiag.
- Scales tau yooj yim, tso cai rau nws kom haum rau cov ntaub ntawv loj heev. Kev tswj hwm version tuaj yeem muab rau ob qho tib si kev txhim kho thiab kev tsim khoom.
cons
- LakeFS yog cov khoom tshiab, yog li kev ua haujlwm thiab cov ntaub ntawv yuav hloov pauv sai dua li cov kev daws teeb meem dhau los.
- Txij li thaum nws tau tsom mus rau cov ntaub ntawv versioning, koj yuav tsum tau siv ntau yam cuab yeej ntxiv rau ntau qhov chaw ntawm cov ntaub ntawv tshawb fawb ua haujlwm.
pricing
Nws yog pub dawb rau siv rau txhua tus.
3. DVC
Cov Ntaub Ntawv Version Tswj yog cov ntaub ntawv dawb versioning kev daws teeb meem tsim los rau cov ntaub ntawv tshawb fawb thiab kev kawm tshuab. Nws yog ib qho kev pab cuam uas tso cai rau koj los txhais koj cov kav dej hauv txhua hom lus.
Los ntawm kev tswj cov ntaub ntawv loj, cov ntaub ntawv teev, cov qauv kev kawm tshuab, cov lej, thiab lwm yam, cov cuab yeej ua rau cov qauv kev kawm sib faib thiab rov tsim dua. Qhov kev zov me nyuam ua raws Git tus thawj coj hauv kev muab cov kab hais kom yooj yim uas tuaj yeem teeb tsa hauv ob peb kauj ruam xwb.
Raws li nws lub npe implies, DVC tsis yog tsuas yog hais txog cov ntaub ntawv versioning. Nws kuj pab tswj kev tswj cov kav dej thiab cov qauv kev kawm tshuab rau pab pawg.
Thaum kawg, DVC yuav pab txhim kho qhov sib xws ntawm koj pab neeg cov qauv thiab lawv rov ua dua. Es tsis txhob siv cov ntaub ntawv nyuaj thiab cov lus hauv cov cai, coj kom zoo dua Git ceg sim tawm tswv yim tshiab. Mus ncig teb chaws, ntiav automated metric-tracking es tsis txhob ntawm daim ntawv thiab xaum.
Txhawm rau xa cov pob khoom sib xws ntawm tshuab kev kawm cov qauv, cov ntaub ntawv, thiab cov cai rau hauv kev tsim khoom, cov khoos phis tawj nyob deb, lossis ib tus npoj yaig lub desktop, koj tuaj yeem siv thawb / rub cov lus txib tsis yog ad-hoc scripts.
pros
- Nws yog lub teeb yuag, qhib qhov chaw, thiab ua haujlwm nrog txhua qhov loj huab platforms thiab hom cia.
- Slexible, agnostic ntawm hom ntawv thiab lub moj khaum, thiab yooj yim rau siv.
- Txhua tus qauv ML tag nrho cov evolution tuaj yeem taug qab mus rau nws qhov chaws thiab cov ntaub ntawv.
cons
- Kev tswj kav kav dej thiab DVC version tswj yog inextricably txuas. Yuav muaj kev rov ua dua yog tias koj pab neeg twb tau siv lwm cov ntaub ntawv xa mus.
- Txij li thaum DVC yog lub teeb yuag, koj pab neeg yuav xav tau tsim cov yam ntxwv ntxiv manually kom nws siv tau ntau dua.
pricing
Nws yog pub dawb rau siv rau txhua tus.
4. DeltaLake
DeltaLake yog qhov qhib-qhov chaw cia txheej uas txhawb nqa cov ntaub ntawv pas dej kev ntseeg siab. Delta Lake txhawb nqa ACID kev lag luam thiab kev tswj hwm metadata ntxiv rau streaming thiab batch data processing.
Nws ua haujlwm nrog Apache Spark APIs thiab zaum ntawm koj lub pas dej cov ntaub ntawv uas twb muaj lawm. Delta Sib Koom yog lub ntiaj teb thawj qhov kev cai qhib rau kev nyab xeeb cov ntaub ntawv sib koom hauv kev lag luam, ua kom yooj yim los pauv cov ntaub ntawv nrog lwm cov lag luam ywj pheej ntawm lawv lub khoos phis tawj.
Delta Lakes muaj peev xwm tuav petabytes ntawm cov ntaub ntawv yooj yim. Metadata yog khaws cia tib yam li cov ntaub ntawv, thiab cov neeg siv tuaj yeem tau txais nws los ntawm kev piav qhia nthuav dav txoj kev. Delta Lakes muaj ib qho qauv tsim uas tuaj yeem nyeem ob qho kwj thiab cov ntaub ntawv batch.
Upserts yog qhov yooj yim ua siv Delta. Cov upserts lossis merges rau hauv lub rooj Delta yog piv rau SQL Merges. Koj tuaj yeem siv nws los ua ke cov ntaub ntawv los ntawm lwm cov ntaub ntawv thav duab rau hauv koj lub rooj thiab ua cov hloov tshiab, ntxig, thiab tshem tawm.
pros
- Ntau lub peev xwm, xws li ACID kev lag luam thiab kev tswj hwm cov ntaub ntawv muaj zog, tuaj yeem muaj nyob hauv koj cov ntaub ntawv khaws cia tam sim no.
- Tam sim no Delta Lake tuaj yeem siv zog tswj cov ntxhuav nrog ntau lab ntawm kev faib thiab cov ntaub ntawv ntawm petabyte-scale.
- Txo qhov xav tau ntawm phau ntawv qhia cov ntaub ntawv version tswj thiab lwm yam kev txhawj xeeb ntawm cov ntaub ntawv, tso cai rau cov neeg tsim khoom kom mloog zoo rau kev tsim cov khoom nyob rau sab saum toj ntawm lawv cov ntaub ntawv pas dej.
cons
- Raws li nws tau tsim los ua haujlwm nrog Spark thiab cov ntaub ntawv loj, Delta Lake feem ntau overkilled rau feem ntau cov dej num.
- Nws tsim nyog siv cov ntaub ntawv tshwj xeeb, uas txwv nws qhov yooj yim thiab ua rau nws tsis sib haum nrog koj cov ntaub ntawv tam sim no.
pricing
Nws yog pub dawb rau siv rau txhua tus.
5. Dolt
Dolt yog SQL database uas ua forking, cloning, branching, merging, thawb, thiab rub tib yam li git repository ua. Txhawm rau txhim kho cov neeg siv kev paub dhau los ntawm cov ntaub ntawv tswj hwm version, Dolt tso cai rau cov ntaub ntawv thiab cov qauv hloov pauv hauv sync.
Nws yog ib qho cuab yeej zoo rau koj thiab koj cov neeg ua haujlwm sib koom tes. Koj tuaj yeem txuas rau Dolt tib txoj kev uas koj xav mus rau lwm yam MySQL database thiab khiav cov lus nug lossis hloov cov ntaub ntawv siv SQL cov lus txib.
Thaum nws los txog rau cov ntaub ntawv versioning, Dolt yog ib qho ntawm-ib-zoo. Dolt yog ib qho chaw khaws ntaub ntawv, tsis zoo li qee qhov kev daws teeb meem uas tsuas yog cov ntaub ntawv version. Thaum lub software tam sim no nyob rau hauv nws cov theem pib, muaj kev cia siab yuav ua rau nws tag nrho sib xws nrog Git thiab MySQL nyob rau hauv lub neej yav tom ntej.
Tag nrho cov lus txib uas koj paub siv nrog Git kuj tseem ua haujlwm nrog Dolt. Git versions files, Dolt versions tables Siv cov kab hais kom ua interface, import CSV cov ntaub ntawv, ua koj cov kev hloov pauv, tshaj tawm rau cov chaw taws teeb, thiab sib koom ua ke koj cov phooj ywg kev hloov pauv.
pros
- Lub teeb thiab Qhib qhov chaw hauv ib feem.
- Hauv kev sib piv rau ntau qhov kev xaiv tsis meej, nws muaj SQL interface, ua rau nws nkag mus rau cov ntaub ntawv txheeb xyuas ntau dua.
cons
- Hauv kev sib piv rau lwm qhov kev hloov pauv database, Dolt tseem yog cov khoom tsim.
- Txij li thaum Dolt yog database, koj yuav tsum hloov koj cov ntaub ntawv rau hauv nws kom tau txais txiaj ntsig.
pricing
Zoo siab txais tos txhua tus tuaj koom kev sib tham hauv zej zog. Lub platform tsis muab tus nqi hwm; Hloov chaw, koj yuav tsum hu rau tus kws kho mob.
6. Pachyderm
Pachyderm yog cov ntaub ntawv tshawb fawb dawb version tswj qhov system nrog ntau yam nta. Pachyderm Enterprise yog cov ntaub ntawv tshawb fawb muaj zog uas tsim los rau kev sib koom tes loj hauv qhov chaw muaj kev nyab xeeb heev.
Pachyderm yog ib qho ntawm ob peb cov ntaub ntawv tshawb fawb platforms. Pachyderm lub hom phiaj yog muab lub platform uas tswj hwm cov ntaub ntawv ua tiav thiab ua kom yooj yim rau kev luam tawm cov kev tshawb pom ntawm cov qauv kev kawm tshuab. Pachyderm yog hu ua "Docker of Data" nyob rau hauv cov ntsiab lus no. Pachyderm ntim koj qhov chaw ua haujlwm siv Docker ntim. Qhov no ua rau nws yooj yim rau duplicate tib cov txiaj ntsig.
Cov kws tshawb fawb cov ntaub ntawv thiab DevOps pab pawg tuaj yeem xa cov qauv nrog kev ntseeg siab ua tsaug rau kev sib xyaw ntawm cov ntaub ntawv versioned nrog Docker. Ua tsaug rau qhov ua tau zoo khaws cia, petabytes ntawm cov ntaub ntawv tsim thiab tsis muaj txheej txheem tuaj yeem khaws cia thaum cov nqi khaws cia kom tsawg.
Thoob plaws hauv cov kav dej theem, cov ntaub ntawv-raws li versioning muab cov ntaub ntawv txheeb xyuas tag nrho rau txhua cov ntaub ntawv thiab cov khoom qub, suav nrog cov txiaj ntsig nruab nrab. Ntau lub cuab yeej muaj peev xwm tau tsav los ntawm cov ncej no, uas pab pawg kom tau txais txiaj ntsig zoo tshaj plaws ntawm nws.
pros
- Raws li cov thawv ntim khoom, koj cov ntaub ntawv ib puag ncig yuav txav tau yooj yim thiab yooj yim rau kev hloov ntawm huab cua.
- Robust, nrog lub peev xwm los ntsuas los ntawm me me mus rau qhov loj heev.
cons
- Txij li thaum muaj ntau yam txav mus los, xws li Kubernetes server tsim nyog los tswj Pachyderm cov ntawv pub dawb, muaj qhov kev kawm nruj dua.
- Pachyderm tej zaum yuav nyuaj rau kev koom ua ke rau hauv ib lub tuam txhab cov txheej txheem uas twb muaj lawm vim nws muaj ntau yam cuab yeej technology.
pricing
Koj tuaj yeem pib siv lub platform nrog kev sib tham hauv zej zog thiab rau kev lag luam tsab, koj yuav tsum hu rau tus neeg muag khoom.
7. Neptune
Tus qauv tsim metadata yog tswj hwm los ntawm ML metadata khw, uas yog ib qho tseem ceeb ntawm MLOps pawg. Rau txhua qhov kev ua haujlwm ntawm MLOps, Neptune ua haujlwm hauv nruab nrab ntawm cov ntaub ntawv khaws cia.
Koj tuaj yeem taug qab, pom, thiab sib piv ntau txhiab tus qauv kev kawm tshuab tag nrho hauv ib qho chaw. Nws suav nrog cov yam ntxwv xws li kev sim taug qab, qauv sau npe, thiab kev saib xyuas tus qauv, nrog rau kev sib koom tes sib tham. Nws suav nrog ntau dua 25 cov cuab yeej sib txawv thiab cov tsev qiv ntawv sib xyaw ua ke, suav nrog ntau tus qauv kev cob qhia thiab cov cuab yeej ntsuas hyperparameter.
Koj tuaj yeem koom nrog Neptune yam tsis siv koj daim npav rho nyiaj. Ib tus account Gmail yuav txaus rau nws qhov chaw.
pros
- Kev koom ua ke nrog cov kav dej, ntws, codebase, lossis lub moj khaum yog qhov yooj yim.
- Kev pom lub sijhawm tiag tiag, API yooj yim, thiab kev txhawb nqa ceev
- Nrog Neptune, koj tuaj yeem ua "thaum" ntawm tag nrho koj cov kev sim 'cov ntaub ntawv hauv ib qho chaw, uas koj tuaj yeem rov qab tau tom qab.
cons
- Txawm hais tias tsis yog qhov qhib nkaus xwb, ib tus neeg version yuav xav tias txaus rau kev siv ntiag tug, txawm hais tias kev nkag mus tau txwv rau ib hlis.
- Muaj ob peb qhov kev tsim me me kom pom.
pricing
Koj tuaj yeem pib siv lub platform nrog tus kheej txoj kev npaj pub dawb rau txhua tus. Nqe nqe pib los ntawm $150/hli.
xaus
Hauv tsab ntawv no, peb tau tham txog cov ntaub ntawv zoo tshaj plaws versioning cov cuab yeej. Txhua lub cuab yeej, raws li peb tau pom, muaj nws tus kheej cov yam ntxwv. Qee tus tau dawb, thaum lwm tus yuav tsum tau them nyiaj. Qee qhov zoo rau cov qauv kev lag luam me, thaum lwm tus zoo dua rau cov qauv kev lag luam loj.
Raws li qhov tshwm sim, koj yuav tsum xaiv cov software zoo tshaj plaws rau koj lub hom phiaj tom qab ntsuas qhov zoo thiab qhov tsis zoo. Peb txhawb kom koj sim tawm qhov kev sim dawb version ua ntej yuav khoom hwm.
Sau ntawv cia Ncua