Kiyaye Haske na Cikin Gida na HDR a Lokaci da Wuri: Tsarin Koyon Zurfi don AR Mai Kama da Gaskiya

1. Gabatarwa

Hasashen haske mai inganci, mai daidaitawa shine ginshiƙi don aikace-aikacen Ƙarfafa Gaskiya (AR) masu kama da gaskiya kamar haɓaka wuri da kasancewa ta nesa. Takardar "Kiyaye Haske na Cikin Gida na HDR a Lokaci da Wuri" ta magance babban ƙalubalen hasashen haske daga cikakkun bayanai marasa yawa, waɗanda ba su cika ba na na'urorin hannu—galibi hoto ɗaya ne kawai na Low Dynamic Range (LDR) wanda ya rufe kusan kashi 6% na wurin da ake kallo. Matsala ta asali ita ce hasashen bayanan High Dynamic Range (HDR) da suka ɓace da sassan wurin da ba a iya gani ba (kamar hanyoyin haske da ke wajen firam) yayin da ake tabbatar da cewa hasashen sun yi daidai a wurare daban-daban na sarari a cikin hoto da kuma a cikin jerin bidiyo na tsawon lokaci. Wannan aikin ya ba da shawarar tsarin farko don cimma wannan daidaitawar biyu, yana ba da damar zane na abubuwa na zahiri tare da kayan aiki masu sarkakiya kamar madubi da saman da ke haskakawa.

2. Hanyoyi

Tsarin da aka ba da shawara tsari ne na cibiyar sadarwa mai zurfi, wanda aka tsara don hasashen haske daga hoton LDR (da zurfin da za a iya zaɓi) ko jerin bidiyon LDR.

2.1. Girman Haske na Gaussian Mai Siffar Kwalliya (SGLV)

Wakilcin asali shine girma na 3D inda kowane voxel ke adana sigogi don saitin Gaussians masu siffar kwalliya (SGs), waɗanda su ne ingantacciyar kusantar haske mai sarkakiya. An ayyana SG kamar haka: $G(\mathbf{v}; \mathbf{\mu}, \lambda, a) = a \cdot e^{\lambda(\mathbf{\mu} \cdot \mathbf{v} - 1)}$, inda $\mathbf{\mu}$ shine axis na lobe, $\lambda$ shine kaifin lobe, kuma $a$ shine girman lobe. SGLV yana wakiltar filin haske a cikin sararin samaniya na 3D na wurin.

2.2. Tsarin Maɓalli-Mai Fassara 3D

Cibiyar sadarwa ta 3D da aka keɓance tana ɗaukar hoton LDR na shigarwa (da taswirar zurfi, idan akwai) kuma ta gina SGLV. Maɓallin yana fitar da siffofi masu ma'auni daban-daban, waɗanda mai fassara ke amfani da su don haɓaka samfurin a hankali da hasashen sigogin SG (axis, kaifi, girman) ga kowane voxel a cikin girma.

2.3. Binciken Hasken Girman Don Daidaitawar Sarari

Don hasashen haske a kowane matsayi na hoto (misali, inda aka sanya abu na zahiri), tsarin yana aiwatar da binciken hasken girma ta hanyar SGLV. Don wani batu na 3D da alkiblar kallo, yana ɗaukar samfurin SGLV tare da hasken kuma yana tattara sigogin SG. Wannan yana tabbatar da cewa hasashen haske suna da tushe a zahiri kuma suna bambanta a hankali kuma suna daidaitawa a wuraren sarari, suna mutunta tsarin wurin.

2.4. Cibiyar Sadarwa Mai Haɗaɗɗu Don Taswirorin Muhalli

An fassara sigogin SG da aka bincika zuwa cikakken taswirar muhalli na HDR. Cibiyar sadarwa mai haɗaɗɗu tana haɗa hasashe mai ƙarfi, mai daidaitawa daga SGLV tare da cikakkun bayanai masu yawan mitar da aka koya don samar da taswirar muhalli ta ƙarshe wacce ta haɗa da cikakkun haskoki da hanyoyin haske da ba a iya gani ba.

2.5. Layer na Monte-Carlo Mai Zane a Cikin Cibiyar Sadarwa

An haɗa Layer na Monte-Carlo mai banbanta a cikin tsarin horo. Yana zana abubuwa na zahiri tare da hasken da aka hasashen kuma yana kwatanta sakamakon da zane na gaskiya. Wannan asarar hoto zuwa ƙarshe tana daidaitawa don manufa ta ƙarshe—shigar da abu mai kama da gaskiya—kuma tana ba da siginar kulawa mai ƙarfi, mai kama da asarar adawa da daidaitawar zagayowar da suka tura samfuran fassarar hoto zuwa hoto kamar CycleGAN [Zhu et al., 2017].

2.6. Cibiyoyin Sadarwar Juyawa Don Daidaitawar Lokaci

Lokacin da shigarwar ta kasance jerin bidiyo, ana amfani da na'urar Cibiyar Sadarwar Juyawa (RNN). Tana kiyaye yanayin ɓoye wanda ke tattara bayanai daga firam ɗin da suka gabata. Wannan yana ba da damar tsarin don gyara a hankali kimantawar haskensa yayin da yake lura da ƙarin wurin a kan lokaci, yayin da ƙwaƙwalwar RNN ke tabbatar da gyaran yana sauƙi kuma yana daidaitawa a lokaci, yana guje wa ƙyalli ko tsalle-tsalle a cikin hasken da aka hasashen.

3. Ƙarfafa Bayanan OpenRooms

Don horar da irin wannan samfurin mai buƙatar bayanai, marubutan sun ƙarfafa bayanan OpenRooms na jama'a sosai. Ƙarfaffiyar sigar ta haɗa da kusan taswirorin muhalli 360,000 na HDR a mafi girman ƙuduri da jerin bidiyo 38,000, duk an zana su ta amfani da hanyar bin hanya mai saurin GPU don daidaiton zahiri. Wannan babban bayanan na roba mai inganci ya kasance mahimmanci ga nasarar samfurin.

Ƙididdiga na Bayanai

Taswirorin Muhalli na HDR: ~360,000
Jerin Bidiyo: ~38,000
Hanyar Zane: Binciken Hanya na Tushen GPU
Amfani na Farko: Horarwa & Ƙididdiga na Samfuran Hasashen Haske na Cikin Gida

4. Gwaje-gwaje & Sakamako

4.1. Kimantawa ta Ƙididdiga

An kimanta tsarin a kan mafi kyawun hanyoyin hasashen haske na hoto ɗaya da na tushen bidiyo ta amfani da ma'auni na yau da kullun kamar Kuskuren Matsakaicin Matsakaici (MSE) da Fihirisar Kamanceceniya ta Tsari (SSIM) akan taswirorin muhalli na HDR, da kuma ma'auni na fahimta akan shigar da abubuwan da aka zana. Hanyar da aka ba da shawara ta ci gaba da fi duk abubuwan da suka gabata a cikin hasashen haske daidai, a sarari da kuma a lokaci.

4.2. Kimantawa ta Halayya & Sakamako na Gani

Kamar yadda aka nuna a Hoto na 1 na takarda, hanyar ta sami nasarar dawo da dukansu hanyoyin haske da ake iya gani da waɗanda ba a iya gani ba da cikakkun haskoki daga saman da ake iya gani. Wannan yana ba da damar shigar da abubuwa na zahiri masu ƙalubalantar kayan aiki. Don shigarwar bidiyo, sakamakon yana nuna ci gaba mai santsi da kwanciyar hankali a kan lokaci, ba tare da ƙyalli ba.

Bayanin Chati/Hoto (Dangane da Fig. 1 & 2): Hoto na 1 yana ba da taƙaitaccen bayani mai gani, yana kwatanta shigar da abubuwa ta amfani da haske daga hanyoyi daban-daban. Sakamakon marubutan sun nuna daidaitattun haskoki na musamman, inuwa mai laushi, da zubar da launi wanda ya dace da wurin na gaskiya, sabanin abokan hamayyarsu waɗanda shigarwar su ta bayyana a fili, launi ba daidai ba, ko rashin inuwa mai daidaituwa. Hoto na 2 yana kwatanta tsarin tsarin gabaɗaya, yana nuna kwarara daga hoto/zurfin shigarwa zuwa SGLV, ta hanyar binciken haske da cibiyar sadarwar haɗawa, zuwa taswirar muhalli ta HDR ta ƙarshe da abin da aka zana.

4.3. Nazarin Cirewa

Nazarin cirewa ya tabbatar da mahimmancin kowane ɓangare: cire SGLV da binciken hasken girma ya cutar da daidaitawar sarari; cire mai zane a cikin cibiyar sadarwa ya rage kamannin gaskiya na shigarwa; kuma kashe RNN ya haifar da hasashe marasa daidaitawa a lokaci, masu ƙyalli a cikin bidiyo.

5. Nazarin Fasaha & Fahimta ta Asali

Fahimta ta Asali

Wannan takarda ba wani ƙarin ci gaba ne kawai a cikin hasashen haske ba; canji ne na tsari zuwa ga ɗaukar haske a matsayin filin lokaci da sarari maimakon panorama mai tsayayye, mai zaman kanta. Marubutan sun gano daidai cewa don AR ya ji "gaskiya," dole ne abubuwa na zahiri su yi hulɗa da haske akai-akai yayin da mai amfani ko abu ke motsawa. Fahimtarsu ta asali ita ce amfani da wakilcin hasken girma na 3D (SGLV) a matsayin babban tsarin bayanan matsakaici. Wannan shine babban nasara—yana haɗa gibin tsakanin yankin hoto na 2D da duniyar zahiri ta 3D, yana ba da damar tunani na sarari ta hanyar binciken haske da daidaitawar lokaci ta hanyar ƙirar jerin gwano. Ya wuce iyakokin hanyoyin da ke daidaita taswirar muhalli kai tsaye daga CNN na 2D, waɗanda a zahiri ke fama da haɗin kai na sarari.

Kwararar Ma'ana

Ma'anar gine-gine tana da kyau kuma tana bin tsarin simintin zahiri bayyananne, wanda shine dalilin da ya sa yake aiki da kyau: Shigarwar 2D -> Fahimtar Wuri na 3D (SGLV) -> Tambayar Zahiri (Binciken Hasken) -> Fitowar 2D (Taswirar Muhalli/Zane). Maɓallin-mai fassara na 3D yana gina samfurin ɓoyayye na rarraba hasken wurin. Mai aikin binciken hasken girma yana aiki azaman hanyar tambaya mai banbanta, mai sanin tsarin jiki. Cibiyar sadarwa mai haɗaɗɗu tana ƙara cikakkun bayanai masu yawan mitar da suka ɓace a cikin rarrabuwar girma. A ƙarshe, mai zanen Monte-Carlo a cikin cibiyar sadarwa yana rufe madauki, yana daidaita manufar koyo tare da aikin fahimta na ƙarshe. Don bidiyo, RNN kawai tana sabunta wakilcin 3D na ɓoye a kan lokaci, yana mai da daidaitawar lokaci a matsayin sakamako na halitta.

Ƙarfi & Kurakurai

Ƙarfi: Cimma daidaitawar biyu alama ce. Amfani da wakilcin tushen zahiri (SGLV+Binciken Hasken) yana ba shi ra'ayi mai ƙarfi, yana haifar da ƙarin gamawa fiye da hanyoyin da aka yi amfani da bayanai kawai. Ƙarfafa bayanan OpenRooms babbar gudummawa ce ga al'umma. Haɗa asarar zane yana da wayo, kamar horon "sanin aiki" da ake gani a cikin samfuran hangen nesa na zamani.

Kurakurai & Tambayoyi: Giwa a cikin ɗaki shine farashin lissafi. Gina da tambayar girma na 3D yana da nauyi. Duk da yana yiwuwa ga bincike, aikin lokaci-lokaci akan na'urorin AR na hannu ya kasance babban cikas. Dogaro da bayanan roba (OpenRooms) takobi ne mai kaifi biyu; yayin da yake ba da cikakkiyar gaskiya, gibin sim-zuwa- gaskiya don rikitattun cikin gida na duniyar gaskiya ba a tabbatar da shi ba. Hanyar kuma tana ɗauka cewa akwai taswirar zurfi, wanda ke ƙara dogaro ga wani firikwensin ko algorithm na kimantawa. Ta yaya yake aiki tare da amo ko rashin zurfi?

Fahimta Mai Aiki

1. Ga Masu Bincike: Ra'ayin SGLV yana cikin lokacin bincike. Za a iya sanya shi ya fi dacewa tare da wakilci mara yawa ko tsarin matsayi? Za a iya daidaita wannan tsarin don hasashen haske na waje? 2. Ga Injiniyoyi/Ƙungiyoyin Samfura: Aikace-aikacen nan take shine a cikin ƙirar abun ciki na AR mai inganci da gani na ƙwararru. Don AR na hannu na mabukaci, yi la'akari da tsarin matakai biyu: mai hasashe mai sauƙi, mai sauri don bin diddigin lokaci-lokaci, da wannan hanyar azaman sabis na baya don samar da ingantattun tasirin kama da gaskiya lokacin da mai amfani ya dakata. 3. Dabarar Bayanai: Nasarar ta jaddada buƙatar babban bayanai, mai inganci mai lakabi a cikin hangen nesa na zane. Zuba jari a cikin kayan aiki don samar da bayanan roba mai inganci (yunkuri da Omniverse na NVIDIA da sauransu ke goyan baya) yana da mahimmanci don ci gaban fagen. 4. Haɗin Ƙirar Kayan Aiki: Wannan aikin yana tura iyakar abin da ake buƙata don AR mai gaskatawa. Alama ce bayyananna ga masu yin guntu (Apple, Qualcomm) cewa zane na jijiyoyi akan na'ura da ikon fahimtar 3D ba abin al'ajabi ba ne amma larura ne ga ƙarni na gaba na gogewar AR.

A ƙarshe, wannan takarda ta kafa sabon matsayi na zamani ta hanyar magance ƙalubalen daidaitawa na asali. Mataki ne mai mahimmanci daga haske "mai kyau" zuwa hasken da zai iya yaudarar ido a cikin yanayin AR mai motsi. Ƙalubalen da suka rage galibi injiniya ne: inganci, ƙarfi ga bayanan duniyar gaskiya, da haɗin kai cikin tsarin na'ura.

6. Misalan Aikace-aikace & Tsarin Aiki

Misalin Harka: Sanya Kayan Daki na Zahiri a cikin AR

App ɗin ƙirar cikin gida yana amfani da wannan tsarin. Mai amfani yana nuna kwamfutar hannu ko kwamfutar tafi-da-gidanka zuwa kusurwar falo.

Shigarwa: App ɗin yana ɗaukar rafin bidiyo na LDR kuma yana kimanta zurfi ta amfani da LiDAR/firikwensin na'urar.
Sarrafawa: Cibiyar sadarwar tsarin tana sarrafa firam ɗin farko, tana gina SGLV na farko da hasashen muhallin haske na HDR don tsakiyar allon.
Hulɗa: Mai amfani ya zaɓi sofa na zahiri don sanyawa a kusurwar. App ɗin yana amfani da binciken hasken girma don tambayar SGLV a wurin 3D na sofa, yana samun kimantawar haske daidai ga sarari don wannan wuri na musamman (wanda ya yi la'akari da taga da ke kusa da shi wanda ba a iya gani kai tsaye a cikin firam ɗin farko).
Zane: An zana sofa tare da hasken da aka tambaya ta amfani da mai zanen Monte-Carlo, yana nuna daidaitattun inuwa mai laushi daga taga, haskoki na musamman akan sassan fata, da zubar da launi daga kafet ɗin da ke kusa.
Gyara: Yayin da mai amfani yake motsa kwamfutar hannu ko kwamfutar tafi-da-gidanka a cikin ɗaki (jerin bidiyo), RNN tana sabunta SGLV, tana gyara samfurin haske. Bayyanar sofa tana sabunta a hankali kuma akai-akai, tana kiyaye daidaitaccen hulɗar haske daga duk sabbin wuraren kallo ba tare da ƙyalli ba.

Wannan misalin yana nuna fa'idodin asali: daidaitawar sarari (daidaitaccen haske a wurin sofa), daidaitawar lokaci (sabuntawa mai santsi), da kamannin gaskiya (zane na kayan aiki masu sarkakiya).

7. Aikace-aikace na Gaba & Jagorori

Ƙarni na Gaba na AR/VR Kasancewa ta Nesa: Ba da damar hotunan mutane masu kama da gaskiya ko mahalarta nesa don haskaka tare da muhallin gida akai-akai a cikin sadarwar lokaci-lokaci, yana haɓaka nutsuwa sosai.
Fim & Bayan Samar da Wasanni: Ba da damar masu fasahar tasirin gani suyi saurin kimanta da kuma kwafin hasken da aka saita don haɗaɗɗun abubuwan CGI cikin faranti na aiki mai rai, har ma daga ƙayyadaddun fina-finai na tunani.
Hangen Nesa na Gine-gine & Gidaje: Ƙirar tafiya mai hulɗa inda hasken kan kayan daki na zahiri ke sabunta kamannin gaskiya yayin da abokin ciniki ke bincika samfurin 3D na sarari da ba a gama ba.
Robotics & AI mai jiki: Samar da robots tare da ƙarin fahimtar hasken wurin, yana taimakawa wajen gano kayan aiki, kewayawa, da tsarin hulɗa.
Jagororin Bincike na Gaba: 1) Inganci: Binciken tarawa na ilimi, matsawa jijiyoyi na SGLV, ko na'urori masu haɓaka kayan aiki na musamman. 2) Ƙarfi: Horarwa akan bayanan roba-gaskiya masu haɗaɗɗu ko amfani da dabarun kai da kai don haɗa gibin sim-zuwa-gaskiya. 3) Gamawa: Ƙaddamar da tsarin zuwa haske mai motsi (misali, kunna/kashe fitilu, motsa hanyoyin haske) da muhallin waje. 4) Samfuran Haɗin kai: Haɗin kimanta haske, tsarin jiki, da kaddarorin kayan aiki daga bidiyo ta hanyar kai-zuwa-ƙarshe.

8. Nassoshi

Li, Z., Yu, L., Okunev, M., Chandraker, M., & Dong, Z. (2023). Kiyaye Haske na Cikin Gida na HDR a Lokaci da Wuri. ACM Transactions on Graphics (TOG).
Zhu, J.-Y., Park, T., Isola, P., & Efros, A. A. (2017). Fassarar Hotuna zuwa Hotuna marasa Haɗin gwiwa ta amfani da Cibiyoyin Sadarwar Adawa masu Daidaitawar Zagayowar. Proceedings of the IEEE International Conference on Computer Vision (ICCV).
LeGendre, C., Ma, W., Fyffe, G., Flynn, J., Charbonnel, L., Busch, J., & Debevec, P. (2019). DeepLight: Koyon Haskakawa don Haɗaɗɗun Haɗin gwiwa na Hannu marasa Ƙuntatawa. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Bayanan OpenRooms. (n.d.). Bayanan buɗaɗɗe don fahimtar wurin cikin gida. An samo daga gidan yanar gizon hukuma ko ma'ajiyar ilimi.
Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., & Ng, R. (2020). NeRF: Wakiltar Wurare azaman Filayen Haske na Jijiyoyi don Haɗin Kallo. Communications of the ACM. (An ambata don haɗin ra'ayi zuwa wakilcin wuri na 3D).