NieR: Yin Hoton Yanayi Mai Haske Dangane da Tsarin Jiki

1. Gabatarwa & Bayyani

NieR (Yin Hoton Yanayi Mai Haske Dangane da Tsarin Jiki) sabon tsari ne da aka ƙera don magance babban ƙalubalen kwaikwayon haske na gaske a cikin yanayin 3D mai motsi, musamman a cikin yanayin tuƙi mai cin gashin kansa. Hanyoyin gargajiya na 3D Gaussian Splatting, duk da cewa suna da inganci, sau da yawa sun kasa ɗaukar rikitarwar hulɗar haske da kayan aiki, musamman ga saman ƙyalli kamar motoci, wanda ke haifar da kurakurai na gani kamar shafewa da wuce gona da iri. NieR ya gabatar da hanya mai kafa biyu: sashen Rabe-raben Haske (LD) wanda ke raba hasken ƙyalli da na watsawa dangane da tsarin saman, da kuma sashen Haɓaka Matsakaicin Yanayin Jiki (HNGD) wanda ke daidaita yawan Gaussian da sauri don adana cikakkun bayanai na haske. Wannan hanyar tana nufin cike gibi tsakanin saurin yin hoto da daidaiton zahiri.

2. Hanyoyin Tsarin Asali

Tsarin NieR yana haɓaka 3D Gaussian Splatting ta hanyar haɗa ƙa'idodi daga Yin Hoto Dangane da Zahiri (PBR). Babban ƙirƙira yana cikin yadda yake kula da hasken da ke nunawa a matsayin tsari mai rabuwa, wanda bayanan saman lissafi (tsarin jiki) ke jagoranta.

2.1 Sashen Rabe-raben Haske (LD)

Sashen LD yana sake tsara tsarin haɗa launi a cikin 3D Gaussian Splatting. Maimakon amfani da sifa guda ɗaya na launi a kowane Gaussian, yana raba hasken da ke fitowa $L_o$ zuwa sassan ƙyalli $L_s$ da watsawa $L_d$:

$L_o(\omega_o) = k_s \cdot L_s(\omega_o, \mathbf{n}) + k_d \cdot L_d(\mathbf{n})$

inda $\omega_o$ shine alkiblar kallo, $\mathbf{n}$ shine tsarin saman, kuma $k_s$, $k_d$ sune ƙididdiga na nunawa dangane da kayan aiki waɗanda aka gabatar a matsayin sifofi masu koyo. An ƙirƙira sashin ƙyalli a matsayin aiki na tsarin jiki da alkiblar kallo, yana ba shi damar ɗaukar tasirin da ya dogara da kallo kamar hasken fitila akan fentin mota ko titunan jika.

2.2 Haɓaka Matsakaicin Yanayin Jiki (HNGD)

Daidaitaccen 3D Gaussian Splatting yana amfani da dabarar haɓakawa ta tsayayye ko ta dogara da kallo, wanda zai iya zama mara inganci don ɗaukar cikakkun bayanai na haske mai sauri. HNGD tana ba da shawarar haɓakawa mai sanin lissafi. Tana nazarin matakin sararin saman tsarin jiki $\nabla \mathbf{n}$ a fadin yanayin. Yankuna masu babban matakan tsarin jiki (misali, gefuna na abubuwa, saman lanƙwasa tare da haske mai kaifi) suna nuna rikitarwar lissafi da hulɗar haske. A cikin waɗannan yankuna, HNGD tana ƙara yawan Gaussians daidai:

$D_{new} = D_{base} \cdot (1 + \alpha \cdot ||\nabla \mathbf{n}||)$

inda $D_{new}$ shine sabon yawa, $D_{base}$ shine tushen yawa, $\alpha$ shine ma'aunin sikelin, kuma $||\nabla \mathbf{n}||$ shine girman matakin tsarin jiki. Wannan yana tabbatar da cewa albarkatun lissafi suna mai da hankali inda ake buƙatar su mafi yawa don ingancin gani.

3. Cikakkun Bayanai na Fasaha & Tsarin Lissafi

Tsarin ya ginu akan bututun 3D Gaussian Splatting. An ba kowane Gaussian ƙarin sifofi: tsarin saman $\mathbf{n}$, ƙididdiga na nunawa mai ƙyalli $k_s$, da ƙididdiga na watsawa $k_d$. An gyara lissafin yin hoto kamar haka:

$C = \sum_{i \in N} c_i \cdot \alpha_i \cdot \prod_{j=1}^{i-1}(1-\alpha_j)$

inda launin $c_i$ na kowane Gaussian $i$ yanzu ana ƙididdige shi kamar $c_i = k_{s,i} \cdot f_s(\mathbf{n}_i, \omega_o) + k_{d,i} \cdot f_d(\mathbf{n}_i, E_{env})$. A nan, $f_s$ shine kusantar BRDF na ƙyalli (misali, ƙaramin samfurin Cook-Torrance), $f_d$ shine aikin watsawa, kuma $E_{env}$ yana wakiltar bayanan hasken muhalli. Tsarin jiki $\mathbf{n}_i$ ko dai ana sake dawo da shi yayin horo ko kuma an samo shi daga bayanan farko na tsarin motsi.

4. Sakamakon Gwaji & Aiki

Takardar tana kimanta NieR akan ƙalubalantattun bayanan tuƙi mai cin gashin kansa waɗanda ke ɗauke da abubuwa masu motsi da haske mai rikitarwa (misali, hasken rana kai tsaye, fitilolin mota da dare).

Mahimman Alamomin Aiki (An Ruwaito vs. SOTA)

Matsakaicin Sigina zuwa Ƙara (PSNR): NieR ya sami matsakaicin ci gaba na ~1.8 dB sama da vanilla 3DGS da sauran tushen yin hoto na jijiyoyi akan jerin abubuwa masu ƙyalli.
Fihirisar Kamancen Tsari (SSIM): Ya nuna ~3-5% haɓaka, yana nuna mafi kyawun adana cikakkun bayanai na tsari a cikin haske da nunawa.
Koyon Kamancen Yankin Hoton Hankali (LPIPS): Ya nuna ~15% raguwa a cikin kuskuren fahimta, ma'ana hotunan da aka yi sun fi kama da na gaske ga masu kallo.

Sakamakon Gani: Kwatancen inganci ya nuna cewa NieR yana rage kurakurai na "blobby" da wuce gona da iri akan jikin mota sosai. Ya yi nasarar yin haske mai ƙyalli mai kaifi da daidaitaccen canjin launi akan saman ƙarfe yayin da alkiblar kallo ke canzawa, wanda hanyoyin da suka gabata suka shafe ko kuma suka rasa gaba ɗaya. Sashen HNGD yana cika gefuna da yankuna masu lanƙwasa da ƙarin Gaussians yadda ya kamata, wanda ke haifar da iyakoki masu kaifi da ƙarin cikakkun sauye-sauyen haske.

5. Tsarin Nazari & Nazarin Lamari

Nazarin Lamari: Yin Hoton Mota a Faɗuwar Rana

Yanayi: Motar ja a ƙarƙashin hasken faɗuwar rana mai ƙarancin kusurwa, yana haifar da haske mai ƙarfi, mai tsayi akan murfinta mai lanƙwasa da rufin.

Yanayin Rashin Nasara na 3DGS na Gargajiya: Wakilcin Gaussian mai santsi zai iya shafa hasken a fadin babban yanki (ya rasa kaifi) ko kuma ya kasa ƙirƙira ƙarfinsa daidai, wanda zai haifar da faci mara kyau ko launi mara daidai.

Tsarin NieR:

Sashen LD: Ya gano yankin murfin a matsayin mai ƙyalli sosai (babban $k_s$). Taswirar tsarin jiki ta nuna cewa siffar hasken da matsayinsa suna canzawa sosai tare da alkiblar kallo.
Sashen HNGD: Ya gano babban matakin tsarin jiki tare da kololuwar murfin. Yana ƙara yawan Gaussians a cikin wannan yanki na musamman.
Yin Hoto: Gaussians masu yawa, masu sanin ƙyalli, tare suna yin haske mai kaifi, mai haske, kuma mai dogaro da kallo wanda ke bin tsarin lissafin motar daidai.

Wannan lamari yana kwatanta yadda sassan tsarin ke aiki tare don magance takamaiman aikin yin hoto da ya kasance matsala a baya.

6. Nazari Mai Zurfi & Fassarar Kwararru

Fahimta ta Asali: NieR ba kawai ƙaramin gyara ga Gaussian Splatting ba ne; yana da dabarar juya zuwa ga yin hoto na jijiyoyi mai sanin lissafi. Marubutan sun gano daidai cewa babban raunin hanyoyin da suka dogara da bayyanar kawai kamar asalin 3DGS ko ma bambance-bambancen NeRF shine rashin saninsu ga kaddarorin saman da ke ƙasa. Ta hanyar sake gabatar da tsarin jiki—ra'ayi na asali daga zane-zane na gargajiya—a matsayin ɗan ƙasa na farko, suna ba da samfurin "tsarin gini" na lissafi da ake buƙata don warware da kuma kwaikwayon abubuwan haske daidai. Wannan yana tunawa da yadda ayyukan farko kamar CycleGAN (Zhu et al., 2017) suka yi amfani da daidaiton zagayowar a matsayin son zuciya don magance matsalolin fassarar hoto mara kyau; a nan, tsarin jiki da rabe-raben PBR suna aiki a matsayin fifikon zahiri mai ƙarfi.

Kwararar Hankali: Hankalin takardar yana da inganci: 1) Matsala: Gaussians suna da santsi sosai don haske mai kaifi. 2) Tushen Dalili: Ba su da sanin kayan aiki da lissafi. 3) Magani A (LD): Rabe haske ta amfani da tsarin jiki don ƙirƙira amsawar kayan aiki. 4) Magani B (HNGD): Yi amfani da matakan tsarin jiki don jagorantar rabon lissafi. 5) Tabbatarwa: Nuna riba akan ayyuka inda waɗannan abubuwan suka fi mahimmanci (abubuwa masu ƙyalli). Kwararar daga gano matsala ta hanyar tsarin gine-gine mai kafa biyu zuwa tabbatarwa mai niyya yana da ban sha'awa.

Ƙarfi & Kurakurai:

Ƙarfi: Haɗin yana da kyau kuma ba shi da cutarwa ga bututun 3DGS, yana adana yuwuwar sa na ainihin lokaci. Mayar da hankali kan tuƙi mai cin gashin kansa yana da hankali, yana niyya ga aikace-aikace mai ƙima, mai mahimmanci ga haske. Ribar aiki akan ma'auni na fahimta (LPIPS) musamman yana gamsarwa don amfanin duniya.
Kurakurai: Takardar ba ta da cikakkun bayanai game da samo ingantattun tsarin jiki a cikin yanayin tuƙi mai motsi, a cikin daji. Shin sun dogara da SfM, wanda zai iya zama mai hayaniya? Ko cibiyar sadarwa da aka koya, yana ƙara rikitarwa? Wannan yana iya zama maƙalar kwalta. Bugu da ƙari, duk da yake HNGD yana da wayo, yana ƙara matakin nazarin yanayin wanda zai iya shafar sauƙin ingantawa. Kwatancen, duk da yake yana nuna ribar SOTA, zai iya zama mafi tsauri a kan sauran hanyoyin haɗin gwiwa na PBR/jijiyoyi fiye da bambance-bambancen 3DGS kawai.

Fahimta Mai Aiki: Ga masu bincike, abin da za a ɗauka a bayyane yake: makomar yin hoto na jijiyoyi mai inganci yana cikin samfuran haɗin gwiwa waɗanda ke haɗa ingancin dogaro da bayanai tare da fifiko mai ƙarfi na zahiri/lissafi. Nasarar NieR tana nuna cewa ci gaba na gaba zai iya zuwa daga mafi kyawun haɗa sauran abubuwan zane-zane na gargajiya (misali, BRDFs masu bambanta ta sarari, sigogi na watsawa ƙasa) cikin tsarukan da za a iya bambanta. Ga masu aiki a masana'antar kwaikwayon mota, wannan aikin yana magance matsala kai tsaye—yin hoton mota mara gaske—wanda ya sa ya zama babban ɗan takara don haɗawa cikin dandamali na tagwayen dijital na gaba da gwaji. Haɗin kai na tsarin yana nufin cewa za a iya gwada sashen LD da kansa a cikin sauran bayanan baya na yin hoto.

7. Aikace-aikace na Gaba & Hanyoyin Bincike

Aikace-aikace na Nan take:

Na'urorin Kwaikwayon Tuƙi Mai Inganci: Don horarwa da gwada tsarin fahimtar mota mai cin gashin kanta a ƙarƙashin yanayin haske na gaske, masu canzawa.
Tagwayen Dijital don Tsara Birane: Ƙirƙirar samfuran birane masu motsi, masu daidaiton haske don nazarin inuwa, nazarin tasirin gani, da ƙirƙira na zahiri.
Ciniki ta Intanet & Nuna Samfura: Yin hoton kayan masarufi (motoci, na'urorin lantarki, kayan ado) tare da daidaitattun kaddarorin kayan aiki daga ƙungiyoyin hotuna marasa yawa.

Hanyoyin Bincike:

Haɗin Ingantawar Lissafi da Tsarin Jiki: Haɓaka bututun ƙarshe-zuwa-ƙarshe waɗanda ke haɗa ingantawar Gaussians 3D, tsarin jikinsu, da sigogin kayan aiki daga bidiyo mai ra'ayoyi da yawa ba tare da dogaro da sake gini na waje ba.
Daidaituwar Lokaci don HNGD: Tsawaita dabarar haɓakawa a cikin lokaci don tabbatar da yin hoto mai tsayayye, mara flicker a cikin jerin bidiyo masu motsi.
Haɗawa tare da Binciken Hasken Rana: Yin amfani da rabe-raben sashen LD don jagorantar hanyar haɗin gwiwa na rasterization/binciken hasken rana, inda sassan ƙyalli ke kula da ƙaramin samfurin Monte Carlo don ƙarin daidaito.
Bayan Tsarin Gani: Yin amfani da ƙa'idar rabe-raben dangane da tsarin jiki ga sauran tsayin raƙuman ruwa (misali, infrared) don kwaikwayon firikwensin nau'i-nau'i.

8. Nassoshi

Wang, H., Wang, Y., Liu, Y., Hu, F., Zhang, S., Wu, F., & Lin, F. (2024). NieR: Normal-Based Lighting Scene Rendering. arXiv preprint arXiv:2405.13097.
Kerbl, B., Kopanas, G., Leimkühler, T., & Drettakis, G. (2023). 3D Gaussian Splatting for Real-Time Radiance Field Rendering. ACM Transactions on Graphics, 42(4).
Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., & Ng, R. (2020). NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. ECCV.
Zhu, J., Park, T., Isola, P., & Efros, A. A. (2017). Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks. ICCV.
Cook, R. L., & Torrance, K. E. (1982). A Reflectance Model for Computer Graphics. ACM Transactions on Graphics, 1(1).
Müller, T., Evans, A., Schied, C., & Keller, A. (2022). Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. ACM Transactions on Graphics, 41(4).