Video Textures and Audio in IOS

🌵 Needle Engine
1

Hi!
We are planning an small web app that displays a transparent video on AR, and currently we have a situation with IOS since it uses Quicklook. I researched a bit here before asking and tested, but for some reason, the samples that are supposed to have audio, doesn’t really play any sound.

On top of that, does Quicklook support the video player component? and if so, is it possible to use a custom shader with it? (since the effect that we are trying to achieve is essentially a chroma key).

Until now all that I have done is to open the sample with the video feed as background and modified it by adding the USDZ Export component, the only addition is a quad with a videoPlayer component (with its corresponding video assigned). When I open it in an IPhone the “open in Quicklook” button appears, but nothing happens if I click it.

Any clues?

Original Post on Discord

by user 368204675922657280

2

I think we didnt export video before for playback in quicklook, right @herbst🌵 ?
AudioPlayback should work tho. Which scenes did you try except your own?

3

Audio works, yes. QuickLook doesn’t support actual video playback, but a workaround is per-frame quads that are animated in a way that looks like playing video. The textures of these can be transparent.

4

@kipash :cactus: did an experiment to auto generate such animated quads from video on export, not sure how far that was, maybe you can upload that somewhere as a starting point?

5

Here’s a thread from me taking apart one of Apple’s WWDC files that essentially plays back a frame-by-frame video:
https://x.com/hybridherbst/status/1663693635643228161?s=46

6

Hey, it was an experiment so take it as a starting point:

https://vxtwitter.com/k1pash/status/1705259600629375287?s=46

Code for sampling video to texture array:
https://github.com/needle-tools/needle-engine-samples/blob/samples/quicklook-video/package/Runtime/VideoPlayer/Scripts/Needle.Sample.VideoPlayer~/SampleVideo.ts

7

Concept was to trigger this conversion on export and then construct sonething that can be apart of the usd.

8

----

Lastly, quicklook doesnt support custom shaders. Any effect has to be baked into the mesh or texture.

9

As I wrote above :slightly_smiling_face: needs to be transparent meshes. The WWDC file uses a nice trick of having very low opacity but very high emissive factor to counter that, which results in good opacity

10

Thank you all very much! Ill check it out

by user 368204675922657280