Bug Description

I have a up-to-date i686 gutsy install on a Dell XPS M1710 laptop with an NVIDIA GeForce Go 7900 GS (except with some packages installed from hardy, including the nvidia driver).

I am using compiz and I am seeing a situation where it sometimes gets into a state where apps are unable to go into fullscreen mode. Attempting to do so results in an X11 BadAlloc error with the explanation of "insufficient resources for operation". Disabling compiz fixes the problem, after which it can be re-enabled and fullscreen will work with it again. I have 2GB of system RAM, 4GB of swap, and 256MB of video RAM, so I don't think I am actually running out of resources.

I have seen this with Totem, MPlayer, and SMPlayer. In the latter two, it occurs with all output modules that I have tried (which are xv, gl, and gl2). Note that in SMPlayer the X11 error simply causes video to not show up in fullscreen, whereas in Totem and MPlayer it causes the program to exit. I will test other apps and other settings (particularly turning off/on the legacy fullscreen option in compiz) when this bug happens next.

I am attaching .xsession-errors, Xorg.0.log, and the output of lspci -vvnn, since I have noticed that people often ask for that. For the first two, they are taken from after I encountered the problem, turned off compiz, and then turned it back on.

I've now tried changing the "Legacy Fullscreen Support" and "Unredirect Fullscreen Windows" options while experiencing this problem, but that had no effect. I also checked my memory usage when this happened and it was only at around 50%. Not sure what else to check.

Happened again today, except this time the bug occurred whenever the video WASN'T in fullscreen mode. I got backtraces for both Totem and MPlayer; they are attached. They don't really say much though, basically just that the program gave a frame to the X server and got back this error.

At the time the bug happened, I was only using 75% RAM and 5% of SWAP. After getting the backtraces, I noticed that I had about a dozen instances of SMPlayer open (probably due to a stuck shortcut key that opened too many). After closing them, the bug went away, even though my RAM usage had barely changed (75% -> 72%). Then I tried opening a dozen instances of SMPlayer again (-> 78% RAM usage) and the bug came back.

Perhaps the problem is that X and/or Compiz reserves video memory for each app that might need it, even when they haven't asked for it? But that the memory for fullscreen mode is shared? Though that wouldn't explain the original situation where the bug happens in fullscreen mode ...

I have the same problem. My card is a nVidia 660GT with 128MB. I'm using Ubuntu Hardy and I tried with the Nvidia restricted drivers 169.12, 173.14.12 (envyng) and the latest 177.80 (from the nvidia web).

I can reproduce the problem whenever I want, I think it's related with the amount of texture memory used. I'm using a resolution of 1920x1200 and when I have a few windows opened I can't watch videos in fullscreen, Totem, Mplayer and VLC just die with a BadAlloc error.

If I have a lot of windows opened I can't even watch the video at all, the players close with the same error before the video is shown. But if I minimize all the windows, I can play the video in fullscreen, and when the video is playing I can re-maximize the previously minimized windows and the video will keep playing (sometimes it fails too).

FYI, I hadn't seen this for a while, but then today it happened a lot when I was trying to play a 1080p video. So it's probably more likely with larger video resolutions. Which of course would make sense.

I have to open many windows to trigger the bug (~10-15 firefox). If I switch to another VT and back / restart compiz with --replace / kill some windows, then I can play videos with XV for a while (5-10 minutes). ALT+TAB in compiz is much slower when the XV doesn't work.

Daniel, I cannot found a method to check used texture memory, googled for hours for nothing.

I see this on my AGP 7600GT (256M); as well as stopping video going full screen, it makes mythtv frontend with the opengl drawing engine unbelievably slow and cpu hungry (presumably it falls back to software 3d rendering or something). Would this be caused by texture memory fragmentation or something? (Does video and/or whatever mythtv does need a large _contiguous_ chunk of it? Or does that not make sense - does texture memory get virtualised by an mmu (gart?) or something?).

Thank you for taking the time to report this bug and helping to make Ubuntu better. We are sorry that we do not always have the capacity to look at all reported bugs in a timely manner. There have been many changes in Ubuntu since that time you reported the bug and your problem may have been fixed with some of the updates. It would help us a lot if you could test it on a currently supported Ubuntu version. When you test it and it is still an issue, kindly upload the updated logs by running apport-collect <bug #> and any other logs that are relevant for this particular issue.

I no longer have the computer that experienced this issue, and my new one has tons more video RAM so I can't determine whether this issue still exists or not. I'll leave it as Incomplete in case someone else is still experiencing it and can provide the info.