Chrome Version : 52.0.2743.82 (64-bit)
URLs (if applicable) :
OS version : 10.11.6
Behavior in Safari 3.x/4.x (if applicable): normal
Behavior in Firefox 3.x (if applicable): normal
Behavior in Chrome for Windows: N/a
What steps will reproduce the problem?(1)Open a website in a tab and leave it
(2) Open a new tab and continue to browser
(3) periodically check back on the inital tab
What is the expected result?
Website has no change in appearance
What happens instead?
Parts of the website will be covered by black boxes, which will flicker as you hover over.
Chroem seems to trigger VTDecoderXPCService which consumes memory. Same goes for Google Chrome helper tasks. When does a simple plain html site need 1gb of memory???

I've been wrangling with this issue for a few hours. Have found after a restart and watching memory usage in OS X and Chrome Task Manager, usage was high-ish but no black boxes. As soon as I opened Atom I got black boxes in Chrome. Wonder if a conflict has arisen with the new version of chrome against some Chromium apps?
OP, can you test if you have any other apps relying on Chromium/chrome rendering that effect this one way or the other?

Could these issues be due to the change in Mac compositing hitting issues on specific machines? I'm trying to figure out why Chromium developers, who use Macs very heavily, don't see it.
Could you both please tell us exactly what machine you're seeing this on, and paste in the contents of chrome://gpu? redinsect@ I see you've already done that over on crbug.com/630394. And what is "Atom" in this context?
And are you sure you have no malicious extensions or other malware?
Assigning to the GPU team. If this were paint or invalidation I would expect to see it everywhere.
I don't think this is the same as 630394.

In your about:gpu, I see that hardware acceleration is disabled. Can you verify that it is?
In particular, could you go to about:settings, go to "advanced", and un-check "use hardware acceleration when available".
Does the issue still happen after you do that and restart Chrome?
WRT the black tiles, that looks like the GPU driver is failing allocations (not much can be done there).
WRT the flickering in #7, that looks vaguely like issue 543324, but we had only seen that on some ATI cards, and it would never happen in software mode.

hardware acceleration is enabled a.k.a "use hardware acceleration if available" is checked. will petrol the toggle and restart later. is there an older version of chrome available to test in parallel (v51 for example)?

seems to happen when chrome pushes VTDecoderXPCService memory usage above 1.5GB
only had shown tabs visible open and runnign for a while. no other apps running
no software updates other than chrome update to latest.
can we get earlier version of chrome to test

@ schenney & ccameron - confirming as above that the issue persists in incognito.
Have disabled hardware acceleration and restarted Chrome - will monitor over the next hour. If black-boxing reappears I'll disable all extensions and continue testing. Note: black boxes only started occuring just under 24 hours ago, seemingly exact time Chrome updated to version 52.x
Machine specs:
Brand new (1 week old) 15" Macbook Pro (lower tier, integrated graphics, 16GB RAM). More detail attached.
As per question above, Atom in this instance is: https://atom.io/ which I believe uses chromium for rendering? Have noticed rare instances where the text area display in atom also started black-boxing, only started happening since the same symptoms occurred in Chrome proper. No issues with Chrome or Atom over the last week until yesterday.

Following up, with hardware acceleration off - flickering has reappeared (tweetdeck is a prime offender), but no outright black boxes yet. Will continue to test for a bit, before disabling all extensions and going from there.
Tweetdeck behaviour attached (.mov screencap).

OK black boxes are back (all extensions disabled, hardware acceleration on) - like jani said in comment #17 this happened immediately on VTDecoderXPCService breaking 1.5GB memory. (See attached).
In summary of my testing:
- Black boxes seem to occur when hardware acceleration is ON and VTDecoderXPCService exceeds 1.5GB memory
- Black boxes occurs in normal or incognito tabs/windows
- Black boxes occur regardless of extensions (tried with all my extensions on and off in all configurations)
- When hardware acceleration is OFF I don't seem to get the black boxes (VTDecoderXPCService never starts up?), but DO get flickering and odd rendering (as per above post with attached .mov tweetdeck is a prime offender and is essentially unusable; image viewing in G+ has odd behaviour like overlays rendering behind the image then hiccuping into view, see attached.)

Can confirm the latest Canary suffers the same problem. 54.0.2808.0
Open a YouTube in Chrome, keep playing video (choose a long one). VTDecoderXPCService will slowly eat more and more memory. Once it crosses 1.5GB (approx) the black box artefacts start appearing.
Restarting Chrome fixes the problem until the next time VTDecoderXPCService gets to 1.5 GB.

Thanks. Just add keep adding detail - after several hours of black-boxing at 1.5-1.6GB usage of VTDecoderXPCService, I've pushed it out all the way to 1.9GB before getting the black-boxing.
No idea what changed, but very possible VTDecoderXPCService is a red herring, or at least a symptom and not a cause.
Anyway, Chromium devs, happy to keep giving you any/all info you need to help trouble shoot this one.
Just to reconfirm (and same with jani in previous posts) this wasn't happening at all on these same machines until the 52 update.

It looks that there are two distinct problems here.
1. With hardware acceleration, it sounds that VTDecoderXPCService is leaking decoded frames, causing allocation failures, which is causing black boxes. In M52 we started using AVFoundation to draw frames produced by VideoToolbox. Our usage of this is very straightforward, and this has been in Canary since March, so this is something of a surprise.
2. In software mode, the issue reported in #19 looks to be a double-buffering issue where the WindowServer is reading from a buffer while we are writing into it. This is theoretically possible -- I can look into making that go away.

I had similar issues while watching HTML5 videos. Didn't find this issue before, so opened one myself: https://bugs.chromium.org/p/chromium/issues/detail?id=632178 for anyone interested.
redins... was so kind to link me to this thread. Thx!

Thanks for the feedback -- I have some theories about what may be causing this, but since I can't reproduce it, I'll need your help to test them.
I have three builds in the file at
https://drive.google.com/file/d/0B6kh5pYRi1dKYVh0WnVkSnRzNUE/view?usp=sharing
They're Chromium (open source version), not Chrome, and you can run them side-by-side with Chrome still open.
The 3 versions are "default_build", which is a control, "no_software_av", which will affect how YouTube and NetFlix is presented on-screen, and "no_av_ever", which affects all video.
Can you try to reproduce the issue with the 3 builds, and let me know what your results are?
If we're lucky, then your results will be
default_build: problem exists
no_software_av: problem does not exist
no_av_ever: problem does not exist
If that's the case, then there is basically no down-side to the fix. If we don't see the problem with default_build, then the results are invalid. If the problem continues to exist with no_software_av but goes away with no_av_ever, then the fix will negatively affect our power consumption (at least until we can figure out the exact cause).

Thank you so much -- that's excellent news!
If it's not too much, could I request another experiment?
I'm trying to find the exact place where this is causing problems. The "no_software_av" build disallowed use of AVSampleBufferDisplayLayer with software-decoded video. I'd like to see if this is unique to the software-decode path, or if the problem exists in the hardware-decode path as well.
To test this, could you install the "h264ify" Chrome extension at
https://chrome.google.com/webstore/detail/h264ify/aleakchihdccplidncghkekgioiakgal?hl=en-US
on the "no_software_av" build and then try to reproduce the problem using YouTube?
The "h264ify" extension will switch the build to requesting hardware-decoded h264 from YouTube, instead of software-decoded vp9. To test if h264ify has taken effect, you should be able to notice that your CPU usage when watching YouTube should be lower.
If the problem does not exist when using h264ify, then that's good news.

Did as you described and when using
h264ify with no_software_av
the problem does! occur. Though, the black boxes seem to be limited to the video window itself. I guess thats h264ify. Added a screen where one can see the increasing memory until video playback fails and the memory is freed up again.
Hope this helps anyway.

I just tested "no_software_av" with h264ify running. Took longer, but finally encountered black boxes (though much less and cleared quickly) and Tweetdeck content jittering and disappearing/reappearing.
When this happened YouTube playback failed as per screenshots attached. (Note, looks like VTDecoderXPCService reset moment prior to these screenshots).

Thanks. It appears that the "h264" versus "vp9" issue is a red herring -- the problem is our use of AVSampleBufferDisplayLayer -- somehow that seems to be causing a leak. What puzzles me is that it only seems to be happening on a few systems (and, by Murphy's Law, none that we have in-house).
When you're experiencing the runaway memory allocation followed by black boxes, what happens if you run "ioclasscount IOSurface" at the command line? Does the count increment as the video plays?
For me it reads
ccameron-macbookpro:src ccameron$ ioclasscount IOSurface
IOSurface = 129
and stays more-or-less constant (if I open more windows, it goes up, but it goes back down when I close them).

Here you go, count doesn't seem to increment as videos play as far as I can tell unless it's happening very gradually which may be the case. To me it seems like over time it matches the increase in the VTDecoderXPCService...
The first run of "ioclasscount IOSurface" in the screenshot was when VTDecoderXPCService was at about 1.1GB and climbing. The last two are after the black boxes appeared.

Woah, 4000-5000 IOSurfaces is a huge leak.
I wonder why this is only affecting a couple of machines (every local machine is having no problems with this).
It would be great if we could track where these IOSurfaces are coming from -- we were able to do that with ioreg prior to 10.10, but that tracking disappeared. I'm reaching out to see if we can get any information there.

It may be that we're enqueueing samples when -[AVSampleDisplayLayer isReadyForMoreMediaData] would return false, which is "safe", but "highly discouraged."
I'll spin up a build tomorrow that blows up if that's ever the case.
In the mean time, could you try running with the command line flag "--show-fps-counter", and report what you see as the memory is blowing up? It could be that we're doing updates too rapidly, and that is getting things into a bad state.
Also, if you are having trouble using chrome, and just want to use it without these issues, the command line flag "--disable-mac-overlays" should fix the problem, albeit at a big cost to your battery.
Regardless of the outcome, the next M52 spin will have this issue fixed.
Also, if anyone affected by this issue is in the bay area and wants me to check out their machine in person, drop me a line in email (@chromium.org).

Thanks for the reports!
I have a new build to test, this time going on the theory that AVSampleBufferDisplayLayer's internal buffers are getting over-stuffed. The build is called "check_ready_and_flush" and is at
https://drive.google.com/file/d/0B6kh5pYRi1dKQXZQUXY4LXdrNEE/view?usp=sharing
This build will work with h264ify or without (no difference). See if this causes the black boxes. Also see if it causes a gray/pink/yellow grid to appear (the grid will tell me if we hit the condition that I'm concerned about).

Oh no.
Quick question to verify: The ioclasscount was small (like ~100) before Chromium started, and then it blew up while watching a video, right? If it's already >1000 before Chromium starts, then there will be black boxes (because there isn't enough memory around anymore). If it's already >1000 before Chromium starts, then you may need to reboot first (but, before that, run the attached program described below!)
Oh -- you're on 10.9! That's before they broke the IOSurface reporting. Instead of ioclasscount, try running this program (attached the source -- the instructions to build and run are at the top -- "g++ iosurface_dump.cc -framework IoKit -framework CoreFoundation && ./a.out").
That will tell me what process is responsible for holding these surfaces. Could you attach its output before starting the Chromium build, and again once Chromium has caused the black boxing problem.
I have one more build, which very aggressively flushes these buffers -- it's "flush_every_frame", and is at
https://drive.google.com/file/d/0B6kh5pYRi1dKUWlvcXNmUXNEcEk/view?usp=sharing
It should put a green border around all videos.

Hey I just tried to produce the output with iosurface_dump.cc still using check_ready_and_flush. This time however I also enabled --show-fps-counter - and nothing happende. No black boxes, not memory overflow. All seemed good. I attached an output of iosurface_dump.cc called "all_ok" during video playback. Then I switched the --show-fps-counter off again to produce an output with black boxes, but when they appeared, I started iosurface_dump but it got stuck. could kill it or anything. Tried to reboot - nothing. Had to do a hard reset. Maybe to much data? The file is empty so nothing got written logged unfortunately. But maybe the iosurface_dump.cc helps?
btw this time I also disabled h264ify.

It's odd that --show-fps-counter helped. I'm worried now that just drawing the green border fixed things in flush_every_frame.
I think we're close -- I've removed the green border and chopped flush_every_frame into a handful of separate parts. There are 3 builds here, can you let me know if they start causing the leak?
https://drive.google.com/file/d/0B6kh5pYRi1dKeXIxdUQyNER1U3M/view?usp=sharing
For my reference, the builds are:
test_a:
same as flush_every_frame, but no green border
- flushAndRemoveImage before every enqueueSampleBuffer
- flushAndRemoveImage in ContentLayer dtor
- no calls to IOSurfaceIncrementUseCount
- no fslp frames
test_b:
- flush before every enqueueSampleBuffer
- flushAndRemoveImage in ContentLayer dtor
test_c:
- no calls to IOSurfaceIncrementUseCount
- flushAndRemoveImage in ContentLayer dtor
Thank you again for all of your help!!

M52 Stable Release is blocked due to this issue. The original plan was to cut the RC at 5.00 pm today.
Chris/Ken, so you think this is critical for M52 Stable? We can take the fixes for further Stable refreshes. Please confirm.

[Changing the title of the bug]
Well, crap. So #55 gave me hope that we had a fix here. But #57 leaves me basically where we started.
AVSampleBufferDisplayLayer is the way to play accelerated video with minimal power usage (see issue 594449), so it's really upsetting to have to disable this.

It looks like there are different issues between 10.9 and 10.11 -- the bug in 10.9 was solved by adding just a "flush" -- the bug in 10.11 seems to have more problems.
Thanks sebastian.@ -- I think that 10.9 has a fix now
jani.@ and redinsect@, a couple of things I'd like you to try:
1. Did you have any luck with the "check_ready_and_flush" build from https://bugs.chromium.org/p/chromium/issues/detail?id=631485#c47 ? Did a grid appear before the black tiles?
2. I reduce Chrome's video decode path into a stand-alone application, which I've put at
https://drive.google.com/file/d/0B6kh5pYRi1dKLTEwdlBmRVdybzg/view?usp=sharing
Could you try to run it and see if your IOSurface count grows while it is running?
It is already built, so, from a command line, do "./test_player 720p30fps.mp4", or alternatively type "make". If it has issues running, and you have XCode, you can also do "make clean && make".
Also, if anyone is having this issue on 10.11 in the SF bay area or in the LA area, drop me a line via email so that I can take a look in person.

Btw, my one remaining theory is that our YUV->RGB [1] OpenGL shader has a bad interaction with AVSampleBufferDisplay layer (like the infamous issue 158469).
I'll spin up a build that just skips that part, and we'll see if that plugs the leak. That may have to wait until Monday.
[1] https://cs.chromium.org/chromium/src/ui/gl/gl_image_io_surface.mm?rcl=0&l=335

#1 Check ready and flush - I'm about 20 mins into testing (running x4 YouTube tabs playing video) and looks promising so far - memory usage is staying low! Will keep testing though and give you a definitive result in the next hour.

Very interesting. Does it also stay low if you leave the YouTube videos visible (like, on-screen). When they're in background tabs, they don't send their frames to the AVSampleBufferDisplayLayer (which is the thing that appears to be causing the leak).

OK another update, again this is running "Check ready and flush"
I managed to spike VTDecoderXPCService above 2GB briefly, before it settling back at around 1.7GB, this is after sitting for a while at about 500MB. I did this by loading facebook, buzzfeed video, and watching video after video in theatre mode, switching every 10-20 seconds or so. On each new video load sometimes there would be no increase in mem usage, and sometimes it would jump 400-500MB at a time.
During this time ioclasscount IOSurface stayed no higher than 3200, though I got it to briefly spike (see attached) then it dropped down again.
During this whole time, no block boxes, no pink/grey/yellow grid.

Re #69 -- the not needing to be visible -- that's *very* concerning!
I just realized, I only had users on 10.9 verifying that the fix that I'm planning to ship to M52 actually works!
The fix that we're pushing to M52 next week is the "no_av_ever" build in #30. Did that fix the issue for you?
If not, then I'll need to scramble to find a new fix (my last remaining guess is #63).

I'm trying to see if there's a way to detect-and-recover from this.
You mentioned that when you quit Chrome then (1) VTDecoderXPCService memory usage returns to normal, and (2) 'ioclasscount IOSurface' returns to normal.
Q1. Does this return to normal if you all of the Chrome windows that are playing video? Or does it remain high?
Q2. What about closing all Chrome windows, but leaving the app running.
Q3. What about when you kill the GPU process (go to "Window" in the menu bar, "Task Manager", select "GPU Process", and press "End Task".
If this is the case, then we can make it so that we use AVSampleBufferDisplayLayer, but we kill the GPU process if we see VTDecoderXPCService's memory blow up, and disallow further use of AVSampleBufferDisplayLayer.
(this is all assuming that no_av_ever works ... if it doesn't, we're pretty hosed).

Also, when you start seeing the memory usage increase, do you always see the error
ERROR:vt_video_decode_accelerator_mac.cc(649) : Illegal attempt to decode without IDR. Discarding decode requests until next IDR.
in your logs in about:gpu?

I run into issues similar to this constantly, but only when I'm my laptop switches from integrated to high perf graphics, or vice versa.
I use an external thunderbolt display which automatically kicks in high perf graphics when I get to work, and then it goes back to integrated when I leave at the end of the day. There are also a number of programs that annoyingly force a high-perf graphics switch because of bugs (skype, hipchat, VLC), necessitating a restart of Chrome.
I don't know if this is the same issue, but the only way I've been able to fix the problem is to just restart Chrome any time the switch happens. Occasionally I can get a tab to show content instead of weird black box designs by tearing it off the tab bar and dropping it back.
Running dev branch 54.0.2810.2

"The fix that we're pushing to M52 next week is the "no_av_ever" build in #30. Did that fix the issue for you?"
Chris, about 45-60 mins of testing just now, and that build seems to be OK to me. Look like it's releasing memory before it has a change to go nuts? Never got higher than 800 ioclasscount IOSurface, and usually settled anywhere between 300-600 depending on switching tabs/vids etc.

To summarize so far:
flush_every_frame (aka test_a):
- makes "reasonable" changes to our use of AVFoundation
- fixes the issue for sebastian (using OS X 10.9)
- does not fix the issue jani (OS X 10.11)
- not tested for redinsect
no_av_ever:
- restores M51 behavior, doesn't allow improved battery life
- fixes the issue for sebastian (using OS X 10.9)
- does not fix the issue jani (OS X 10.11)
- not sure for redinsect
This appears only related to accelerated video. I believe this because VTDecoderXPCService should not be invoked for software decoded video.
I'm looking at all of the other diffs that went into M51->M52. The other thing that comes to mind is how we handle CVPixelBufferRefs. In particular, the changes in
https://codereview.chromium.org/1910633004
- in M52 but not M51
- pass CVPixelBuffer to AVSampleBufferDisplayLayer
https://codereview.chromium.org/1881783002
- in M52 but not M51
- remove an assertion about CVPixelBufferRef lifetime
https://codereview.chromium.org/1861923002
- in M51 and M52
- initialize GLImageIOSurface with CVPixelBufferRef instead of IOSurfaceRef
So, with this, I have one more idea of how to handle this -- treat CVPixelBufferRef the way we did in M51. I'll spin up a build to do this.
I'm also going to merge that into the M52 branch, in case we're doing another build.

The above patch merges the changes from the "no_cvpixel_or_avlayer" build in #80 to M52 (because there is no risk).
If "no_cvpixel_or_avlayer" works, then I'll have one more patch to see if we can keep the battery improvement (and roll it out in M53). In theory it should be possible.
Fingers crossed that the CVPixelBufferRef issue is the problem.

started testing no_cvpixel_or_avlayer"
immediate observation is that .mp4 video playback is brokebn. it starts and immediately stops. example in tweetdeck this https://pbs.twimg.com/tweet_video/CoqL7nEVIAA39Cq.mp4 or channel 4 f1 race rewind from today. all mp4 videoes in tweedeck played fine with all the other versions on chromium i've tested. youtube plays fine though
see the issue https://drive.google.com/open?id=0B77k0uPbvCnvTWZvQU9KTngtTkk

tested "no_cvpixel_or_avlayer" : black boxes
its also been one of the quickest to increase VTDecoderXPCService's memory consumption
closing down youtube in this scenario didnt make a dent in VTDecoderXPCService's memory. shutting done tweedeck site did - rest it back to normal 24 mb
chrome://gpu errors
https://gist.github.com/anonymous/c79b2f7971281585facf287473b5cb42

Sorry about the no_cvpixel_or_avlayer build (oops, now I need to fix that in the M52 tree too ... sorry TPMs!!!!). Frantically un-doing that. I've deleted no_cvpixel_or_avlayer from Google Drive.
I'm beginning to suspect that TweetDeck is the source of all of these problems, and it's only coincidence that we're seeing issues with video playback.
Can you ever reproduce the problem without using TweetDeck?
Is TweetDeck an extension, or just a webpage?
[I'm creating a twitter account to test this now, to see if I can reproduce the problem].

One more thing to check -- can you to to Task Manager, and add the column "Gpu Memory" (right-click on the column titles), and attach a screenshot sorted by that? That might give some hints as to who is eating tons of memory.

as i cant play any other video other than youtube with this build, the VTDecoderXPCService memory usage is remaining constantly low (2.4 mb)
see attached screenshot.
I will not add tweetdeck tab and take a screenshot with task manager mem usage

Oh -- sorry, for the "Gpu Memory", use any build that causes the issue (Chrome Canary, Chrome 52 are fine). And keep TweetDeck there.
What I'm looking for is if our Chrome task manager sees the memory usage too (if it does, then the problem should be easier to track).

to sdd to this. my daily chrome usage has the same (work and home) for the last year or so: 5 pinned tabs on start-up: 2x Inbox, feedly, hangouts, tweetdeck and they run all day. Not sure what change in v52 upgrade....hence i noticed this issue immediately after the update

^^^ Same here, Pinned: G+, Gmail, Inbox, Photos, Hangouts, Tweetdeck, Trello - all pinned all day - never had problems till build 52.
Sorry, Chris - do you want us to use Chrome stable, including Tweetdeck - get the black boxes to happen, then give you the chrome://gpu output AND taskmanager screenshot?

Sorry, I wasn't clear.
I'm looking for a screenshot of Task Manager with the "Gpu Memory" column when the black boxes problem is occurring. You can use whichever build is most convenient for you to reproduce the black boxes problem (so, Chrome 52 would be fine).

Thank you ccameron@ for trying very hard to fix this issue. M52 is already in stable and bar is VERY high. We can take change only if it is baked/verified in canary and safe to merge.
Per our chat on Friday, I only approved change listed at #58 to directly land on M52 branch 2743 as it just disables the feature that is new in M52 and it is safest to do (see #59). We're planning to cut M52 Stable RC tomorrow, so please revert any changes (#58 and #81) if anyone or both of them are not safe.
Any other merges to M52, please re-request a merge by applying "Merge-Request-52" label. Thank you very much.

Got the black boxes on stable. Attached is Task Manager, Activity Monitor, ioclasscount IOSurface, and Chrome://GPU
Chris, yeah looks like Tweetdeck may be the cause? (How effing frustrating). Still weird that prob surfaced in 52... but Tweetdeck development seems neglected at best :/

Thanks! That very effectively shoots down my theory about blaming tweet deck.
FYI, my local tweet deck is still hanging out at <200 IOSurfaces.
I'm continuing to scrape through the changes that went into M52, and there was a change to our VTDecompressionSession instance in https://codereview.chromium.org/1882533002.
I've pulled that out, and also torn out all of the other things I'd already torn out (plus canvas), and put it in the build "no_cvpixel-no_av-no_seekcl-no_canvas", which I tested (sorry about the last build), and I've uploaded to:
https://drive.google.com/file/d/0B6kh5pYRi1dKNy01SXhTNEVfRTg/view?usp=sharing
Could you see if the black box issue reproduces with that.
(now fingers crossed that it's the VT patch -- that would make a lot of sense, given the other observations).

Wow! 40 instances of accelerated h264 decoders, and 550 decoded frames -- that's crazy-high! And I would bet it tracks with "ioclasscount IOSurface" (like, there are probably ~100-200 more IOSurfaces than VTVideoDecodeAccelerator::Frames, but as they grow, they grow in tandem).
So, it looks to me that tweetdeck is DoSing the system. We don't put any guards in place to prevent this from happening (any webpage is perfectly welcome to DoS the system this way).
My tweetdeck only gets to ~10 decoders, but that's probably because of the content that I'm looking at. But I do see "ioclasscount IOSurface" growing, and I do see VTDecoderXPCService growing.
I'd guess that this appeared in M52 because we held on to IOSurfaces "just a bit longer" than in M51, and that pushed things over the edge.
Also, the "Gpu Memory" column in task manager doesn't take into account these decoded frames (hmm, maybe I should add an IOSurface memory column).
So, the plan for this:
* We should look into ways to prevent this sort of DoSing (that's a longer-term project)
* Maybe devrel can reach out to Twitter about these problems
* I'll leave AVSampleBufferDisplayLayer disabled for M52
* I'll add the "flush" fix to AVSampleBufferDisplayLayer for M53, and ship that then

Thanks Chris! FYI - *** VTVideoDecodeAccelerator::Frame count: 2453 (decoder count: 165)
So what do you think is the rough ETA for this to be resolved in stable?
For other users who have found this thread: close/reopen tweetdeck.twitter.com *regularly* to resolve OOM errors in the meantime :)

Wow! That definitely solves the mystery!
WRT when we'll get a fix on stable for this, I'm not sure -- we'll need to figure something out between devrel, media team, etc. Chrome 52 is almost-completely-frozen (so if this is mostly tweetdeck's fault, we're not likely to push further changes for it).
I'll update tomorrow when I get more information about this from the rest of the team.

Hi folks – I'm Tom, lead on the frontend of TweetDeck. Thanks for your extensive efforts looking into this — it's something I've been seeing lots, probably because I often run 3 or 4 instances of TweetDeck at a time!
There is one likely culprit for this: gifs. We render them as videos (mp4, <video>), and don't pause them when they're offscreen, although we sometimes remove them from DOM completely to keep a lid on the DOM node count.
--> How can we prove it's the gifs? Can we see the source of these decoders?
The team can put some work into pausing offscreen videos, but we don't have the resources to do it right now. However, we're planning to replace our infinite list implementation this quarter. DOM-thrift is at the top of that project's priorities.
(Alternatively, we recently changed the way we embed videos in TweetDeck to use an iframed player (video/mp2t, <video>) but this is probably not the issue as it takes a few clicks to watch one and you're seeing high instances of the h264 decoders.)

Hi tashworth@, you'll likely want to resolve this sooner than later as the M52 release will be going out soon and we don't feel that this is a stable blocker. Infinitely loading videos like this will either cause OOM or thread cap issues. We can idle collect software-decoded paused instances, but if the videos are playing or hardware decoded they won't be collected.
There's some improvements we could make in later versions to collect hardware decoded videos and possibly auto-pause videos w/o sound once off-screen, but that's not something we can land with M52 or even M53.
The most reasonable way to fix this is by clearing the src attribute (set '') _and_ removing the element from the dom -- this is the advice we gave Vine a long time ago. Unfortunately the HTML5 media spec allows elements to continue to function even when removed from the DOM, so it's necessary to set src='' to completely delete them.
The IntersectionObserver API should make it fairly easy to implement this; clear src='' some distance off page and bring it back once the page comes back into visibility.

Thanks for the tips — we'll look at those options. We already know what's onscreen pretty well — the hard bit is managing starting & stopping correctly.
The rendering engine supporting natively this would be *very* helpful. This is tough in JavaScript without dropping frames, especially alongside the other things TweetDeck is doing concurrently.

Thanks, happy to help. FWIW, anything we implemented in the rendering engine would not be any better at preserving frames than JavaScript. Suspend/resume works by destroying the internal playback engine and seeking to the last known time upon resume.

If it helps, here's some more info: I'm having the same issue (even as we speak) running the latest Chrome (stable channel) on a mid-2011 iMac with 20GB or RAM, running OSX 10.9.5. I don't use tweetdeck, but Tom mentioned something about GIFS. If it helps, I keep a http://messenger.com (the standalone facebook chat) tab most of the time open, and there are constantly animated GIFs on the conversations (can be seen in the screenshot - in messenger.com all the images in the current conversation reappear in the form of a mosaic at the bottom right - when they are GIFs, the mosaic is "animated"). The black-boxes problem for me starts some time after opening some youtube video. In the screenshot you can see the ioclasscount IOSurface count (north of 5000 and goes up merely by scrolling). I haven't tested any of the builds, but gladly would test something if you need me to!
-Fotis

To #125, that's a similar-but-separate issue 632178, which is related to any video playback on some macOS 10.9 systems. That issue has been fixed, and the fix should be pushed to users in the next day or so (if it hasn't already).

Hi,
David here, also frontend dev on TweetDeck. We have recently put out an update to explicitly clear the "src" attribute before removing videos from the DOM. Whilst this doesn't eliminate the problem, it hopefully helps ease it a bit. There's certainly cases where the issue will still crop up (for example: a user who filters their content to display *ONLY* animated GIF videos)
While we are discussing this, I was wondering what your thoughts are on browser behaviour in this particular scenario...
The media spec makes a valid case for media elements with audio to continue playing in the background - even when detached from the DOM. But in our case, the videos don't have an audio track. Is it a reasonable expectation that removing a video with no sound from the DOM, makes it a valid candidate to be collected? The spec does allow for GC in this scenario, and I totally understand that implementation details are never easy. But I'm interested in the conversation, and curious on what you think.
Having said that, I'm going to re-iterate what my colleague Tom has said above, and we are working on a larger body of performance work, which includes improving how videos are handled on our side.

as soon as i load the site up, the VTDecoderXPCService which has been running for a while at 8.9MB shoots up to 35.8MB
It then continues to rise, eventhough there is barely a new tweet, no scrolling the timeline
see attachements and note the time stamp

My bug 698844 was merged into this issue as a duplicate. I'm seeing the issue with Chrome 56, but I see no activity on this bug to indicate that it is still open. Is this bug really a duplicate and not fixed?

We are seeing this with VS Code (using Electron) ever since we updated Electron to Chrome 56. We did NOT see this while we were using Electron with a Chrome version of 53.
It reproduces no matter if GPU is enabled or disabled (via --disable-gpu flag).
More steps to reproduce using Chrome 58:
1. Connect your MacBook Pro laptop (mine is a Retina, 15-inch, Mid 2014) to an external 4k or 5k monitor
2. Mirror the display and make sure the resolution is set to at least 3360x1890
3. open https://sourcegraph.com/github.com/Microsoft/vscode/-/blob/src/bootstrap.js#L42:6 and 3 editors side by side (to open more than one editor, click on the Split icon next to the "View on GitHub" label top right of the editor)
4. make sure the window is maximized
5. scroll around and generally use the web site UI
=> flickering