http://bugzilla.opensuse.org/show_bug.cgi?id=1046962 Mircea Kitsune changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |sonichedgehog_hyperblast00@ | |yahoo.com -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1046962 http://bugzilla.opensuse.org/show_bug.cgi?id=1046962#c2 --- Comment #2 from Mircea Kitsune --- Created attachment 730948 --> http://bugzilla.opensuse.org/attachment.cgi?id=730948&action=edit Xorg.0.log -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1046962 http://bugzilla.opensuse.org/show_bug.cgi?id=1046962#c4 --- Comment #4 from Mircea Kitsune --- Created attachment 730950 --> http://bugzilla.opensuse.org/attachment.cgi?id=730950&action=edit dmesg -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1046962 http://bugzilla.opensuse.org/show_bug.cgi?id=1046962#c6 --- Comment #6 from Mircea Kitsune --- Created attachment 730952 --> http://bugzilla.opensuse.org/attachment.cgi?id=730952&action=edit journalctl -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1046962 Stefan Dirsch changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #730947|application/x-trash |text/plain mime type| | -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1046962 Stefan Dirsch changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P5 - None |P3 - Medium Status|NEW |CONFIRMED -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1046962 Stefan Dirsch changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #732016|text/x-log |text/plain mime type| | -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1046962 http://bugzilla.opensuse.org/show_bug.cgi?id=1046962#c10 --- Comment #10 from Max Staudt --- (In reply to Mircea Kitsune from comment #8)

Could be we talking a VRAM memory leak?

Hmm, interesting. I don't have a radeon system at hand, but you can typically monitor memory usage through debugfs. Get a root shell and have a peek through the files in /sys/kernel/debug/dri/0/ - they're named differently for each driver, but radeon definitely has at least one that displays the amount of memory allocated to BOs. Thus, you can SSH in, or run this on a second screen (as root!): watch -n 1 cat /sys/kernel/debug/dri/0/THE_FILE_NAME -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1046962 http://bugzilla.opensuse.org/show_bug.cgi?id=1046962#c12 --- Comment #12 from Mircea Kitsune --- Created attachment 733531 --> http://bugzilla.opensuse.org/attachment.cgi?id=733531&action=edit Screenshot of sys/kernel/debug/dri/0 The exact same story repeats with Mesa 17.1.5 (as with 17.1.4); The release notes claim that a core crash has been fixed, yet inexplicably this freeze persists after updating to the latest version. Something very unusual happened however: I rebooted my machine and started testing under Xonotic. After about 10 minutes of playing I got my first freeze, however it did not block the machine; Only Xonotic itself crashed (image froze and sound died), so I was able to alt-tab switch to my desktop... the system detected that the process was unresponsive and killed it, after which I could notice that it did NOT eat up any CPU or memory while it was frozen. I tested again and after about 15 minutes I got another freeze... this time though it froze the entire computer as usual (including taking down SSH). I preformed the suggested test of monitoring the files in sys/kernel/debug/dri/0 through my SSH connection, to check whether this might be caused by a vram leak. The most relevant file in there was radeon_vram, which seems to have 2.0 GB at all times (makes sense as that's the amount of vram on my video card). I used the command "watch -n 1 cat /sys/kernel/debug/dri/0/radeon_vram" to monitor it, but that has not printed any changes in the file itself. Adding a screenshot of that directory and its contents. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1046962 http://bugzilla.opensuse.org/show_bug.cgi?id=1046962#c14 --- Comment #14 from Mircea Kitsune --- (In reply to Max Staudt from comment #13)

From what I remember seeing, all of them were 0 byte files at all times. Only radeon_vram and a few binary files in there seemed to have any size.

I'm also not sure how efficient the test can be at this point; The SSH connection goes down the moment the system freezes, alongside all other processes. As such, both the other computer as well as the console would only capture the last moment before the freeze. I initially assumed that the SSH connection and other background processes might stay up, but it seems the whole computer stops responding at the exact moment the freeze occurs. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1046962 http://bugzilla.opensuse.org/show_bug.cgi?id=1046962#c16 --- Comment #16 from Mircea Kitsune --- (In reply to Max Staudt from comment #15) I was referring to the files in sys/kernel/debug/dri/0, I didn't look in procfs/sysfs/debugfs. In any case, I'm doubtful as to how much this can help... given that whatever file or process I'd monitor, I would only see its instance right before the freeze, since the entire system dies at that moment. I'm hoping there is another safe test I can preform, as I need to debug this problem rather urgently. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1046962 http://bugzilla.opensuse.org/show_bug.cgi?id=1046962#c18 --- Comment #18 from Mircea Kitsune --- (In reply to Max Staudt from comment #17) The problem is that, I'm not sure which file to follow or how, so I might need more specific instructions. The little script you suggested looks like it might work, but there are over 20 files inside debugfs. Perhaps the watch command can be used to track changes in all of them? This might however work in case we're talking a gradual leak, and not something that all happens instantly (in less than a second) so that the system freezes before it gets recorded. I'm still assuming it's all instant, but I just realized the other option could still be the case too. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1046962 http://bugzilla.opensuse.org/show_bug.cgi?id=1046962#c20 --- Comment #20 from Mircea Kitsune --- I was able to use parallel SSH sessions to monitor changes in the files suggested by Max Staudt, which I did by using the command: watch -n 1 cat filename The relevant files that existed and I was able to watch are: /sys/kernel/debug/dri/0/clients /sys/kernel/debug/dri/0/gem_names /sys/kernel/debug/dri/0/radeon_gtt_mm /sys/kernel/debug/dri/0/radeon_vram_mm /sys/kernel/debug/dri/0/ttm_dma_page_pool /sys/kernel/debug/dri/0/ttm_page_pool I will attach the captures of each output, each showing its file <= 1 second before the freeze. I understand those files should retain information about VRAM, which indicates whether this could be a progressive memory leak. Very important note: It has taken me hours to obtain those outputs, and for a while I thought the freeze was fixed by an update altogether. For over 2 hours I was able to run all game engines that produced this crash without getting any freeze whatsoever, which has never happened before! However the freeze returned after I restarted my machine, meaning it's still present. I have no idea whether there's a switch in my system that causes it to happen only sometimes, but hopefully those files will say something. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1046962 http://bugzilla.opensuse.org/show_bug.cgi?id=1046962#c22 --- Comment #22 from Mircea Kitsune --- Created attachment 734015 --> http://bugzilla.opensuse.org/attachment.cgi?id=734015&action=edit watch-gem_names.txt -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1046962 http://bugzilla.opensuse.org/show_bug.cgi?id=1046962#c24 --- Comment #24 from Mircea Kitsune --- Created attachment 734017 --> http://bugzilla.opensuse.org/attachment.cgi?id=734017&action=edit watch-radeon_vram_mm.txt -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1046962 http://bugzilla.opensuse.org/show_bug.cgi?id=1046962#c26 --- Comment #26 from Mircea Kitsune --- Created attachment 734019 --> http://bugzilla.opensuse.org/attachment.cgi?id=734019&action=edit watch-ttm_page_pool.txt -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1046962 http://bugzilla.opensuse.org/show_bug.cgi?id=1046962#c28 --- Comment #28 from Max Staudt --- This... is weird. So if I understand correctly, you've been running 3D applications, yet the only client shown in "clients" is X? Okay, that file isn't being helpful... So it seems that the most interesting are radeon_gtt_mm, radeon_vram_mm, and ttm_dma_page_pool. I'm not sure what the last one says, but the former two should, together, list all allocations. Unfortunately we can only see the first few lines of each now, since watch cuts off the output where it overruns the screen. So... while true ; do clear ; echo $(( $(cat /sys/kernel/debug/dri/0/radeon_vram_mm | tail -n +3 | cut -d : -f 2 | sed "s/$/ + /") 0 )) ; sleep 1 ; done and while true ; do clear ; echo $(( $(cat /sys/kernel/debug/dri/0/radeon_gtt_mm | tail -n +3 | cut -d : -f 2 | sed "s/$/ + /") 0 )) ; sleep 1 ; done should sum them up. This way, you can watch whether some operation reserves more VRAM without freeing it. Silly me forgot to also ask for dmesg -w which is dmesg in 'watch' mode. Your kernel will probably start screaming once the problems appear. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1046962 http://bugzilla.opensuse.org/show_bug.cgi?id=1046962#c30 --- Comment #30 from Mircea Kitsune --- (In reply to Max Staudt from comment #29) Unfortunately I don't know exactly what the last version of Mesa was that didn't have the problem; I started noticing it in 17.1.3 but it could have been introduced in 17.1.0 or prior. As downgrading system packages this far might cause the system to mess up, it's something I prefer not to risk. An upstream bug was already filed. Unfortunately the folks at freedesktop.org enjoy completely ignoring my bug reports for months on end, even after I label them as major and post countless tests and descriptions to help. https://bugs.freedesktop.org/show_bug.cgi?id=101672 I will try to preform the tests you mentioned as well. I really hope you or someone else can make good use of them, as generating those files often proves to be an ordeal... I need to get rid of this problem rather urgently too. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1046962 http://bugzilla.opensuse.org/show_bug.cgi?id=1046962#c32 --- Comment #32 from Max Staudt --- Great, thank you for the update! Sorry, I had missed the upstream bug in the "See Also" section previously. Okay, if upstream is quiet, then let's hope your BIOS setting change did the trick. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1046962 http://bugzilla.opensuse.org/show_bug.cgi?id=1046962#c34 --- Comment #34 from Max Staudt --- I'm afraid my knowledge of OpenGL is not deep enough to follow you here. On the other hand, if you have such deep insight, maybe that will help get upstream's attention? If there's a common open source game that exhibits this problem, it could serve as a minimal test case. The question remains whether this is caused by a simple failure to free() some memory, as you initially suspected, or something else. In any case, your new comment sounds like you finally have a good reproducer, so that's great! As for your report back in Comment 31: The dmesg output is uneventful I'm afraid. If anything, it would be [radeon] printing interesting messages. I'm surprised the bash line didn't work - I suppose that's because the last line is different? It seems like there is a summary being printed - maybe it's enough to do the following: watch -n 1 tail /sys/kernel/debug/dri/0/radeon_vram_mm watch -n 1 tail /sys/kernel/debug/dri/0/radeon_gtt_mm ...and then run a game with skeletal meshes. Can you please check whether supertuxkart exhibits this issue, too? That would greatly simplify reproduction of the bug. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1046962 http://bugzilla.opensuse.org/show_bug.cgi?id=1046962#c36 --- Comment #36 from Max Staudt --- (In reply to Mircea Kitsune from comment #35)

For the time being, I've decided to test whether this also happens with the RadeonSI scheduler. To make sure I'm applying it to all games across the system, I've added the following line to ~/.profile and restarted:

export R600_DEBUG=sisched

Oh, I didn't know that. Thanks! As for SuperTuxKart: I'm not sure it actually uses the functionality that breaks your system. It's just a guess since AFAIK it's a rather modern game using an engine that supports skeletal animations. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1046962 http://bugzilla.opensuse.org/show_bug.cgi?id=1046962#c38 Mircea Kitsune changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P3 - Medium |P2 - High Hardware|Other |64bit Target Milestone|--- |Current OS|Other |Linux --- Comment #38 from Mircea Kitsune --- Wasn't sure whether to bump this same bug report, as the original issue has clearly been fixed during nearly an year of countless Kernel + Mesa + driver updates. Unfortunately I now experience a new issue acting just like what I described here at the time: When certain 3D engines are running, there is a chance that after a few minutes the machine instantly freezes and becomes fully unusable until powered off and back on. I don't know when the new crash was implemented since I haven't played a lot of 3D games recently, but I'd assume somewhere within the last few months. I now have Kernel 4.15.3 and Mesa 18.0.0. Again my video card is a Radeon R7 370 from Gigabyte (RadeonSI, GCN 1.0, AMD Pitcairn Islands). I'm running the openSUSE Tumbleweed x64 rolling release distribution. Can someone please explain a way to debug those instant system freezes as they're added to the system components? I can't get an output at the time of the crash as the entire machine stops working and becomes bricked until restarted (likely including SSH), but maybe I can make it log info that I can retrieve after I reboot? Any useful info will help, just please nothing dangerous that might permanently break my OS. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1046962 http://bugzilla.opensuse.org/show_bug.cgi?id=1046962#c39 --- Comment #39 from Mircea Kitsune --- I can confirm that the exact same system freeze happens with both the radeon and amdgpu driver: Using either module makes absolutely no difference. This is the last piece of confirmation I needed to conclude that what's happening must be a deliberate malware: There is simply no way a GPU crash bug could behave 100% the same way on two entirely different video drivers. Further more, this freeze is completely identical to the one I initially reported here... which was obviously fixed since there's been so many updates to every system component it would have been solved by sheer chance at this point! Functionality like this should only be seen if someone is actively re-implementing the problem on top of updated system components, with the active intent of keeping its effects identical each time. It's possible that my machine may be used to test malware usable to shut down Linux systems, in which case I need to find out where it's hidden and how it's bricking computers before it spreads. This attack must exploit vulnerabilities that keep coming and going in X11, Mesa, or some other system component... those are hopefully holes which can be discovered and plugged to render the attacks impractical altogether. Again I only know that it's triggered while certain 3D engines are running (possibly aimed primarily at gamers?) and has a random chance of happening roughly once every 30 minutes (likely to make testing harder and better hide the exploit). -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1046962 http://bugzilla.opensuse.org/show_bug.cgi?id=1046962#c40 Mircea Kitsune changed: What |Removed |Added ---------------------------------------------------------------------------- Status|CONFIRMED |RESOLVED Resolution|--- |FIXED --- Comment #40 from Mircea Kitsune --- This specific bug has been fixed by various updates during all this time, and its logs are also very outdated. I've made a new report about a new and highly similar GPU crash which may be related to what happened here: https://bugzilla.opensuse.org/show_bug.cgi?id=1084767 -- You are receiving this mail because: You are on the CC list for the bug.