Crash on Windows (was: High memory usage) #964
Labels
No Label
bug
build
dependencies
developers
documentation
duplicate
enhancement
formatting
invalid
legal
mobile
obsolete
packaging
performance
protocol
question
refactoring
regression
security
test
translation
usability
wontfix
No Milestone
No project
No Assignees
1 Participants
No due date set.
Dependencies
No dependencies set.
Reference: Bitmessage/PyBitmessage-2024-08-21#964
Loading…
Reference in New Issue
Block a user
No description provided.
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Bitmassage uses over 1GB memory.
It might be caused by a ton of threads for many connections.
I suggest using "gevent" to reduce the number.
Thank you.
Duplicate of #870
Sorry I didn't notice that.
No problem, I hope it will be fixed in 0.6.3.
Sorry for my English.
After upgrade to 0.6.2,The memory usage is too much.
On my Windows 7 SP1 x64 ,When Bitmessage running more than 8 hours, The bitmessage.exe use most 2GB memory.
I try set maxtotalconnections = 16 by edit key.dat file,but not work.
( https://twitter.com/PeterSurda/status/840029569235202048 )
This issue is not happen at 0.6.1.
If need more information please let'me know.
Thanks.
Thank you, this is helpful. I think it's the threaded networking system causing this, it will be replaced in 0.6.3. I can send you a prerelease once it's done so that you can test it.
it can be memory leak.
it uses 767MB for just 20 connections.
key.dat
I'm open to the idea that it's a memory leak but I'm also observing a lot of memory being used with 0.4.4 (for comparison), scaled down to the number of connections. The memory with 0.6.2 also seems to scale with the number of connections rather than uptime. I have nodes which have been running 0.6.2 since it was released. One has 300 connections and if you divide its memory allocation and compare it to another node with 0.4.4 which has 18, it seems to allocate the same amount of memory per connection.
The reason why it looks like it was using less memory before is that the networking subsystem had bugs and deadlocks, the connections were short-lived and died frequently, so you couldn't have a large number of connections.
So in other words, it has required an excessive memory for a long time, but the bottleneck was in the network bugs. Now that that is fixed, the memory is the bottleneck.
Please try the latest development snapshots (20170529) at https://bitmessage.org/download/snapshots/ and let me know if it's better. You can bump maxtotalconnections to 100 if you want.
Test is running,I will tell you result ASAP.
And Thanks. :)
After 9 hours test,The memory usage look's stable.
But this version (Bitmessagedev_x64_20170529.exe) bitmessage look's fully use one cpu core all the time.
And this crashed one time,I set up a memory dump collection ,If crash again , I can send you the memory dump file.
Edit:Dump file has send to your email address. @PeterSurda
Thank you, I got other reports similar to yours. Unfortunately I don't know what to do with a windows coredump but maybe someone else can help. On Linux I haven't had a crash, but I occasionally get a "maximum recursion depth exceeded" error, maybe that's the same thing, I have a rough idea about what is causing it and hope to fix it within a couple of days. Once I can't see the errors anymore and I can't reproduce the crash on my Windows VMs anymore, I'll make a new snapshot.
Thank you,Waiting new update. 💯
Edit:
I'm try use WinDBG to analyze the dump file.But no symbol file,the stack may not correct.
@PeterSurda
in the current 0.6 branch I get such also:
_Exception RuntimeError: 'maximum recursion depth exceeded while calling a Python object' in <_ctypes.DictRemover object at 0x7f2dcae4c570> ignored
Exception RuntimeError: 'maximum recursion depth exceeded while calling a Python object' in <ctypes.DictRemover object at 0x7f2dcae4ce50> ignored
no crash yet, BM uses about 208 MB RAM, linux
(on request I can send more info)
That's odd, I thought I fixed that recursion thing yesterday. Are you sure you're running the latest code? I don't see it anymore on any of my test linux systems and I haven't had a crash on a windows VM today either. One of my linux sytems has 132 connections and the process occupies slightly less than 1GB RAM.
I am running this branch with this commit as last
ok thanks I'll check it out.
I see it again now.
Can you try with the current code?
I Tested Bitmessagedev_x64_20170609.exe ,Look's fine.
Memory usage 100MB~150MB.
No crash for last 24 hours.
great :-). If you're having problems in the future, just try the latest dev snapshot, they are now generated automatically every day. I'll leave the ticket open because there are still some bugs I have to fix.
It crash again :(
I guess need more details in logs,Like debug mode etc..?
still seeing it in the 0.6 branch (A):
Exception RuntimeError: 'maximum recursion depth exceeded while calling a Python object' in <_ctypes.DictRemover object at 0xaa9275c0> ignored
Exception RuntimeError: 'maximum recursion depth exceeded while calling a Python object' in <_ctypes.DictRemover object at 0xaa927420> ignored
Exception RuntimeError: 'maximum recursion depth exceeded while calling a Python object' in <_ctypes.DictRemover object at 0xaa9275d0> ignored
(A) with today's version, commit
189578cThe recursion problem should now be handled (there will still be an error in the log, but it shouldn't crash anymore). Let me know if it still appears. For Windows, wait about 8.5 hours and try tomorrow's snapshot (20170707), they are now generated automatically.
Thank you,I'm testing Bitmessagedev_x64_20170707.exe .
Actually there are bugs in the latest code, you're better off waiting until tomorrow.
OK,I'm wait,Thank you. :)
The Bitmessagedev_x64_20170708 have a strange issue:
After most 2 days,Bitmessage has disconnect form network ( red light )
I'm wating 1 hour,It still connecting.
I use Process Explorer to check Bitmessage TCP connect,It look's try connect to other client,but no one can connect.
The Internet connection is working,And in my same network,Another Bitmessage still working fine. (0.6.1)
Edit:After check my Route,The route has re-dial at 10 hours ago.This issue maybe about public IP change?
I restart bitmessage,And it connected.
Let me know if it gets better in tomorrow's snapshot.
I tested Bitmessagedev_x64_20170711.exe ,After few hours,the Bitmessage has crashed.
Thank you for the report, I'm working on it.
I tested Bitmessagedev_x64_20170719,The disconnect still happen.
Try tomorrow's (July 22nd) snapshot, I'm still not sure if it fixes everything but maybe it will be better.
I tested the Bitmessagedev_x64_20170723 version,The link look's keep connection ,but no sync and upload / download data after 2 days.
I try send a message from another 0.6.1 client,It can't get the message.
If I restart the bitmessage,everything is working again.
Update:
After test Bitmessagedev_x64_20171104.exe ,The Sync look's fine,No crash in 80 hours.
But memory usage still a bit high.
I think most of the crashes have been solved and the rest of them I think I know how to fix. Regarding memory, the screenshot shows 44 connections and 374MB of RAM. The high "Virtual size" is probably due to how python works (on linux it also shows gigabytes of virtual memory for pybitmessage). It doesn't seem to have much practical meaning.
Currently, pybitmessage requires about 10MB per connections, which includes IO buffers and inventory tracking (queues are global so they don't require per-connection memory anymore). I don't think I can improve that much. The default connection limit of 200 should fit into 2GB or RAM.
My test instance has now been running for almost 2 weeks and peaked at 262 connections and about 2.6GB of RAM.
After 90 hours,The Bitmessage crash again. 😌
Here is after crash memory usage status:
That's probably the recursion bug which should be fixed soon. I'll let you know when the fix is in the snapshot.
I may have fixed the recursion bug (I'm not sure as I still don't have a reliable way to reproduce it). Try tomorrow's (20171123) snapshot.
After 2 days,6 hours,The 20171124 version is crashed.
Thank you for the report. It looks like this is something Windows specific. I'll do more testing on Windows.
Were you doing something with it when it crashed?
The crash look's random,When I play games,watch youtube,or using ms word sometimes.
Could it be that it has something to do with your nvidia card, e.g. a driver update? Try turning off OpenCL under the "Max acceptable difficulty" tab in the settings.
All my drivers is manually update by myself.
And OpenGL is disabled in bitmessage by default (selected None).
I tested running PyBM while updating the nvidia driver (on Windows) and it seems to be fine.
I tested 20171201 After 8 days running, everything look's fine for now.
Thank you for the report. I wasn't able to reproduce the crash at all on any systems. There may be / might have been something on your computer (e.g. a bug in the nvidia driver that was fixed in the meantime). I'll close it for now.
after 10 days,it's crash again. :(
I'm not enable GPU accelerate or update drivers when this time.
how about the latest official release, 0.6.2, does that crash too?
After update to Bitmessage_x86_0.6.3.2.exe, I moving the Bitmessage to a new server (Windows Server 2008 R2).
The server only has a built-in video card in CPU ( Intel(R) HD Graphics Family ).
But after 6 days, It's crashed.
Log look does not have much info:
Crash report:
Thank you for the info, @chenshaoju . Maybe it has to do with Chinese locale. I will try to get a W2008 with Chinese locale to test.
It looks like the MSVCR90.DLL is not the latest one. I'll upgrade and rebuild.