WhatPulse Forums » Support » Client software v » [SOLVED] Whatpulse using 300MB of RAM Welcome back, Guest.


Thread Closed 
 
Thread Rating:
  • 1 Votes - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
[SOLVED] Whatpulse using 300MB of RAM
04-15-2014, 05:04 AM (This post was last modified: 09-28-2014 02:55 PM by smitmartijn.)
Post: #1
[SOLVED] Whatpulse using 300MB of RAM
So previously I posted about Whatpulse using 200MB of RAM though it didn't seem like anyone could come up with a solution. Since then I've updated from version 2.3 to 2.3.1 hoping that would fix it and now it's using 300MB of RAM.


Attached File(s)
.jpg  ram.JPG (Size: 28.64 KB / Downloads: 433)
Find all posts by this user
04-15-2014, 08:13 AM
Post: #2
RE: Whatpulse using 300MB of RAM
The solution from all the others posts was; don't leave the window open. Have it closed when you're not using it.
Visit this user's website Find all posts by this user
04-19-2014, 03:08 PM
Post: #3
RE: Whatpulse using 300MB of RAM
Doesn't help, mine goes up to 190 MB after a while.

Windows 10 Pro x64 / WhatPulse 2.7.2b1 - Beta
Find all posts by this user
04-26-2014, 02:40 AM
Post: #4
RE: Whatpulse using 300MB of RAM
Closing the window for me does the same thing as it does for Andreasvb. It drops down to around 5MB and climbs its way back up to 180MB within 10-15 minutes. It also seems like the larger the database the higher the RAM usage, because on my desktop which has the largest database by far I sit around 180MB and that number has slowly been growing over time as I add more input. My laptop, which I only use sparingly, has a much smaller database because of my lack of use and only uses 60MB of RAM. Finally on my computer I almost never touch because I have just been seeding things on it only uses 35-40MB of RAM and has the smallest database because of the fact that almost all of what is recorded is just network stats. Not sure if that helps any, but that seems to be the case for me.
Find all posts by this user
04-27-2014, 05:50 PM
Post: #5
RE: Whatpulse using 300MB of RAM
I usually test with small and rather large databases (+- 100MB), but haven't seen this happen. Could you put your database somewhere (dropbox/skydrive?) for me to download and run with and see what happens?

Maybe there's something about the way your stats are recorded.
Visit this user's website Find all posts by this user
05-07-2014, 03:07 PM
Post: #6
RE: Whatpulse using 300MB of RAM
(04-27-2014 05:50 PM)smitmartijn Wrote:  I usually test with small and rather large databases (+- 100MB), but haven't seen this happen. Could you put your database somewhere (dropbox/skydrive?) for me to download and run with and see what happens?

Maybe there's something about the way your stats are recorded.

https://drive.google.com/file/d/0B1XGacZ...sp=sharing

Sorry for the late response. End of the semester, so a mixture of doing nothing and everything at the same time.
Find all posts by this user
05-17-2014, 10:41 AM
Post: #7
RE: Whatpulse using 300MB of RAM
Here's mine: https://mega.co.nz/#!h5cETZLA!T5Y8z2Lmsv...8_qYDXWeMA

Windows 10 Pro x64 / WhatPulse 2.7.2b1 - Beta
Find all posts by this user
05-19-2014, 10:18 PM
Post: #8
RE: Whatpulse using 300MB of RAM
I think that it may be worth looking at the upload/download stats including the unplused traffic being seen on the systems that are showing the high memory usage.

For example I get this 250-300MB RAM issue on the one machine which is typically uploading at a fairly steady rate of around 1-2MB/s to the internet. Interestingly this same machine will also climb even higher when using my second life viewer remotely over the gigabit lan which when I do I use the -c rgb method (uncompressed video for higher image quality on high bandwidth networks) when this is in use the upload on the system is typically in the 1-2MB/s to the public internet plus an additional 70-90MB/s of traffic to 10.0.0.0/8 addresses.

When the latter is in use whatpulses memory will continue to rise even further prolonged session I have seen it exceed the 1.0GiB mark. I strongly suspect that on systems with large volumes of sustained network traffic.


I did some admittedly not particularly scientific testing around this issue which appeared to support the theory that the increase in memory usage in this particular case may very well be related to the fact that whatpulse for some reason opted to go the route of full libpcap packet capturing of the all network traffic (For reference I can't comment on such as windows operating system in this area, but for Linux/BSD and most likely Macs also this is not the optimal way to obtain the data necessary however that is a separate issue that I will try to find the time to elaborate on in a feature suggestion some time).

During the various tests I attempted I believe that I may have identified something that may point towards an optimization that could not only reduce this excessive memory consumption on these systems but also reduce the CPU load which is also significant when the host has significant network activity.

My testing mostly involved comparing the memory consumed by whatpulse and the memory consumed by the also libpcap based packet capture application tcpdump. During the course of this testing I found that both programs when started at the same time increase memory utilization initially linearly before levelling off at a peak which is somewhat proportional to traffic. Experimenting I found that whatpulse appears to do so at an average of a little over 105% of the increase with tcpdump. However this figure was *only* true when tcpdump was capturing the entire full length ethernet frames, this would make sense for an IDS like snort or similar which needs to perform packet payload inspection.

What does not make sense to me in my testing is that when compared with tcpdump using the -s128 option to only capture the required data and avoid capturing, buffering and processing an extra kByte or more of utterly irrelevant payload bytes then the difference between whatpulse and tcpdump grows significantly, by approximately by a multiplier slightly over the average packet size divided by 128. This indicates to me that for some reason whatpulse is not using the snaplen feature to capture the data as minimally and efficiently as possible and I am unsure why this would be the case.


Maybe I am missing something that whatpulse needs to collect for the statistics that means it needs more than just the header, these are the data points that I am aware of which whatpulse absolutely needs:

IP/IP6 Source Address (To identify downloded data from the internet)
IP/IP6 Destination Address (Same for upload to the internet)
IP/IP6 Packet Length (To add this packets bytes to the byte counter)

Whatpulse premium I believe also would require the following data points:

IP Protocol Number/IP6 Next Header (Particularly proto 9 TCP or 13 TCP)
TCP/UDP Source/Destination Port (To match the socket & application)


And then I hit the point where I am at a loss, what else does whatpulse need to collect the stats? I can't think what I am perhaps not thinking of right now or perhaps the above list is sufficient, if it is then I would suggest checking the code and making sure that you are setting a snaplen that is no greater than 128 which is in most cases overkill in itself. Even a snaplen of 96 will be enough to capture the entire Ethernet, IPv6 and TCP headers including IPv6 and TCP options fields and into the payload I have not actually seen any standard networking software which ever seems to produce packets with longer headers, only specialized network probing and testing software and international vulnerability probes or active network attacks around the 1 packet per million sort of range.

It could be this is completely an unrelated issue or that my attempts at testing admittedly far from ideal, I'd have preferred to test and analyse this in a more controlled and rigorous testing situation and provided some hard data of to go with it but I have no idea how I would go about that with whatpulse as I am not aware of any published source or anything. Anyway hope that something here might be of some value.
Find all posts by this user
05-20-2014, 05:05 PM
Post: #9
RE: Whatpulse using 300MB of RAM
Thanks for your pretty long research post, looks good.

(05-19-2014 10:18 PM)MttJocy Wrote:  This indicates to me that for some reason whatpulse is not using the snaplen feature to capture the data as minimally and efficiently as possible and I am unsure why this would be the case.

You're probably unsure on the reason, as the client doesn't not capture the entire packet, as you're thinking. It just takes the header (64bytes) so it can see the packet length, IPs and ports.

I have not been able to see the connection which you are seeing with the network traffic. I've got an offline 2.3.1 client running as a test on a file server for a few months and the memory usage is around 20MB, as it should be.

I'm going to look into the database combination with the databases posted here.
Visit this user's website Find all posts by this user
06-09-2014, 02:24 AM
Post: #10
RE: Whatpulse using 300MB of RAM
So I thought I'd come to check if there was a solution or anything because I got reminded of the issue when I looked at my task manager, screenshot here http://gyazo.com/d1455b38a9568ebff53ec6981d4f22d6

I never have the window open so that's not the issue. It's always minimized to tray.

I don't really understand what MttJocy's research is, I'm not that technically inclined to try to interpret the results of the research. However, I did turn off the network statistics in WhatPulse so it's not recording network stats anymore but is still using just as much ram.

There is one thing that strikes me as odd though, I'm looking at WhatPulse's uptime menu and it says my current uptime is 14 hours, but I only turned on the computer a couple of hours ago... and my longest uptime is 1993 days long. Either I'm misunderstanding something or it's displaying something wrong.

And lastly, database(260MB). https://drive.google.com/file/d/0BwZo1JL...sp=sharing
Find all posts by this user
Thread Closed 


Possibly Related Threads...
Thread: Author Replies: Views: Last Post
  Whatpulse could not find OpenSSL Adrriii 0 28 12-03-2019 12:52 AM
Last Post: Adrriii
  Having trouble running whatpulse on Neon Linux BlackWidow 1 65 12-03-2019 12:02 AM
Last Post: Adrriii
  Cannot start WhatPulse using Ubuntu 19.10 Adrriii 1 127 11-01-2019 02:37 PM
Last Post: Adrriii
  Whatpulse crash [GoT]Jewest 3 243 10-28-2019 01:18 PM
Last Post: [GoT]Jewest
  Updating from jessie to buster breaks whatpulse [GoT]Jewest 2 233 10-28-2019 01:17 PM
Last Post: [GoT]Jewest
  Server error: Missing input! with whatpulse 2.8.4 on Win10 Franck.Dernoncourt 3 578 09-24-2019 10:56 PM
Last Post: kolah
  Whatpulse isn't running after upgrading Ubuntu 1geeky 0 258 08-06-2019 05:46 AM
Last Post: 1geeky
  WhatPulse 64bit install on Lubuntu 19.04 SiKTheGreatOne 0 420 06-30-2019 02:49 AM
Last Post: SiKTheGreatOne
  [SOLVED] Can't run the Linux version [Driver not loaded Driver not loaded] radioz 2 1,121 06-20-2019 11:54 AM
Last Post: radioz
  WhatPulse not opening on Manjaro Linux (Arch distro) tboxley 4 808 05-26-2019 11:30 AM
Last Post: FamilienVater

Forum Jump:


User(s) browsing this thread: 1 Guest(s)