WhatPulse Forums » Support » Client software v » Whatpulse 2.0 saving to db file every second? Welcome back, Guest.


Thread Closed 
 
Thread Rating:
  • 6 Votes - 4.17 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Whatpulse 2.0 saving to db file every second?
12-23-2012, 12:55 PM
Post: #1
Whatpulse 2.0 saving to db file every second?
The title says it all. Why oh why is the client saving the data to the sqlite database every friggin second?

That's a <4kb write every second. That is going to kill peoples' SSDs.
And for the people that are still using HDDs it means that their harddrives will always be in full operational mode and never ever start to use their energy saving features.

This is completely unacceptable behavior. If that won't change, I'm no longer using it. I can think of way more reasonable and useful ways to kill my SSD.
Find all posts by this user
12-23-2012, 03:45 PM (This post was last modified: 12-23-2012 03:46 PM by borandi.)
Post: #2
RE: Whatpulse 2.0 saving to db file every second?
Well, yes and no.

An SSD can write to a page at a time - a page is typically 4 kB. Though when the SSD has to delete data, it has to delete a block (512kB) at a time. When Whatpulse writes 4kB at a time, it writes to a page. When it needs another 4kB, it writes to another page, and makes the previous page 'invalid', so the SSD can garbage it later. When the block is full, it will just move onto another block, until the whole drive is full of invalid pages. Then it'll clean them up (garbage collection or other) and start again.

http://www.anandtech.com/show/2738/5

Every 4kB write makes an SSD write a whole page of data. Even with wear levelling, write amplification and spare area, those 3000-20000 P/E cycles (depending on the drive) will not wear down quite quick. Let's do the math...

Assume 128GB SSD (including spare area), empty.
Each 4kB write requires a whole to be written
The way an SSD works is that if 4 kB is free in a block, it will write only the new page and 'lose' the address to the old data.

A 128GB drive will have
128 GB / 4 kB = 33554432 pages
Which means (33554432 / 3600) = 9320 hours to go through one entire P/E cycle

@ 3000 P/E cycles = 873813 hours = 3192 years
@ 5000 P/E cycles = 5320 years
@ 10000 P/E cycles = 10640 years

While I agree with you that having continuous writing could be bad for an SSD, it is not as bad as you think. It is when the drive has to delete and rewrite a whole block of data when the drive gets full which really screws it up.

Hence why Anand + Kristian @ Anandtech always suggest leaving 10-20% free space on your SSD.
Find all posts by this user
12-23-2012, 05:22 PM (This post was last modified: 12-23-2012 05:23 PM by dgschrei.)
Post: #3
RE: Whatpulse 2.0 saving to db file every second?
True, in comparison to how many P/E cycles you have in a modern SSD the amount it will write is small. It won't outright kill the SSD but still consume a sizable amount of writes.
In one of his articles Anand mentioned that he averaged 7GB of writes a day and that he considered himself an above average user.

With a 4kb write every second and let's assume a write amplification of 2 for such small writes:

4kb * 3600* 24 * 2 = 691200 kb = 675 MB

That's almost 10 % of Anand's workload. And that's just for the log of Whatpulse. That's a bit excessive.
The problem with standard HDDs that I pointed to also still remains.
Basically with a write every second, the drive heads will never be parked and the drive will also never have the possibility to shut down entirely. In Desktop systems that's not ideal but also not that big of a problem. On notebooks on the other hand this will not only impact battery life by a noticable amount, it will also make the harddrive a lot more vulnerable to shocks, because the drive heads will always be positioned above the platters.

And that's for basically no good reason. At least I can't think of a reasonable excuse as to why Whatpulse needs to persist every datapoint it collects directly and immediatly into the database. Even Windows' own logfiles don't use such a high frequency.
It just seems excessive to almost borderline paranoid to persist every second.
Find all posts by this user
12-23-2012, 05:50 PM
Post: #4
RE: Whatpulse 2.0 saving to db file every second?
I'm pretty sure Anand quotes in his recent articles that ~10GB writes a day for an 'average' user, but it's nothing for a power user.

Perhaps I should mention Anand is my direct boss Big Grin

I do wholeheartedly agree that it is an issue though, especially when you put mechanical drives or mobile devices in the mix.

I suspect that it is writing every second to update the 'uptime' component. Realistically having it to the nearest second is pretty pointless. Once a minute is more than enough.
Find all posts by this user
12-23-2012, 07:02 PM (This post was last modified: 12-23-2012 07:04 PM by dgschrei.)
Post: #5
RE: Whatpulse 2.0 saving to db file every second?
First of all: Man I wish I was working for Anand Big Grin

Recording the uptime of the system and the applications running on it might be the culprit here, but that could just as easily be saved out on exit of the application. Or since the Application activity page seems to work in hourly increments, once per hour.

Well I got curious so i peeked into the db file and boy does this track a lot of data points. They basically save out all the applications, all the network activity (there's even a table network_interface_bandwidth_second and another one "application_bandwith_second"), all the keystroke frequencies (duh) all clicks and all applications and their uptimes. Let's just hope that EA never gets their hands on this code or Origin will turn into Skynet in no time Wink

Then there is the "mousepoints" table. It does exactly what it sounds like. It contains the position of the mouse pointer at any given time. Sure that needs to be recorded in order to generate the mouse-pointer heatmap. But over a few hours that table alone has accumulated 2206 entries.

In fact with that many tables and that many datapoints the db file will most likely grow quite big over the months. Over the past 5 hours it has grown by a 120 kb. And that's even though I didn't install the network monitoring component. Wink
Find all posts by this user
12-24-2012, 10:45 AM
Post: #6
RE: Whatpulse 2.0 saving to db file every second?
I'm quite concerned about this. The client does indeed seem to perform an awful lot of writes per unit of time. I wouldn't want this to wear down my SSD while I can't think of a reason for not buffering the collected data in RAM before writing to disk. Could one of the developers comment on this? I'd like to be reassured it's either not harmful or absolutely necessary and worth it Wink

In the meantime I'll downgrade to the final 1.x version... oh wait, that's not allowed due to some kind of "security reason". In my opinion that's a pretty lame move since with the release of a major rewrite it's to be expected people will run into issues.
Find all posts by this user
12-24-2012, 10:53 AM
Post: #7
RE: Whatpulse 2.0 saving to db file every second?
(12-24-2012 10:45 AM)buzzer9v Wrote:  In the meantime I'll downgrade to the final 1.x version... oh wait, that's not allowed due to some kind of "security reason". In my opinion that's a pretty lame move since with the release of a major rewrite it's to be expected people will run into issues.

I'm guessing that's because you can run the 2.0 client and an 1.x one at the same time, so that security measure prevents you from being able to effectively use both on different profiles and double your key and click count.
Find all posts by this user
12-24-2012, 11:13 AM
Post: #8
RE: Whatpulse 2.0 saving to db file every second?
I didn't have much to add to borandi's post. It's really not a big issue, just a mindset. Would you rather have a big burst of data, or a steady stream?

You can restore pulsing for 1.x clients by recreating the computers you use with 2.0, so the '2.0' tag disappears from your computers and the 1.x pulse process doesn't see any 2.0 clients in your account.
Visit this user's website Find all posts by this user
12-24-2012, 11:57 AM
Post: #9
RE: Whatpulse 2.0 saving to db file every second?
(12-24-2012 11:13 AM)smitmartijn Wrote:  I didn't have much to add to borandi's post. It's really not a big issue, just a mindset. Would you rather have a big burst of data, or a steady stream?
All right, thank you for sharing your views. I'll consider the information borandi posted. Thanks for creating the client by the way, it has some pretty cool new features. I'm by no means ungrateful, just trying to be protective of my rig Smile

(12-24-2012 11:13 AM)smitmartijn Wrote:  You can restore pulsing for 1.x clients by recreating the computers you use with 2.0, so the '2.0' tag disappears from your computers and the 1.x pulse process doesn't see any 2.0 clients in your account.
Ah, that's perfectly fine then. Perhaps you should re-phrase the notice on the download page a bit to reflect this, since it pretty much suggests it's a point of no return.
Find all posts by this user
12-24-2012, 04:14 PM
Post: #10
RE: Whatpulse 2.0 saving to db file every second?
(12-24-2012 11:13 AM)smitmartijn Wrote:  You can restore pulsing for 1.x clients by recreating the computers you use with 2.0, so the '2.0' tag disappears from your computers and the 1.x pulse process doesn't see any 2.0 clients in your account.


How exactly do you recreate the computers?


Cheers,

Pete
Find all posts by this user
Thread Closed 


Possibly Related Threads...
Thread: Author Replies: Views: Last Post
Bug Whatpulse keeps me logging off shadowrune 1 291 09-13-2020 10:08 AM
Last Post: shadowrune
  WhatPulse hangs and crashes on Windows startup td95 0 93 08-17-2020 07:20 AM
Last Post: td95
  “Cannot open shared object file” pops up when installing WhatPulse Gaurav Mall 0 317 04-22-2020 06:08 PM
Last Post: Gaurav Mall
  Cannot start WhatPulse using Ubuntu 19.10 Adrriii 7 1,677 03-31-2020 09:39 PM
Last Post: Lieutenant_Dan
  Cannot start Whatpulse successfully on Manjaro 18.1.4 (Arch Linux) anijatsu 2 1,077 01-12-2020 04:06 PM
Last Post: Adrriii
  Whatpulse could not find OpenSSL Adrriii 0 645 12-03-2019 12:52 AM
Last Post: Adrriii
  Having trouble running whatpulse on Neon Linux BlackWidow 1 819 12-03-2019 12:02 AM
Last Post: Adrriii
  Whatpulse crash [GoT]Jewest 3 1,331 10-28-2019 01:18 PM
Last Post: [GoT]Jewest
  Updating from jessie to buster breaks whatpulse [GoT]Jewest 2 1,017 10-28-2019 01:17 PM
Last Post: [GoT]Jewest
  Server error: Missing input! with whatpulse 2.8.4 on Win10 Franck.Dernoncourt 3 1,567 09-24-2019 10:56 PM
Last Post: kolah

Forum Jump:


User(s) browsing this thread: 1 Guest(s)