Logbook file storage

Post a question about the software or find answers here.

Re: ST4

Postby admin » Wed Jan 13, 2016 12:38 pm

@FredericB. Nope. We actually don't find file storage to be a technical limitation at all. Disks today are huge, fast, and cheap. In any case, you're probably aware that a SQL database is stored on your computer as files. :D

But more importantly, code changes like this are mostly invisible to the user, and as already stated in the reply to @texmurphy, they wouldn't be enough to motivate an entirely new major product version. I know if a product announced a brand new major version update (possibly including an upgrade fee), and the feature list said: "handle users with very large workout histories", I would be pretty disappointed.

Seems like we veered a bit off into technical conjecture. I'll split these posts into a more relevant topic if I see people want to keep talking about it, or merge with the existing "memory limit" post.

Cheers
Get the latest info: Fan us on facebook or follow us on twitter
admin
Site Admin
Site Admin
 
Posts: 3013
Joined: Tue Apr 05, 2005 8:52 pm
Location: USA

Re: ST4

Postby admin » Wed Jan 13, 2016 1:12 pm

SportTracks doesn't store sample data in XML. The app uses a binary packed format, which is adaptive to the data sample range, workout duration, and sample frequency. The data storage even has a simple compression algorithm tailored to the typical usage of time-based GPS and sensor samples. It may also interest you that the online .mobi platform doesn't use SQL for sample data storage either.
Get the latest info: Fan us on facebook or follow us on twitter
admin
Site Admin
Site Admin
 
Posts: 3013
Joined: Tue Apr 05, 2005 8:52 pm
Location: USA

Re: Logbook file storage

Postby texmurphy » Wed Jan 13, 2016 1:47 pm

The Logbook size issue is a problem with MS .net which faults with OOM at working set over 1.6GB. Even below 1.6 there can be significant performance issues due to .net garbage collection and cleanup. Until MS allows larger memory architecture with .net, ST3 will limit the Logbook. I can hold about 3.5 years at 13000km/year with one second recording of trackpoints, elevation, HRM,Cad, & Power using admin's Compression Plugin.

For those with performance issues, I suggest 12GB main memory, a SSD hard drive, and processor upgrade or overclocking. If you are running on a laptop .computer, then a SSD will yield huge bang for buck!
texmurphy
Donated!
Donated!
 
Posts: 2126
Joined: Wed Jul 05, 2006 7:38 pm
Location: Maryland, USA

Re: Logbook file storage

Postby admin » Wed Jan 13, 2016 2:17 pm

IIRC a quick diagnostic we did a year or so ago showed about 60-80% of memory was caching calculation results to speed up program responsiveness. I may be remembering that wrong, but we've got a minimal 2-3x headroom on max size with more intelligent caching. Or just not caching at all, and letting people sit and wait for the program to respond.

Plugins are an entirely different scenario. We don't have much control over their implementations. With very large data histories, it's more likely a "sloppy" plugin will clog up memory.

@texmurphy - What is the average total workout hours / per year you're looking at currently? That number is more relevant than km.
Get the latest info: Fan us on facebook or follow us on twitter
admin
Site Admin
Site Admin
 
Posts: 3013
Joined: Tue Apr 05, 2005 8:52 pm
Location: USA

Re: Logbook file storage

Postby texmurphy » Wed Jan 13, 2016 4:56 pm

Veloviewer.com stats based on Strava uploads for 2015, 2015 was a just below average year. I am not near Logbook to give actual with ST3 stats.

2015 Hours 461
2015 Miles 7584
2015 Activity days 194
texmurphy
Donated!
Donated!
 
Posts: 2126
Joined: Wed Jul 05, 2006 7:38 pm
Location: Maryland, USA

Re: Logbook file storage

Postby admin » Wed Jan 13, 2016 5:41 pm

Good enough.

A rough analysis shows your history size is in our top 0.5%, or about 1 in 200 of our users.

About 50% have a 5+ sensor setup, but the combination of 5+ sensors with high yearly total workout duration is unusual.

I can't quickly see sample frequency. I doubt 100% are using 1-second recording, so that 1 in 200 is a low end.
Get the latest info: Fan us on facebook or follow us on twitter
admin
Site Admin
Site Admin
 
Posts: 3013
Joined: Tue Apr 05, 2005 8:52 pm
Location: USA

Re: Logbook file storage

Postby the5krunner » Thu Jan 14, 2016 5:10 am

texmurphy wrote:
For those with performance issues, I suggest 12GB main memory, a SSD hard drive, and processor upgrade or overclocking. If you are running on a laptop .computer, then a SSD will yield huge bang for buck!


yes an SSD is amazing for those who are yet to try (Samsung 850 evo will do nicely)

but, anyway, an SSD will only speed up what is there.

If there are structural/architectural limitational issues then it won't help...I think that's right?

so there must be personal performance bottlenecks and sporttracks/plugin performance bottlenecks (caused by .net or whatever). ?
the5krunner
 
Posts: 458
Joined: Sun Feb 22, 2009 1:41 pm

Re: Logbook file storage

Postby texmurphy » Fri Jan 15, 2016 11:48 am

texmurphy wrote:Veloviewer.com stats based on Strava uploads for 2015, 2015 was a just below average year. I am not near Logbook to give actual with ST3 stats.
2015 Hours 461
2015 Miles 7584
2015 Activity days 194

Below are current Logbook Stats:
Tex Logbook Stats.jpg
Tex Logbook Stats.jpg (182.84 KiB) Viewed 10624 times
texmurphy
Donated!
Donated!
 
Posts: 2126
Joined: Wed Jul 05, 2006 7:38 pm
Location: Maryland, USA

Re: Logbook file storage

Postby texmurphy » Mon Jan 18, 2016 4:54 pm

A couple of interesting threads on .net addressing. The first also notes the 1.6 GB working set limit which I have experienced:

Memory limit of a 32bit .NET process on 64bit OS wrote:https://social.msdn.microsoft.com/Forums/en-US/c7f5b1bc-c50f-4747-91e9-cd122593124b/memory-limit-of-a-32bit-net-process-on-64bit-os?forum=netfx64bit
I'm working on a heavy server application, that allocates huge amount of memory (this is a normal behavior).
As we are stuck to the 1.6 GB memory limit, we tried to make it run on a 64bit version of Windows, with no success.
The application is full .NET, except some external unmanaged librairies, compiled in 32bit. Thus, our application can run only in 32bit mode.
On a 32bit server, the application works (somewhat) fine until the 1.6GB limit. On 64bit, it crashes at 1GB with an OutOfMemory.
I have added the /LARGEADDRESSAWARE flag on the binaries, and it could go beyond the 1GB limit, but it keeps doing GC2.
Note here that GC2 are expected. The application is creating a lot of long term objects.

What is not expected is that the GC2 keeps running, limiting the memory usage to 1.4GB, even with the /LARGADDRESSAWARE. 25% of the running time is used by GC2, which is not acceptable for our application. The OS has 16GB RAM, with 14GB free before running the application.
My question is : why do the GC2 keeps running to keep the process under the 1.4GB limit, where it could go up to 3GB? Is there something I'm missing to remove this threshold ?


And from Mark Russinovich's technical blog covering topics such as Windows troubleshooting, technologies and security see:
Pushing the Limits of Windows: Virtual Memory

It looks like it might be possible to up the addressing limit to 4GB, but likely the various plugins and their libraries might fail.
texmurphy
Donated!
Donated!
 
Posts: 2126
Joined: Wed Jul 05, 2006 7:38 pm
Location: Maryland, USA

Re: Logbook file storage

Postby texmurphy » Mon Jan 01, 2018 2:57 pm

Adding new activities to my Logbook has been getting real iffy since July. But I have been persevering! The strategy the past several months has been to always have the Route=NONE on ST3 close. Then it will open without any maps and be able to read the Logbook.

But it is now the new year - 2018 - and I have removed 2012 and 2013 from my "current" logbook. This always takes a bit of work to preserve the mileage use of my equipment! I hope to get two more years going forward with this platform.

remove 2012&2013.jpg
Removed 2012 and 2013 so I can use Logbook for 2018.
remove 2012&2013.jpg (36.87 KiB) Viewed 8196 times
texmurphy
Donated!
Donated!
 
Posts: 2126
Joined: Wed Jul 05, 2006 7:38 pm
Location: Maryland, USA

Re: Logbook file storage

Postby gregoryx » Thu Feb 08, 2018 1:05 pm

Thanks for posting your details on this. My logbook seems to be getting large and slow as well... but seeing that yours is much worse is quite encouraging.

:lol:

st3log.jpg
st3log.jpg (18.19 KiB) Viewed 7765 times
Image
It's all running.
gregoryx
 
Posts: 76
Joined: Tue Aug 09, 2011 9:46 pm


Return to Questions

Who is online

Users browsing this forum: No registered users and 1 guest

cron