[ST3] [FIXED] ST3 OutOfMemory exception when saving logbook

On the rare occasion you might find a bug...

[FIXED] ST3 OutOfMemory exception when saving logbook

Postby jibberjim » Sat Jul 16, 2011 3:18 am

Windows XP, ST 3.0.4205

When saving my logbook, I now get a System.OutOfMemory Exception, it's non-fatal, it just prompts me to save somewhere else - which fails of course. The problem is presumably that the 1.5gb HEAP size limit of .NET is being hit as I have plenty of system memory available. I'm guessing the logbook serialisation allocates everything and doens't release it until after the save is complete - even though it's only constructing a file that ends up at under 100mb.

1004 Activities
4.1million GPS points
10.5 million data track points

Just over 3 years of data

Can the save routines be updated to not exhaust the heap?

(I've actually managed to keep a larger logbook in ST2.1 with the OOM happening with around 300 more activities with their associated data logs - 10mb in file size)

I can provide the logbook for testing if you like.

Cheers!
jibberjim
Donated!
Donated!
 
Posts: 66
Joined: Thu Dec 18, 2008 2:15 pm
Location: Kingston-Upon-Thames

Re: OutOfMemory exception when saving logbook

Postby texmurphy » Sat Jul 16, 2011 7:43 am

Admin has my huge logbook which will also cause this fault if I do too many edits (either to Notes or Map or calculations like Course Score).
I am careful to limit such edits on newly imported activities. The fresh import seems to exhaust memory more then edits to existing activities.
ST will let you export fitlogs after the OutOfMemory, just will not allow logbook save.
texmurphy
Donated!
Donated!
 
Posts: 2186
Joined: Wed Jul 05, 2006 7:38 pm
Location: Maryland, USA

Re: OutOfMemory exception when saving logbook

Postby admin » Sat Jul 16, 2011 11:09 am

Our logbook save/load uses the .NET XML technology which has some limitation with large files (over 100MB). The upside of using this is it's quick to write the feature.

A complete rewrite of this is quite a lot of time and effort. We will continue to track the priority of this issue as other people report the problem.

The only possible workaround is to reduce the size of your data. You have two options:
1) Drop some of the older workouts completely (reduce your log to say.. 2 or 1 year of data)
2) Keep all the workouts, but drop some of the detail precision (switch the data source to "manual", and completely delete the GPS/Data tracks, or use something like the "reduce" methods available in plugins)

A second benefit to either of these methods: your system will run much faster.
admin
Site Admin
Site Admin
 
Posts: 3711
Joined: Tue Apr 05, 2005 8:52 pm
Location: USA

Re: OutOfMemory exception when saving logbook

Postby jibberjim » Sat Jul 16, 2011 12:41 pm

admin wrote:Our logbook save/load uses the .NET XML technology which has some limitation with large files (over 100MB). The upside of using this is it's quick to write the feature.

A complete rewrite of this is quite a lot of time and effort. We will continue to track the priority of this issue as other people report the problem.

The only possible workaround is to reduce the size of your data. You have two options:
1) Drop some of the older workouts completely (reduce your log to say.. 2 or 1 year of data)
2) Keep all the workouts, but drop some of the detail precision (switch the data source to "manual", and completely delete the GPS/Data tracks, or use something like the "reduce" methods available in plugins)

A second benefit to either of these methods: your system will run much faster.


It runs perfectly adequatedly all round already, there is no speed problem in use - in fact I think it's pretty unlikely that they'll be any speed improvement in the areas where speed is a problem as these are mostly in poor UI performance of .NET making things feel clunky (and the lack of things like the combo boxes working correctly with X-Mouse, but that I'm sure I'm alone in noticing) and network delays getting maps etc - rather than in processing delays which are always very impressive.

The improved performance of 2.1 over 3.0 does suggest that it's not simply a .NET issue - as despite more efficient compression on the data itself leading to smaller file sizes, the actual data you can have in the file is less.

The workarounds are no use to me, equally they do little but postpone the problem - for example removing all cadence data (easy to do in 2.1 - tougher in 3 data files) doesn't actually change anything - so the amount of useful data that can remain in such a file is minimal. Given that over 80% of my data is within an 18month period and dropping all old data means that SportTracks would really only be able to track a small amount of performance - particularly useless for comparing events which happen every year (how did I do compared to the same event the previous two years etc.) Yes you could split the file into many years and still access the previous data, but whilst you can have two instances running at once, you cannot do easy comparison of workouts across years - which as I say is an important use case.

I do believe there are drop in alternatives to XMLSerializer available which would overcome the memory implementation without a full rewrite being required?
jibberjim
Donated!
Donated!
 
Posts: 66
Joined: Thu Dec 18, 2008 2:15 pm
Location: Kingston-Upon-Thames

Re: OutOfMemory exception when saving logbook

Postby texmurphy » Sat Jul 16, 2011 1:51 pm

jibberjim wrote:It runs perfectly adequatedly all round already, there is no speed problem in use

Agreed. In fact some of the recent mapping improvements have greatly speeded up moving between activities.

The unfortunate thing is that ST suggests the problem can be resolved by selecting a different logbook save location. That does not work. The user must export via fitlog any important edits and then quit ST. The user can then restart ST and reload those fitlogs.
I have been doing this for several months now and I have yet to hit the wall or is it a cliff!?!

Win 7/64 Home Premium on i7 Quad with 6GB
Tex's Logbook wrote:Size: 123767 KB
Activities: 2050
GPS Data Points: 4487148
Data Track Measurements: 12893742
Activity Distance: 56689 miles
texmurphy
Donated!
Donated!
 
Posts: 2186
Joined: Wed Jul 05, 2006 7:38 pm
Location: Maryland, USA

Re: OutOfMemory exception when saving logbook

Postby admin » Sun Jul 17, 2011 12:47 pm

jibberjim:
Sorry no - unfortunately, there are no quick fixes or "drop in" alternatives.
Regarding my workaround suggestions - I think we're not communicating and/or you have some misassumptions. Much of what you are saying makes no sense to me, from a technical standpoint. And you can trust that I have a pretty deep understanding of the code. :wink:
If you want to pursue discussing performance workarounds in depth I'd suggest you email support.

texmurphy:
Only about 10 people have seen the 3.1 performance improvements. So...
admin
Site Admin
Site Admin
 
Posts: 3711
Joined: Tue Apr 05, 2005 8:52 pm
Location: USA

Re: OutOfMemory exception when saving logbook

Postby DerSegler » Wed Aug 24, 2011 2:03 am

Perhaps it is possible to implement an "auto-store-feature"?
As far as I understood the discussion (I had the problem yesterday) the problem comes with to much changes in the database; if there would be an auto-store every 10 min, I think, there would not be so much changes in this time.
Until there will be such a feature I will try to remember to store the logbook after every changing....

Manfred
DerSegler
Donated!
Donated!
 
Posts: 23
Joined: Mon Dec 22, 2008 6:23 am

Re: OutOfMemory exception when saving logbook

Postby Sirrocco » Fri Sep 23, 2011 6:38 am

I am experiencing "System.OutOf MemoryException" was thrown" faults on attempting to save the ST Logbook. I do not see in the posts regarding this problem what I should be doing. Can anyone in plain step by step explain how I am to fix this please. It has been doing this for about three days. I have all my training data in this program and do not want to lose anything so am in a bit of a panic in case it just stops and I lose everything. :?:
Sirrocco
 
Posts: 11
Joined: Sat Apr 30, 2011 8:31 pm
Location: Queensland, Australia

Re: OutOfMemory exception when saving logbook

Postby admin » Fri Sep 23, 2011 7:59 am

Sirrocco wrote:I am experiencing "System.OutOf MemoryException" was thrown" faults on attempting to save the ST Logbook. I do not see in the posts regarding this problem what I should be doing. Can anyone in plain step by step explain how I am to fix this please. It has been doing this for about three days. I have all my training data in this program and do not want to lose anything so am in a bit of a panic in case it just stops and I lose everything. :?:


You'll need to archive some of your older data into a separate logbook. Step by step fix:
1) Open Windows Explorer and copy and paste your current logbook to an archive, for example called "2010 History.logbook3"
2) Open SportTrack with your current logbook and change to Activity Reports view
3) Click the "Group by" button and select "Year, Month"
4) Click on the 2010 year row
5) Click the Edit > Delete button
6) Repeat for older data

You might try to start by purging old data first, say workouts before 2008 or 2009 and see if your logbook size is significantly reduced. It depends on your recording sampling rate (1-second, v smart), how long your workouts are, and how many data tracks you are recording (GPS, HRM, cadence, power, temp), etc.

Can I ask: Before doing this please go to Logbook > Settings and post a reply with the number of GPS points & data points and size of your logbook.
admin
Site Admin
Site Admin
 
Posts: 3711
Joined: Tue Apr 05, 2005 8:52 pm
Location: USA

Re: OutOfMemory exception when saving logbook

Postby texmurphy » Fri Sep 23, 2011 9:07 am

I am continuing to be able to add activities to my bulging logbook by some careful steps:
1. Untick to deactivate all plugins you do not use on a regular basis (only activiate those plugins when needed);
2. Minimize the number of edits you do to multiple activities between Saves;
2a. [edit] Any edits to Athlete (and maybe Equipment) should be done in a separate Save session from Activity edits
3. Before a Save, switch to Daily Activity to a day without any activities and switch to Map=NONE;
4. Use the Logbook->Save command to save the logbook, rather than Close ST3 and Save.

If Out Of Memory occurs, then you can export new/edited activities as a fitlog, close ST3, restart ST3 and import the fitlog. Then immediately Save. This usually works.

Tex's Logbook on 23 September 2011 wrote:Size: 132,439 KB
Activities: 2,104
GPS Data Points: 4,841,785
Data Track Measurements: 13,939,917
Activity Distance: 58,433 miles

I have acceptable to great ST3 startup and logbook load times and once Reports is first used, access to all activities is fast. If I do not get a OOM failure, then my logbook saves are also quick.

There are problems with a split logbook if you use Equipment tracking. You will need to compute which equipment items are in use at the split border and what their mileage is at that time. And there are many reasons to keep an complete single logbook. If your logbook is smaller than mine then you should be able to proceed for some time, as I intend to, without resorting to logbook splitting.
texmurphy
Donated!
Donated!
 
Posts: 2186
Joined: Wed Jul 05, 2006 7:38 pm
Location: Maryland, USA

Re: OutOfMemory exception when saving logbook

Postby Sirrocco » Fri Sep 23, 2011 7:51 pm

You'll need to archive some of your older data into a separate logbook. Step by step fix:
1) Open Windows Explorer and copy and paste your current logbook to an archive, for example called "2010 History.logbook3"
2) Open SportTrack with your current logbook and change to Activity Reports view
3) Click the "Group by" button and select "Year, Month"
4) Click on the 2010 year row
5) Click the Edit > Delete button
6) Repeat for older data

You might try to start by purging old data first, say workouts before 2008 or 2009 and see if your logbook size is significantly reduced. It depends on your recording sampling rate (1-second, v smart), how long your workouts are, and how many data tracks you are recording (GPS, HRM, cadence, power, temp), etc.

Can I ask: Before doing this please go to Logbook > Settings and post a reply with the number of GPS points & data points and size of your logbook.[/quote]

Activities 829
GPS Data Points 3683440
Data Track Measurements 11954677
Size 101166 KB

I have now separated the logbooks into years. I notice however that the athlete data is still in all of the new logbooks for all years. Whilst this is OK for me how does one delete this also. Also is this going to be fixed so that all the activities are in the one logbook? I like to compare Races etc whilst it is a moot point I guess opening another logbook still can do but why can not all the data be in one logbook. Thanks :)
Sirrocco
 
Posts: 11
Joined: Sat Apr 30, 2011 8:31 pm
Location: Queensland, Australia

Re: OutOfMemory exception when saving logbook

Postby rwborree » Fri Dec 02, 2011 12:32 pm

I didn't realize that it was a FEATURE of Version 3 that you are limited to the amount of data you could store in your data base. Had I know this was limiting FEATURE I am not sure I would have paid for this FULL PRICED feature the workarounds take me back to the FREE VERSION (limited functionality).

Is this something that is going to be addressed????????
rwborree
 
Posts: 7
Joined: Thu Sep 25, 2008 12:08 am

Re: OutOfMemory exception when saving logbook

Postby admin » Fri Dec 02, 2011 4:54 pm

rwborree wrote:I didn't realize that it was a FEATURE of Version 3 that you are limited to the amount of data you could store in your data base. Had I know this was limiting FEATURE I am not sure I would have paid for this FULL PRICED feature the workarounds take me back to the FREE VERSION (limited functionality).

Is this something that is going to be addressed????????


This has nothing to do with Version 3 and nothing to do with FREE vs TRIAL. I have no idea why you would even assume that.

The bug has been around since day 1. If you use 1-second recording and record 6 data tracks (GPS/ele/speed/HRM/cad/power) and you consistently do 4 hour bike rides, after some time (a year or so) you will probably hit this memory limitation. These kinds of data recording are relatively new, not possible a few years ago (smart recording, natch).

For those that hit this problem (2 people out of 80,000 so far), I am sending some experiemental data compression plugins. Contact admin via PM if you need assistance.
admin
Site Admin
Site Admin
 
Posts: 3711
Joined: Tue Apr 05, 2005 8:52 pm
Location: USA

Re: OutOfMemory exception when saving logbook

Postby rwborree » Fri Dec 02, 2011 6:23 pm

Curious if you understand what causes the memory error...., it happens about one out of every 5 to 7 saves of the log file..., sometimes you can save to a different location and it saves it some times is doesn't..., I can exit Sporttracks..., come right back in and it will save it..., just curious if you understand what the root cause is....,
rwborree
 
Posts: 7
Joined: Thu Sep 25, 2008 12:08 am

Saving Logbook

Postby DavidCHall » Mon Jan 09, 2012 6:43 am

DavidCHall
Donated!
Donated!
 
Posts: 938
Joined: Mon Sep 24, 2007 8:29 am
Location: Cape Town, South Africa

Re: OutOfMemory exception when saving logbook

Postby Renaldo » Sun Mar 25, 2012 9:57 am

admin wrote:
rwborree wrote:I didn't realize that it was a FEATURE of Version 3 that you are limited to the amount of data you could store in your data base. Had I know this was limiting FEATURE I am not sure I would have paid for this FULL PRICED feature the workarounds take me back to the FREE VERSION (limited functionality).

Is this something that is going to be addressed????????


This has nothing to do with Version 3 and nothing to do with FREE vs TRIAL. I have no idea why you would even assume that.

The bug has been around since day 1. If you use 1-second recording and record 6 data tracks (GPS/ele/speed/HRM/cad/power) and you consistently do 4 hour bike rides, after some time (a year or so) you will probably hit this memory limitation. These kinds of data recording are relatively new, not possible a few years ago (smart recording, natch).

For those that hit this problem (2 people out of 80,000 so far), I am sending some experiemental data compression plugins. Contact admin via PM if you need assistance.


Hi, Could you send me the experiemental data compression plugin? I contacted you via PM (providing details about my logbook) but haven't heard back from you.

Thanks in advance!
Renaldo
 
Posts: 1
Joined: Tue Sep 23, 2008 7:59 am

Re: OutOfMemory exception when saving logbook

Postby rwborree » Sun Mar 25, 2012 12:39 pm

Has there been any progress on this issue? Beside the crazy slow load time and the slow load of report view..., the SAVE function running out of memory has made the product unuseable. There was some talk of a compression plug in..., does it exist?

Is this an issue that is going to be addressed in the future or is it time to switch products? If you can't maintain a couple years of data it seems like a major shortcoming.

Interested in what others are doing...,
rwborree
 
Posts: 7
Joined: Thu Sep 25, 2008 12:08 am

Re: OutOfMemory exception when saving logbook

Postby admin » Mon Mar 26, 2012 7:04 am

rwborree wrote:Has there been any progress on this issue? Beside the crazy slow load time and the slow load of report view..., the SAVE function running out of memory has made the product unuseable. There was some talk of a compression plug in..., does it exist?

Is this an issue that is going to be addressed in the future or is it time to switch products? If you can't maintain a couple years of data it seems like a major shortcoming.

Interested in what others are doing...,

I have seven years of data and it loads very fast and is extremely responsive.

We are definitely looking at solutions to reduce data size for people that are doing 1-second recording on multiple sensors. The short term strategy is to ask support for the data compression plugin.
admin
Site Admin
Site Admin
 
Posts: 3711
Joined: Tue Apr 05, 2005 8:52 pm
Location: USA

Re: OutOfMemory exception when saving logbook

Postby rwborree » Mon Mar 26, 2012 7:55 am

How do I go about requesting the data compression plug-in?
rwborree
 
Posts: 7
Joined: Thu Sep 25, 2008 12:08 am

Re: OutOfMemory exception when saving logbook

Postby admin » Mon Mar 26, 2012 9:49 am

rwborree wrote:How do I go about requesting the data compression plug-in?

The best way to contact support is via email.
admin
Site Admin
Site Admin
 
Posts: 3711
Joined: Tue Apr 05, 2005 8:52 pm
Location: USA

Things to try for ST3 Not Responding

Postby texmurphy » Fri Apr 06, 2012 5:00 pm

I am running a Quadcore i7-930 with minor overclocking on Asus P6X58DE with 6GB of RAM.
Win7 with .NET will only utilize a working set of under 2GB.

I use a logbook with 3+ years of complete data and then a full-Compressed logbook with 6+ years of compressed data.
My usual routine is to import to the 3+ logbook (usually Garmin .fit) and do any edits needed there.
Then I Save the 3+ logboook and then Export a fitlog of the imported activities.
I will close ST and restart ST on the full-Compressed logbook, do a fitlog import, activity compress, and then save.

This first image is an example of memory useage with import/compress/save(successful). See how the working set tops out at 1627472KB and this with successful save :o

Image

The image below is slightly different. Before import I have done an edit in Equipment view to the notes field of an existing piece of equipment (adding 33 text characters). Then I try the import/compress/save(fail). Notice the smaller working set at System OutOfMemory fault is 1491972KB and the Save is unsuccessful :cry:

Image
texmurphy
Donated!
Donated!
 
Posts: 2186
Joined: Wed Jul 05, 2006 7:38 pm
Location: Maryland, USA

Re: OutOfMemory exception when saving logbook

Postby texmurphy » Sun Apr 08, 2012 1:30 pm

Ironically, the above operation was the last time I have been able to do a Logbook->Save(Success).
All subsequent attempts to add to that logbook have met with an OOM fault.
texmurphy
Donated!
Donated!
 
Posts: 2186
Joined: Wed Jul 05, 2006 7:38 pm
Location: Maryland, USA

System.OutOfMemoryException

Postby hmspain » Thu May 02, 2013 12:56 pm

For the past couple of weeks, I can't seem to import any of my workouts. I have a Garmin 800 and just recently upgraded to the latest SportTracks (hoping for the best).

When I import, the SW freezes and throws a System.OutOfMemoryException. I'm running on Windows 7 64-bit.

My logbook is kinda big 138,096 KB but I tried an earlier smaller copy, and it still throws the exception.

My DotNet is Framework 4.

Any suggestions?
hmspain
Donated!
Donated!
 
Posts: 32
Joined: Mon Jul 14, 2008 9:12 pm
Location: Los Alamitos, CA

Re: System.OutOfMemoryException

Postby hmspain » Thu May 02, 2013 12:59 pm

The import failure got worse and worse over the past 4 weeks or so.

Another item; SportTracks no longer stores a BACKUP copy of the log file (symptom?).

I could import and save after 3-5 tries earlier. Now, I can't get it to import without "SportTracks has stopped working" much less save.
hmspain
Donated!
Donated!
 
Posts: 32
Joined: Mon Jul 14, 2008 9:12 pm
Location: Los Alamitos, CA

Re: OutOfMemory exception when saving logbook

Postby hmspain » Fri May 03, 2013 1:35 pm

I did a little debugging and found the following;

DISABLING all my plugins allows SportTracks to run and NOT crash!
I still could not import since my Garmin files are not recognized without the Garmin plugin.
I re-enabled the GARMIN Plugin, saw support for my Garmin 800 (I used to browse for the files), and was able to import and save! :-)

I re-enabled all my plugins one by one (to see if I could make SportTracks unstable) until I enabled the SportTracks Map Provider for Google Maps...

If SportTracks Map Provider for Google Maps is enabled, SportTracks will crash. It takes a while, which leads to confusion about WHAT is causing the crash. Depending on what you are doing, you blame that thing. Just leave SportTracks alone, after the memory ramps up... it will crash (or not) depending on the SportTracks Map Provider for Google Maps plugin check box.

If you watch the memory allocation, with SportTracks Map Provider for Google Maps enabled, memory will ramp up (in my case to 1.4GB roughly) and crash.

If you disable SportTracks Map Provider for Google Maps, memory allocation ramps up to about 1.5GB and levels off... no crash.
hmspain
Donated!
Donated!
 
Posts: 32
Joined: Mon Jul 14, 2008 9:12 pm
Location: Los Alamitos, CA

Next

Return to Bugs

Who is online

Users browsing this forum: No registered users and 1 guest

cron