[ST3] [FIXED] ST3 OutOfMemory exception when saving logbook

On the rare occasion you might find a bug...

Re: OutOfMemory exception when saving logbook

Postby the5krunner » Thu Jun 06, 2013 10:21 am

just saw this bug after posting about a memory related bug that causes a different error for me.

my workaround on the other bug as to close garmin ant agent.

try that and see if it makes any difference...probably not , I know. just trying to offer possible help.
----------------------------
tfk
http://the5krunner.com
the5krunner
 
Posts: 458
Joined: Sun Feb 22, 2009 1:41 pm

Re: OutOfMemory exception when saving logbook

Postby texmurphy » Thu Jun 06, 2013 2:56 pm

OOM on Save is a function of ST using .NET which has a large but limited memory address space.
ANT agent is a separate issue for the5krunner. Besides with an Edge 800 you should not be using the ANT Agent.
texmurphy
Donated!
Donated!
 
Posts: 2126
Joined: Wed Jul 05, 2006 7:38 pm
Location: Maryland, USA

Re: OutOfMemory exception when saving logbook

Postby PSJoyce » Fri Sep 06, 2013 5:55 pm

Is there any solution to this? SportTracks crashes every few seconds with the 'System.OutOfMemoryException' error.
PSJoyce
Donated!
Donated!
 
Posts: 8
Joined: Mon Sep 21, 2009 7:19 am
Location: Cape Cod, MA

Re: OutOfMemory exception when saving logbook

Postby hmspain » Fri Sep 06, 2013 8:39 pm

texmurphy wrote:OOM on Save is a function of ST using .NET which has a large but limited memory address space.
ANT agent is a separate issue for the5krunner. Besides with an Edge 800 you should not be using the ANT Agent.


Hmmm. So our only hope is for ST to abandon .net (not likely) or have a future release of .net with more memory address space?

Does anyone have any insight as to either of these possibilities?

BTW, ST support is working on my logbook (splitting it into two). Not the best solution, but I'm VERY grateful they are willing to do this for me!
hmspain
Donated!
Donated!
 
Posts: 27
Joined: Mon Jul 14, 2008 9:12 pm
Location: Los Alamitos, CA

Re: OutOfMemory exception when saving logbook

Postby texmurphy » Sat Sep 07, 2013 8:53 am

PSJoyce wrote:Is there any solution to this? SportTracks crashes every few seconds with the 'System.OutOfMemoryException' error.

Unless your logbook is very large, your OOM error is likely to be caused by a bad plugin or corrupted activity.
  • In Daily Activity, Set your ROUTE to NONE;
  • Go to Settings->Plugins and untick all the non-core ST3 plugins;
  • Close ST and then restart ST.
If you are no longer getting OOM's, then one at a time re-enable your plugins. You must close and restart ST for each re-enable. This should allow you to find any bad plugin install (if any).

To check for a corrupted activity, try and recall when the OOM's started. Be sure to archive your current logbook. Export as fitlog all recent (from OOM) activities and then Remove those activities. Close and restart ST. If no OOM's, then gradually add back the activities until any failure. You may need to exit and restart ST at each activity.

How large is very large for a logbook? Go to Logbook->Properties and if it is larger than mine then it might be too large:
Activities: 818
Equipment: 124
GPS Data Points: 5239166
Data Track Measurements: 13253478
texmurphy
Donated!
Donated!
 
Posts: 2126
Joined: Wed Jul 05, 2006 7:38 pm
Location: Maryland, USA

Re: OutOfMemory exception when saving logbook

Postby PSJoyce » Sat Sep 07, 2013 9:14 am

Thanks for the help! My database goes back to 2009. I have already disabled all nonessential plugins. The file is large – I am unable to go back and delete past years because ST crashes too quickly. I have requested the experimental data compression plugin, but not heard back. At this point, the only option to keep using ST that I see is to create an entirely new logbook and lose all past records. I’m hoping there is a better way.

Thanks again

Activities: 1,569
Routes: 3
Equipment: 64
Athlete History Entry: 261
GPS data points: 6,073,507
Data Track Measurements: 25,996,404
PSJoyce
Donated!
Donated!
 
Posts: 8
Joined: Mon Sep 21, 2009 7:19 am
Location: Cape Cod, MA

Re: OutOfMemory exception when saving logbook

Postby texmurphy » Sat Sep 07, 2013 12:49 pm

PSJoyce wrote:...The file is large – I am unable to go back and delete past years because ST crashes too quickly...

I have mentioned in other posts (Search) how I delete activities to split a logbook. But to repeat here:
1. Make copy of your logbook (or several). Rename the Logbook copy which is to be the original to something like PSJoyce's History Complete to Aug2013. Rename the Logbook copy which is to be the current to something like PSJoyce's History Current
2. Use only Daily Activity to remove activities (with map=NONE). You can only remove a few activities at a time, since both the Remove Activity and the Save process consume a lot of memory. Determine a split point in time. I suggest 31 December 2011 to give your newer activities logbook some trend data over time.
3. Decide what to do about Equipment History. This is a significant problem as Equipment screens consume lots of Memory (almost as much as Reports). If you can get to Equipment, then note which currently active Items were in use BEFORE the split date (e.g. 31 Dec 2011 and prior). The Equipment History chart is helpful here. Record the miles/hours used up to the split date. You will want to fill in the "Extra mi used:" item field with the miles used up to the split date after split is complete. You can only use miles, not hours, in this field (I have posted this as an Enhancements request) :!: ).
4. Begin the activity Removal process using only Daily Activity and a fresh start of ST:
Use the Calendar to get to the oldest activity month (Calendar often targets the wrong month/date so be careful (I have posted this as an Enhancements request) :!: ). Try and select only a day without any activity as you want to keep memory use at a minimum. Click once to select the "This Week" of the oldest activity in time. Now do an X Remove Activity which will remove that entire week. At first do not try to do more than a week at a time. Now do a Logbook->Save and then quit ST. Do NOT do an Exit with Save as that uses a little more memory :lol:
Restart ST and repeat finding the oldest week with activities, click on "This Week" to select the week and remove those activities, followed by a Logbook save and then ST exit.
5. Eventually you will have cleared enough activities that you can delete a month at a time by selecting the month line from Daily Activity. Do not try and do this using the Reports screens. Reports is a big memory user. ST opens with Daily Activity, so stay there. You can try doing 2 months at a time by shift-click select the current and the prior/next month lines.

Be careful and keep backups. Check you Logbook closely when done to see that all is ok. A pain to do and it may take more than one attempt. Until the process is automated, this is the best proceedure I have found. I now keep only about 2.5 to 3 years per logbook before they are full, even using admin's Data Compression Plugin. Request the plugin again from admin, it will make things a little easier.
texmurphy
Donated!
Donated!
 
Posts: 2126
Joined: Wed Jul 05, 2006 7:38 pm
Location: Maryland, USA

Re: OutOfMemory exception when saving logbook

Postby jibberjim » Sat Sep 07, 2013 2:07 pm

So we have very large logbooks, which regularly OOM, so what I did was write a windows script host file, which deleted just the GPSRoute and Cadence Data from every item before, or after a year, this means that the size is smaller and the file is manageable, but you still have the basic info about the activities and TrainingLoad has all the data for that.

Copy and paste the text below from and including <job> to </job> into a new file called "split.wsf", but a copy of your logbook named test.logbook3 in the same folder. Then run the file at a command prompt with "cscript split.wsf" It will then create two more logbooks, one from before 2011, and one with everything after. But they both have all the activities in, just not all the data about all of them.

<job>
<script language=jscript>

// Enter the name of your log book here:
var LOGBOOK = "test.logbook3"
// Enter the year you want to split here:
var splitYear = 2011;




var output = new ActiveXObject("Msxml2.DomDocument");
output.async=false;
output.load(LOGBOOK);
var activity=output.selectNodes("/Logbook/Activities/Activity");

var outArray=[];
for (var i=0;i<activity.length;i++) {
var item=activity.item(i);
var node=item.selectSingleNode("GPSRoute");
if (node) {
var date=item.getAttribute("startTime");
outArray.push({len:node.text.length,date:date});
var d= new Date(date.substring(0,date.indexOf("T")).replace(/-/gim,"/"))
if (d.getFullYear()<splitYear) {
removeNodeByName(item,"GPSRoute");
removeNodeByName(item,"CadenceTrack");
}
}
}


function removeNodeByName(item,name) {
var node=item.selectSingleNode(name);
if (node) {
item.removeChild(node);
}
}
output.save("Before "+splitYear+" "+LOGBOOK);


output.load(LOGBOOK);
var activity=output.selectNodes("/Logbook/Activities/Activity");

var outArray=[];
for (var i=0;i<activity.length;i++) {
var item=activity.item(i);
var node=item.selectSingleNode("GPSRoute");
if (node) {
var date=item.getAttribute("startTime");
outArray.push({len:node.text.length,date:date});
var d= new Date(date.substring(0,date.indexOf("T")).replace(/-/gim,"/"))
if (d.getFullYear()>=splitYear) {
removeNodeByName(item,"GPSRoute");
removeNodeByName(item,"CadenceTrack");
}
}
}


output.save("After "+splitYear+" "+LOGBOOK);

</script>
</job>
jibberjim
Donated!
Donated!
 
Posts: 59
Joined: Thu Dec 18, 2008 2:15 pm
Location: Kingston-Upon-Thames

Re: OutOfMemory exception when saving logbook

Postby PSJoyce » Sat Sep 07, 2013 2:43 pm

Thanks! I had searched and found the year by year deletion procedure. This activity by activity procedure did allow me to eventually delete two years of data. Crashing has subsided now, looks like this needs to be a yearly or so task.
PSJoyce
Donated!
Donated!
 
Posts: 8
Joined: Mon Sep 21, 2009 7:19 am
Location: Cape Cod, MA

Re: OutOfMemory exception when saving logbook

Postby hmspain » Sat Sep 07, 2013 5:39 pm

jibberjim wrote:So we have very large logbooks, which regularly OOM, so what I did was write a windows script host file, which deleted just the GPSRoute and Cadence Data from every item before, or after a year, this means that the size is smaller and the file is manageable, but you still have the basic info about the activities and TrainingLoad has all the data for that.

Copy and paste the text below from and including <job> to </job> into a new file called "split.wsf", but a copy of your logbook named test.logbook3 in the same folder. Then run the file at a command prompt with "cscript split.wsf" It will then create two more logbooks, one from before 2011, and one with everything after. But they both have all the activities in, just not all the data about all of them.


Thank you for a simple and elegant workaround to the OOM problem!!! I suspect ST support is doing this very thing to my logbook :-).
hmspain
Donated!
Donated!
 
Posts: 27
Joined: Mon Jul 14, 2008 9:12 pm
Location: Los Alamitos, CA

Re: OutOfMemory exception when saving logbook

Postby Svante84 » Tue Dec 10, 2013 12:03 pm

What is the long term strategy for solving this problem? I have training data from 2006 (the first years only with HR data, the last years with lots of data including GPS, power++), and the last couple of weeks I have ended up with disabling almost all plugins to keep it from crashing (Google map plugin seems to be the one making most of the problems).

Of course I can compress some of my old data. But if this means deleting some data like cadence etc or convert from 1s to 5s recording, I don't feel good about it, as I would like to keep all the data that is recorded. And even if i comress it, what happens when I have 10 or maybe 30 years of traning data? I would assume 30 years of data would make ST crash, even if all data is compressed. I would like to have the oppurtunity to look back at my old training diary when I get old, like my dad is looking at his 30 year old training diary.

I would like to have a clear answer to this, as I am beginning to see no other option than converting to another training software. And as far as I can see, it is just a matter of time until this will be a problem for ALL users, as logbooks will grow day by day (and probably the amount of data for every time unit will grow with more equipment for data recordings in the future). I really hope I am wrong, as I like using ST and I really don't like thinking about the amount of work converting to another traning software..
Svante84
 
Posts: 4
Joined: Fri Aug 02, 2013 2:09 am

Re: OutOfMemory exception when saving logbook

Postby Tisztul_A_Visztula » Tue Dec 10, 2013 12:40 pm

+1
Cheers,
Zoltan
Tisztul_A_Visztula
 
Posts: 525
Joined: Thu Jul 15, 2010 8:38 am
Location: Hungary

Re: OutOfMemory exception when saving logbook

Postby texmurphy » Tue Dec 10, 2013 12:56 pm

Svante84 wrote:...Of course I can compress some of my old data. But if this means deleting some data like cadence etc or convert from 1s to 5s recording, I don't feel good about it, as I would like to keep all the data that is recorded..

The Compress Data Plugin will not change recording frequency nor granularity of the data points. Rather it will remove identical successive values. E.g. You have a heart rate 1 second recording of ..., 135, 136, 136, 136, 136, 137, ... The plugin will remove the middle two 136 values from the datatrack.

The Plugin will allow for about a 30% reduction in data points in my experience with cycling with power and 1 second recording. I can keep about 3 years of data per logbook :cry: at about 12500 km per year.
texmurphy
Donated!
Donated!
 
Posts: 2126
Joined: Wed Jul 05, 2006 7:38 pm
Location: Maryland, USA

Re: OutOfMemory exception when saving logbook

Postby Tisztul_A_Visztula » Tue Dec 10, 2013 1:53 pm

The real question was the inquiry about the targeted LT strategy.
Cheers,
Zoltan
Tisztul_A_Visztula
 
Posts: 525
Joined: Thu Jul 15, 2010 8:38 am
Location: Hungary

Re: OutOfMemory exception when saving logbook

Postby PSJoyce » Tue Dec 10, 2013 2:34 pm

+1
PSJoyce
Donated!
Donated!
 
Posts: 8
Joined: Mon Sep 21, 2009 7:19 am
Location: Cape Cod, MA

Re: OutOfMemory exception when saving logbook

Postby texmurphy » Tue Dec 10, 2013 4:26 pm

Tisztul_A_Visztula wrote:The real question was the inquiry about the targeted LT strategy.

Agreed!
texmurphy
Donated!
Donated!
 
Posts: 2126
Joined: Wed Jul 05, 2006 7:38 pm
Location: Maryland, USA

Re: OutOfMemory exception when saving logbook

Postby Svante84 » Wed Dec 11, 2013 3:55 am

texmurphy wrote:
Svante84 wrote:...Of course I can compress some of my old data. But if this means deleting some data like cadence etc or convert from 1s to 5s recording, I don't feel good about it, as I would like to keep all the data that is recorded..

The Compress Data Plugin will not change recording frequency nor granularity of the data points. Rather it will remove identical successive values. E.g. You have a heart rate 1 second recording of ..., 135, 136, 136, 136, 136, 137, ... The plugin will remove the middle two 136 values from the datatrack.

The Plugin will allow for about a 30% reduction in data points in my experience with cycling with power and 1 second recording. I can keep about 3 years of data per logbook :cry: at about 12500 km per year.


OK, so this type of compression is not actually loss of data but it still just is delaying the problem.

From 2006 until april 2011 i have only data from a Polar watch (HR and elevation I think), and the last 2,5 years of my training diary is with GPS, power etc. I do about 600 hrs of traning each year. My logbook file is 135 MB. As mentioned this makes ST crash for me if I don't disable (almost) all plugins.
Svante84
 
Posts: 4
Joined: Fri Aug 02, 2013 2:09 am

Re: OutOfMemory exception when saving logbook

Postby texmurphy » Wed Dec 11, 2013 9:35 am

Svante84 wrote:...OK, so this type of compression is not actually loss of data but it still just is delaying the problem...

Make a copy of your Logbook to a new name. ST will allow this without license issues. Open the new Logbook. Using Daily Activity view, select a month at a time and do Edit->Compress Data using the Data Compression Plugin. Save your Logbook after one or two compresses and exit St then restart ST to be sure you have enough memory headspace. Note the Logbook will not look smaller on disk, but Logbook->Properties in ST will reveal a decrease in datapoints. When you have fully shrunk this new Logbook, test it out for completeness (e.g. summary stats in Reports, shape of Training Plugin summary graphs, etc.).

Use normal safe data precautions (backup, etc.). You can likely get another year of use from your Logbook. I am not using Mobi, and I do not know what will happen with Mobi synchs.
texmurphy
Donated!
Donated!
 
Posts: 2126
Joined: Wed Jul 05, 2006 7:38 pm
Location: Maryland, USA

Re: OutOfMemory exception when saving logbook

Postby Svante84 » Wed Dec 11, 2013 10:05 am

texmurphy wrote: You can likely get another year of use from your Logbook. I am not using Mobi, and I do not know what will happen with Mobi synchs.


I understand that I will be able use ST for another year or two if I compress data. But hopefully I will be traning for the rest of my life, and I want to be able to have a complete training diary for all years in one software.

One option is to extend the period I can use ST by compressing data, and hoping that the problem is solved before my (compressed) logbook will be to large. But if ST have no definite plan of how to fix this, I think I would rather convert to another training sowtware right now, in order to minimize the amount of training data to be converted to another system (even though there are many export/import/convert tools, this is always a time demanding job)

I appreciate that you are trying to help me solving this problem, but all solutions presented in this thread seems just to delay the entire problem. And I don't think I'm comfortable with this approach.
Svante84
 
Posts: 4
Joined: Fri Aug 02, 2013 2:09 am

Re: OutOfMemory exception when saving logbook

Postby smaryka » Wed Dec 11, 2013 10:14 am

Admin needs to come along and answer the question about long term storage. Otherwise your options are create new logbooks, compress as above, or switch to something else.

Agree that it's a problem though, I have gone with keeping the very basics of my previous years' data in my current logbook (all the details are in an old "retired" logbook) which currently has about 2 years' current stuff in it. However I confess that since Strava came along, I use ST less and less for the details and use Strava more for analysis. However that doesn't change the fact that I would love to have all details of all my years' worth of data in the same logbook.
smaryka
Donated!
Donated!
 
Posts: 69
Joined: Wed Jan 07, 2009 11:38 am
Location: United Kingdom

Re: OutOfMemory exception when saving logbook

Postby admin » Wed Dec 11, 2013 1:22 pm

If you are using 1-second recording with a power meter and doing 10,000s of kilometers per year, you will hit a memory limit on your logbook size in about 3 years. This completely depends on your recording sampling frequency, the number of sensors, and the numbers of hours you ride. A "normal" runner will likely never see this problem - they simply don't generate the data sizes that serious cyclists do.

To work around the issue, you will have to archive the older workouts by making a copy of your logbook, then deleting older workouts from the current logbook.

To increase the number of workouts you can store in your "current" logbook, use the compression plugin here:

http://www.zonefivesoftware.com/sporttr ... press-data

Admin needs to come along and answer the question about long term storage.


Our latest product SportTracks.mobi has none of these limitations. No limitation on size of a single workout, number of workouts, or sensor sampling rate. Because it's a web app, we have no limitation on data size of your logbook at all. If we need, we just buy more disk space. We've also taken a fundamentally different approach to data management which means the app won't slow down as you add more data. And, we can easily add more web servers as number of users increases, or size of data tracked. Here is your "long term storage" answer today.

The answer regarding the ST3 desktop app is that this is a fundamental limitation we can't easily change without breaking lots of plugins. So for ST3, this limit will remain, and we will have to wait until ST4 development comes. At this point we will take what we've learned in ST.mobi, and use similar approaches to managing really big data logbooks so there are no limitations in the PC app.

To answer your question before you ask it - we don't even have a timeline for ST4 yet. It's "on the horizon", that's about it. I can't project what the tech landscape will be in 30 years... I doubt you'll still be running Windows 7, so I doubt you'll be running ST3 then.
Get the latest info: Fan us on facebook or follow us on twitter
admin
Site Admin
Site Admin
 
Posts: 3013
Joined: Tue Apr 05, 2005 8:52 pm
Location: USA

Re: OutOfMemory exception when saving logbook

Postby Svante84 » Sun Dec 15, 2013 5:21 am

admin wrote:Our latest product SportTracks.mobi has none of these limitations. ... Here is your "long term storage" answer today.


So now I have compressed all my traning data, and have lots of logbook copies for different years, different years compressed etc. So if I can live with different logbooks for different years when using ST, how is it possible to have all data available in SportTracks.mobi? As far as I can see, all data is syncronized between ST and mobi and it is no possibilty to control the sync peridod. If I for example delete training data from 2006-2010 in ST, it is also deleted in mobi..
Svante84
 
Posts: 4
Joined: Fri Aug 02, 2013 2:09 am

Re: OutOfMemory exception when saving logbook

Postby Tisztul_A_Visztula » Sun Dec 15, 2013 11:01 am

admin wrote:The answer regarding the ST3 desktop app is that this is a fundamental limitation we can't easily change without breaking lots of plugins. So for ST3, this limit will remain, and we will have to wait until ST4 development comes. At this point we will take what we've learned in ST.mobi, and use similar approaches to managing really big data logbooks so there are no limitations in the PC app.

To answer your question before you ask it - we don't even have a timeline for ST4 yet. It's "on the horizon", that's about it. I can't project what the tech landscape will be in 30 years... I doubt you'll still be running Windows 7, so I doubt you'll be running ST3 then.


It's good to know that ST4 exists in your plan at least. There are some people who are not willing to upload all their personal data to the web.

It is just my business and not of the others when I bonked mid-ride. :lol:
Cheers,
Zoltan
Tisztul_A_Visztula
 
Posts: 525
Joined: Thu Jul 15, 2010 8:38 am
Location: Hungary

Re: OutOfMemory exception when saving logbook

Postby texmurphy » Sun Dec 15, 2013 11:20 am

Svante84 wrote:
admin wrote:Our latest product SportTracks.mobi has none of these limitations. ... Here is your "long term storage" answer today.

So now I have compressed all my training data, and have lots of logbook copies for different years, different years compressed etc. So if I can live with different logbooks for different years when using ST, how is it possible to have all data available in SportTracks.mobi? As far as I can see, all data is synchronized between ST and mobi and it is no possibility to control the sync period. If I for example delete training data from 2006-2010 in ST, it is also deleted in mobi..
(emphasis and color in quote added)

@admin - Please answer this question. Since I am not on Mobi, I have no idea what will happen in the synch process. As a high data user of ST3 (and I am about to do another Logbook split), it is hard for me to offer advice about the Out of Memory issue with ST3.
texmurphy
Donated!
Donated!
 
Posts: 2126
Joined: Wed Jul 05, 2006 7:38 pm
Location: Maryland, USA

just got this error-any ideas

Postby paul.d » Tue Dec 31, 2013 12:23 pm

paul.d
 
Posts: 23
Joined: Fri Mar 12, 2010 2:31 am

PreviousNext

Return to Bugs

Who is online

Users browsing this forum: No registered users and 1 guest