I'm looking to see if there are any tools that can "skinny" down historical data. We are now supporting some hosts that had overly high poll rates when they were commissioned, the historian subsequently got bloated and is now using up too much HDD space.

Are there any tools available that can go back and remove swaths of data, or perhaps replace a 10 min span of 1 second data with a single average or something like that?

Most of the data isn't needed as the deadband was also too small. We could set the historian purge value to 365 days or something and let the old data fall off, but that wouldn't be ideal. There are thousands or tens of thousands of tags in these systems, so a manual process would be a bit prohibitive as well.

Any suggestions are appreciated. Thanks.

I have no idea about editing the historian... however if I was caught short with this, maybe a custom report to pull the historical data - i.e. have it run a snapshot type to take a 10 min average - though I would limit the reporting period to something sensible, you could then configure it to re-run until you have all the info you need.
Then set the historian purge value to the 365 days... at least you have some of the data, although it won't be in the application.

Also you could take a copy of the application folder on the drive, and store it elsewhere, before doing the purge. If the data needs to be reviewed, you can create a new application using this directory... though you'd need a suitable PC and licence to do this too - you would also be missing any new data.

Interested to see the correct answer...

Hi Kurtis, The VTScada Historian was was designed to not allow deletions. Its high efficiency design makes it unnecessary to back off or archive historical data assuming deadbands are properly configured.

I assume that the poll rates and deadbands that bloated your HDD have now been corrected. Going forward the historian should now be growing at a normal rate. I would recommend simply increasing the size of the HDD once it approaches full or the next time you upgrade the computer(s).

Also Note: One clarification is that the poll rate generally has nothing to do with the size of the historian. It is the historian dead band value along with the maximum time between writes where, generally, the dead band is the most important factor regarding historian size.

Hope that helps.

A suggestion from another member of our support team.

@TomE suggests copying the application directory. Only the Data directory needs to be copied and this should not be done while the application is running. A conflict could occur with a write from VTScada happening while a history file is being copied. This results in a locked file and the data will no longer update. If it is a multi server system, stop the backup, copy the files, restart the backup. The backup will be from when the application was stopped.

After you fix your tag deadbands, the only way would be to migrate the data to a new tag.

You'd still lose alarm and event history, but you could query the data using the REST interface, develop tools to compress the data, map it into new tags with the CSV History importer.

It would be very nice to have this built in as it'd be nice to be able to poll data much faster (like 0.5 seconds instead of 10 seconds), but even with aggressive deadbands, large systems can easily grow well beyond a terabyte of data in a decade, which is just unacceptable for a VM or cloud system.

It'd also be nice to be able to move data to new tags as stations and systems often get redeveloped but the instrumentation stays the same. Luckily the same tools would still work, if one were to put the time in.