I’ve been tasked with looking into what storage space we’re currently using in Liberty Create and how we can manage it in future. I’ve got a list of questions that I’m hoping someone can help with.
Does the storage for backups come from the total LC space allocated? I assumed it did, but I deleted a backup and it didn’t make a difference to the space the Controller reports as being used.
The Controller and the Disk Usage page on the Monitor Studio don’t agree, even on the amount of storage allocated. The Controller says 196.3 GB, but the Monitor Studio says 210.78 GB. 14GB is quite a big difference.
In the Disk Usage page the breakdown of the database object space usage doesn’t come close to the reported database size. In our build env, the db usage is reported as 7.04 GB. Adding up the storage of all the objects in the db gives 153 MB. I’m guessing all the rest is widgets, code, theme packs etc. Is there any way to break down the space used by that stuff? Or is that part of the App Data reported?
I’ll be grateful for any help on this. Also, if anyone is willing to share how they manage their storage needs with LC that would be great.
We have been going through this exercise too. One thing that has become apparent is that for file and message objects, field history and deleted records can account for a lot of disk usage (we off-site docs to Azure, but it turns out each time we export or import a doc, it gets retained in the history). We have saved a fair bit of space by only storing a short amount of history, and making sure deleted records don’t hang around for too long in the retention page of each object. Also we have noticed that some space doesn’t get freed up immediately, I think there are daily and weekly schedules that delete records, snapshots, backups etc.
I’ll keep watching this thread, as I’m interested to learn more about the mechanics of retention and disposal…
For 1.
No, the Controller backups do not take up the diskspace you see allocated in the Controller.
And @kevin.rowe is correct in that some deleted items recover space immediately, while others are retained on the disk until a daily or weekly purge script runs.
For example, if you delete an environment or application, it is retained on the disk until purged - usually every Sunday - in case it was deleted by accident. And only then will you see the space being recovered.
and 3. -
It sounds like someone our side needs to take a look at your specific situation, I would suggest you raise a Support Ticket for this via the Community > Support.
A couple of things that I tend to recommend - and Kevin is spot on about retention settings:
Ensure you have set appropriate retention settings in Build Studio > Data Store > Data Settings, Build Studio > System Variables and each of your objects.
Different object types can have different retention settings. Ie. Message objects also have a setting for retention of EML files.
Imports and Exports can also quickly fill up disk space, even though you might not need the files again once imported / exported - the retention settings for those are defined in Build Studio > Data Store > Data Settings.
(Also check that the setting “Retain deleted records in database for” is set appropriately. This setting means that even though you delete the record from the app itself, it is still retained on the disk in case you need it to be recovered.)
Ensure you do not keep more snapshots than needed, as these do take up storage space.
The document being retained in the history is very interesting. We’re not uploading or exporting anything yet, but we’ve a couple of things that we’re working on were we will be.
I’ll log a support ticket about the monitor studio/controller not agreeing.
There are a few considerations when passing documents/data out to external storage that I’ve learned, two spring to mind immediately:
Retention settings on the original data in Create, as you’ve already discussed. Whether documents or record data, if the purpose is to reduce storage on Create side, then it’s likely you’ll want to remove the original. This could be done via rules if it’s about partial deletion, or object settings if the whole record can be scrubbed.
In at least one scenario I’ve encountered, we had to keep meta-data for documents in Create so that we could retrieve specific documents from a long-term filestore, so we only wanted to delete the file attribute and not the whole record. This makes it tricky to remove the whole file, and in fact we’ve got a minor issue with still retaining previews/thumbnails (I’ve raised an Idea to allow us to control this, https://ideas.netcall.com/i/ideas/p/feature/vote/idea/view?context_record_id=643847)
Retention settings on the API. Particularly if the data is being transferred by base64, the payload is now larger than the original file was, so if it isn’t removed you haven’t actually saved any data yet, even if the original file has been removed!
If the transfer was successful, then removing the payload (or the whole record) as soon as sensible can help reduce the space here. If unsuccessful, then obviously you might need to keep the details for review.
Generally speaking, the biggest use of data are files and messages, so anything you can do to only retain what is sensible will help keep your storage needs from getting excessive.
We have a similar problem and do have some concerns over disk storage.
To put into context we are moving from a Dynamics platform (which has 7 years of data across a number of services and processes) and consumes no more than 100GB. We have migrated around 10% of functionality, data objects, data into Netcall and are consuming 490GB. We use Amazon for other solutions/data storage also and no issues, so there is clearly something amiss with the Netcall platform here. I know Netcall will absolutely look to help us but the thought of paying big bucks for terabytes of storage (based on my simple maths above) ain’t going to fly with anyone.
