Management of the DB

[permcody]

The Production CIVET database at the INL is growing and growing in size. Some of what is in there are considered records so we can't just delete it. However, we also can't just let it grow unbounded. We need to work out a way for the server to delete large portions of the DB that aren't records. I believe the DB uses keys so data integrity will mostly handle itself. What we need to do is engineer a way for the server to selectively delete records and then work on what to delete. I know you need more details, but let's see if we can engineer how deletion would/could work (e.g. Probably a daily or weekly triggered task).

[brianmoose]

Django has "management" commands, see ci/management/commands for some examples. A couple of these are routinely called via cron. This is how civet recipes are loaded into the DB. It should be easy to add a new command to delete stuff, as long as you can precisely specify what you want to delete.

[socratesgorilla]

What exactly constitutes a record entry vs a non-record entry? Is there an easy way to identify this programatically?

[permcody]

Back when I wrote this, I believe I had a good grasp of what was a record versus what was not. I'm less sure now that we are nine months out from the original ticket. If I had to guess, I'd say any record linked to one of our releases and that's it!

However, we don't want to delete records from a few days ago either in case we need to go back in time a bit as part of our normal development cycle.

I would propose that we deleted records older than a month or two (of course this will be a variable that we can change), that are NOT part of a release. We could probably be generous and keep a whole year's worth of build results if we want. So what's a release? Well these are linked on our webpage so we'll have to think about how to formalize them in CIVET land so we don't accidentally dump anything. This will be critical! For now, let's skip that part and assume I've get a hash of what I'm calling a release (see the releases for MOOSE here: https://mooseframework.inl.gov/sqa/index.html). Can you come up with a way to not remove anything that's browsable from one of those links? That might be pretty difficult!