Remove entry and sense "Import Residue" fields from models and UI

[myieye]

So, I went to expand the tests to cover the missing entryImportResidue field and discovered that it was intentionally removed🤔.

So...I guess it's a mistake that we show it in the UI.

senseImportResidue on the other hand was never removed from our models, but is ignored when decoding lift files (as is the entryImportResidue). So, I'm guessing we want to just remove both from our models and the UI.

Originally posted by @myieye in #1718 (comment)

[myieye]

From the FLEx docs:

TL;DR: import residue (IR i.e. unmapped import data) isn't supposed to live long-term and there are tools in FLEx for migrating the IR to the most appropriate fields. So, that's supposed to be handled in FLEx.

LF projects that don't make use of FLEx also don't have IR, because we've been discarding it from LIFT imports for a long time.

Currently:

• Entry IR can't exist in LF, because we removed it from our models a long time ago (see above) (it could still exist in the database, but will never be read)
• You can't sync sense IR from FLEx to LF, because...it just doesn't work 🤷 - it doesn't arrive in LF.
• You can sync sense import residue from LF to FLEx, but that doesn't make sense - if it's coming from LF then it wasn't imported, it was entered manually

These fields hardly/don't work in LF and there doesn't seem to be a good reason for them to exist. So, it makes sense to me to remove them.

[myieye]

Based on recent production data, there are:

• 0 usages of entryImportResidue
• a hand-full of usages of senseImportResidue. The only value that could potentially have real meaning is "Dal17":

[
  {
    "project": "5e1f32c550b51945ff768fa3",
    "entryId": "5e1f32f3bea602641cd56b10",
    "value": {
      "en": {
        "value": "Dal17"
      }
    }
  },
  {
    "project": "5e1f32c550b51945ff768fa3",
    "entryId": "5e1f32f9bea602641cd57f64",
    "value": {
      "en": {
        "value": "Dal17"
      }
    }
  },
  {
    "project": "5f440787d28a39460673fc79",
    "entryId": "5f4407c7d28a39460673fc7e",
    "value": {
      "en": {
        "value": "field Meaning 1 Import Residue"
      }
    }
  },
  {
    "project": "61b155d57dfdc546ab4939c3",
    "entryId": "61b156419c8c8847e2e0af00",
    "value": {
      "qxh": {
        "value": "test"
      }
    }
  },
  {
    "project": "58e3bb9c8b2147751a47952e",
    "entryId": "58e3bced8b21476bb305b97f",
    "value": {
      "en": {
        "value": "Import Residue<br/>"
      }
    }
  },
  {
    "project": "58f866bb8b21476bb305b9f2",
    "entryId": "58f867cb8b214751bb514922",
    "value": {
      "en": {
        "value": "asxdhkm wg g asdgf asdf asdf<br/>"
      }
    }
  }
]

[myieye]

But, how do we remove fields that are persisted in (offline) project configs? 😆

• Dirty approach: black-list them somehow.
• Clean: MongoDb migration and let offline data sync in time

This simple script could make for a rudimentary tool for estimating the damage of a migration.

[megahirt]

But, how do we remove fields that are persisted in (offline) project configs? 😆

While there is plenty of "offline" code in our AngularJS client, the config is not persisted in localstorage and our offline feature was never finished - so it's not functional.

[alex-larkin]

After reading this thread it's not clear to me what has been changed or exactly which Import Residue fields are affected. @myieye, could you please confirm that the current state of Import Residue fields is consistent with the documentation at https://github.com/sillsdev/web-languageforge/wiki/Known-Issues-and-Limitations ?

Also, has there (or will there be) a change in the data viewable by users at languageforge.org? And this doesn't affect the underlying database or what is viewable in FLEx, right?

Sorry for all the questions. Thanks so much!

[megahirt]

could you please confirm that the current state of Import Residue fields is consistent with the documentation
@alex-larkin thanks for that. I have added Import Residue to the sense list (it was already in the entry list) on the Known Issues and limitations. I think that is consistent.

Also, has there (or will there be) a change in the data viewable by users at languageforge.org?
@alex-larkin good question. @myieye has determined that in our entire data set of all projects, there isn't really any import residue data, so removing the UI code that shows this field has no user effect.

This change is just cleans up a field that should have never been showing in the first place.

@myieye , can we close the issue? I note that it's still open but want you to close it.