- Reverse dictionary transform. We load the dictionary when it is found after decompressing it. Text has a separate buffer from coded byte stream buffer.
- Natural language processing using stemmer (from paq8px(d)).
- Stemmer has new word types: Article, Conjunction, Adposition, ConjunctiveAdverb.
- Some word (related) contexts are changed based on what type of word was last. Some words are removed from word streams depending on the last word type. This improves compression.
There are four word streams:
-
- basic stream of undecoded words.
-
- decoded word stream after stemming for sentences. Contains all words. Reset when sentence ends.
-
- decoded word stream after stemming for paragraphs. Contains words that are not: Conjunction, Article, Male, Female, ConjunctiveAdverb. Reset when paragraph ends.
-
- decoded word stream after stemming. Contains words that are not: Conjunction, Article, Male, Female, Adposition, AdverbOfManner, ConjunctiveAdverb.
- Word limit per stream is increased from 64 to 256 words.
- New context that uses stemmer and decoded plaintext. Some global context are changed when word type is:
ConjunctiveAdverb or Conjunction - skip updating in stream 1. Conjunction for sentence reset. etc. Knowing these new words allowed large amounts of compression improvements. - In some cases words are removed between certain chars from stream 2 and 3 when following is true:
=| - wiki template
<> - html/xml tags
[| - wiki links
() - usually words in sentences - Main predictors are split between three different ContextMaps. This provides better compression. Sizes for hash tables are 32, 64 (standard for paq8 versions) or 128 bytes per contexts. 32 byte size is good for small memory context (below 256 KB), 64 is good for medium sized context (up-to 16MB), 128 is good for large memory context (more than 16MB).
- One state table is removed and replaced with another one. State tables are generated at runtime to reduce code size.
- Added sparse match model. With a gap of 1-2 bytes and minimum length of 3-6 bytes. Mostly for escaped UTF8.
- Detection of math, pre, nowiki, text tags in decoded text stream. Some word related contexts are not used when 3 first tags content is compressed. Improves compression speed.
- More parsing of lists and paragraphs. So that context for predictors is best as they can.
- Optimized context skipping in main predictors.
- Main predictor context bias is not forwarded to cmix floating point mixers, instead a single prediction bias is set. This avoids unnecessary expansion of mixer weight space and maintains lower memory/cpu usage. There are some other predictions that are not forwarded as they make compression worse and slower.
- Some mixers and APM’s context size is larger so that prediction can be better.
- partially/fully decoded word index into dictionary is used as a context for mixer in fxcm mixer
- Some variables are renamed for better readability.