fxcm_v22

• Reverse dictionary transform. We load the dictionary when it is found after decompressing it. Text has a separate buffer from coded byte stream buffer.
• Natural language processing using stemmer (from paq8px(d)).
• Stemmer has new word types: Article, Conjunction, Adposition, ConjunctiveAdverb.
• Some word (related) contexts are changed based on what type of word was last. Some words are removed from word streams depending on the last word type. This improves compression.

There are four word streams:

• 1. basic stream of undecoded words.
• 2. decoded word stream after stemming for sentences. Contains all words. Reset when sentence ends.
• 3. decoded word stream after stemming for paragraphs. Contains words that are not: Conjunction, Article, Male, Female, ConjunctiveAdverb. Reset when paragraph ends.
• 4. decoded word stream after stemming. Contains words that are not: Conjunction, Article, Male, Female, Adposition, AdverbOfManner, ConjunctiveAdverb.
• Word limit per stream is increased from 64 to 256 words.
• New context that uses stemmer and decoded plaintext. Some global context are changed when word type is:
  ConjunctiveAdverb or Conjunction - skip updating in stream 1. Conjunction for sentence reset. etc. Knowing these new words allowed large amounts of compression improvements.
• In some cases words are removed between certain chars from stream 2 and 3 when following is true:
  =| - wiki template
  <> - html/xml tags
  [| - wiki links
  () - usually words in sentences
• Main predictors are split between three different ContextMaps. This provides better compression. Sizes for hash tables are 32, 64 (standard for paq8 versions) or 128 bytes per contexts. 32 byte size is good for small memory context (below 256 KB), 64 is good for medium sized context (up-to 16MB), 128 is good for large memory context (more than 16MB).
• One state table is removed and replaced with another one. State tables are generated at runtime to reduce code size.
• Added sparse match model. With a gap of 1-2 bytes and minimum length of 3-6 bytes. Mostly for escaped UTF8.
• Detection of math, pre, nowiki, text tags in decoded text stream. Some word related contexts are not used when 3 first tags content is compressed. Improves compression speed.
• More parsing of lists and paragraphs. So that context for predictors is best as they can.
• Optimized context skipping in main predictors.
• Main predictor context bias is not forwarded to cmix floating point mixers, instead a single prediction bias is set. This avoids unnecessary expansion of mixer weight space and maintains lower memory/cpu usage. There are some other predictions that are not forwarded as they make compression worse and slower.
• Some mixers and APM’s context size is larger so that prediction can be better.
• partially/fully decoded word index into dictionary is used as a context for mixer in fxcm mixer
• Some variables are renamed for better readability.

fxcm_v17-18

• Some cleanup
• Adjust contexts

fxcm_v15-16

• Add one new context
• Adjust on mixer
• Some cleanup
• Add comments so reader can understand what is happening
• Tune some variables and contexts
• Improve compression speed

fxcm_v13-14

• Word&Sentence context.
• Adjust some contexts
• Update "for CMIX" to most recent version

fxcm_v11-12

• Wiki table/row & column context
• Add 3 new context based on table
• Change some context

fxcm_v9-10

• Change some contexts
• Increase one mixer size

fxcm_v7-8

Page

One article as seen by first char context

fxcm_v5-6

-Add another context based on first char
-Adjust some contexts

fxcm_v3-4

• Remove one mixer, add one mixer
• Changes to some contexts

fxcm_v1-2

Public version