Very new tokens may not get picked up by the bot / Redesign Event Mapping

[NIXBNT]

Bug Description

• Summary of the bug
  For the bot to be aware (i.e. capable of arbitraging) any pools in the ecosystem, a pool of interest must be present in one of the event_mappings files. The event mapping files allow events with the same name to be correctly mapped from their pool address to their exchange.
  This issue is most pronounced when a new token/pool(s) is added to the ecosystem and the bot is not aware.
  Typically this data is maintained by running the terraformer module and updating the static files - including the event_mappings. However the state of the market demands a more dynamic solution to support speculation on new tokens.

Severity (High/Medium/Low)

• Rate the severity. High: "I can't/won't use the bot due to this", Low: "I can work around it", Medium: (anything in between).
  Medium severity as it only affects new tokens - but can leave Carbon orders unfilled.

Steps to Reproduce

• Step-by-step reproduction method, include environment information if relevant - such as Python version and OS.
  Occurs in all modes

Expected Behavior

• What should happen
  The minimal solution should ensure that all tokens present on Carbon (within reason) have external pools mapped such that an arb route could be created. Best: direct pair match; 2nd: support by triangle.
  A better solution would permit any tokens to be arbed (independent of Carbon) but this may cause slowdown of the bot

Actual Behavior

• What currently happens
  New pools are added but not recognized by the bot. Bot cannot arb these. If this token appears or Carbon the order may go unfilled.

Proposed Solution

Much of the infrastructure required to solve this problem already exists in the terraformer some repurposing should occur but the minimal solution should look like this:

1. On startup, fetch all Carbon pairs (from all deployments on that chain)
2. Search all exchange Factory events for exact pair matches
3. Add the new pools to the appropriate event_mapping file

Optimizations

There are a number of optimizations that could be considered:

1. Fetching all Carbon pairs on startup PLUS every time a new strategy is created
2. Fetch all historical new pairs/pools can be time consuming - we could store in a static file and periodically update this - elaboration below
3. Add a minimum WEI filter for the pools that are added. Having low-liquidity pools is the same as not supporting the pair AND it can lead to failed txs and error messages PLUS bloats runtime
4. Exact match for PAIR plus find matches for TRIANGLES (could start with gas token / stables only)
5. The pairs list for Carbon.pairs() will contain pairs with no CURRENT liquidity on Carbon, so the pairs list could be created or cross-referenced with the StrategyUpdated events to ensure we only add relevant external exchanges

Storing historical pools

If fetching all the new pair events from the entire ecosystem is time-consuming or has RPC implications, we could store this history in the bot. The reason it would be separate from the event_mappings is that the event_mappings are iterated over during run time.
The reason it might be prohibitive to fetch every time is the shear number of blocks passed since the exchanges started e.g. multiple years worth of uniswap_v2 events.
Flow:

1. Seed all historic new pairs in a file (think of this as comprehensive event_mapping)
2. On run, update the historic file from its last block to the current block
3. Get the Carbon pairs, find exact matches in the historic file, and add these to the event_mappings
4. Default mode could ONLY store event_mappings for relevant Carbon strategies so that runtime is minimized

Impact Analysis (to be filled by contributors)

• Automated Tests: Are there existing tests covering this? Do we need new tests?
• Risks: Potential areas affected by this bug and its probable fix
• Performance: Is this bug affecting performance?
• Monitoring: Does this affect any current monitoring? Any new alerts needed?
• Logging: Any changes required in logging?
• Documentation: Updates needed in documentation?

[Lesigh-3100]

I believe event-fetching on bot start will be prohibitively time-consuming.

I want to suggest an alternative:

Manual Pair finding via Factory Contracts:
We could gather the Carbon Pairs and ensure we have viable pool combinations for them by querying Uni V2/3 & Solidly Factory contracts for pools - for example the Uniswap V2 getPair, Uniswap V3 getPool, and Solidly getPool.

The process could be:

1. Gather all Carbon pairs.
2. Find pairs without viable routes (this is more tricky if considering available liquidity).
3. Multicall each Factory searching for pools with token combinations that can create viable pairs/triangles.
4. Add each pool found to static data. If the pools are added only to the mapping files, the entire process would be fairly fast.