Refactored `Error` type + `Eff` and `Aff` applicative functors

There have been a number of calls on the Issues page for a ValidationAsync monad, which although it's a reasonable request (and I'll get to it at some point I'm sure), when I look at the example requests, it seems mostly the requestors want a smarter error handling story in general (especially for the collection of multiple errors).

The error-type that I'm building most of the modern functionality around (in Fin, Aff, and Eff for example) is the struct type: Error. It has been designed to handle both exceptional and expected errors. But the story around multiple errors was poor. Also, it wasn't possible to carry additional information with the Error, it was a closed-type other than ability to wrap up an Exception - so any additional data payloads was cumbersome and ugly.

Extending the struct type to be more featureful was asking for trouble, as it was already getting pretty messy.

Error refactor

So, I've bitten the bullet and refactored Error into an abstract record type.

Error sub-types

There are a few built-in sub-types:

• Exceptional - An unexpected error
• Expected - An expected error
• ManyErrors - Many errors (possibly zero)

These are the key base-types that indicate the 'flavour' of the error. For example, a 'user not found' error isn't
something exceptional, it's something we expect to happen. An OutOfMemoryException however, is
exceptional - it should never happen, and we should treat it as such.

Most of the time we want sensible handling of expected errors, and bail out completely for something exceptional. We also want to protect ourselves from information leakage. Leaking exceptional errors via public APIs is a sure-fire way to open up more information to hackers than you would like. The Error derived types all try to protect against this kind of leakage without losing the context of the type of error thrown.

When Exceptional is serialised, only the Message and Code component is serialised. There's no serialisation of the inner Exception or its stack-trace. It is also possible to construct an Exceptional message with an alternative message:

    Error.New("There was a problem", exception);

That means if the Error gets serialised, we only get a "There was a problem" and an error-code.

Deserialisation obviously means we can't recover the Exception, but the state of the Error will still be Exceptional - so it's possible to carry the severity of the error across domain boundaries without leaking too much information.

Error methods and properties

Essentially an error is either created from an Exception or it isn't. This allows for expected errors to be represented without throwing exceptions, but also it allows for more principled error handling. We can pattern-match on the
type, or use some of the built-in properties and methods to inspect the Error:

• IsExceptional - true for exceptional errors. For ManyErrors this is true if any of the errors are exceptional.
• IsExpected - true for non-exceptional/expected errors. For ManyErrors this is true if all of the errors are expected.
• Is<E>(E exception) - true if the Error is exceptional and any of the the internal Exception values are of type E.
• Is(Error error) - true if the Error matches the one provided. i.e. error.Is(Errors.TimedOut).
• IsEmpty - true if there are no errors in a ManyErrors
• Count - 1 for most errors, or n for the number of errors in a ManyErrors
• Head() - To get the first error
• Tail() - To get the tail of multiple errors

You may wonder why ManyErrors could be empty. That allows for Errors.None - which works a little like Option.None. We're saying: "The operation failed, but we have no information on why; it just did".

Error construction

The Error type can be constructed as before, with the various overloaded Error.New(...) calls.

For example, this is an expected error:

    Error.New("This error was expected")

When expected errors are used with codes then equality and matching is done via the code only:

    Error.New(404, "Page not found");

And this is an exceptional error:

    try
    {
    }
    catch(Exception e)
    {
        // This wraps up the exceptional error
        return Error.New(e);
    }

Finally, you can collect many errors:

   Error.Many(Error.New("error one"), Error.New("error two"));

Or more simply:

    Error.New("error one") + Error.New("error two")

Error types with additional data

You can extend the set of error types (perhaps for passing through extra data) by creating a new record that inherits Exceptional or Expected:

public record BespokeError(bool MyData) : Expected("Something bespoke", 100, None); 

By default the properties of the new error-type won't be serialised. So, if you want to pass a payload over the wire, add the [property: DataMember] attribute to each member:

public record BespokeError([property: DataMember] bool MyData) : Expected("Something bespoke", 100, None); 

Using this technique it's trivial to create new error-types when additional data needs to be moved around, but also there's a ton of built-in functionality for the most common use-cases.

Error breaking changes

• Because Error isn't a struct any more, default(Error) will now result in null. In practice this shouldn't affect anyone.
• BottomException is now in LanguageExt.Common

Error documentation

There's also a big improvement on the API documentation for the Error types

Aff and Eff applicative functors

Now that Error can handle multiple errors, we can implement applicative behaviours for Aff and Eff. If you think of monads enforcing sequential operations (and therefore can only continue if each operation succeeds - leading to only one error report if it fails), then applicative-functors are the opposite in that they can run independently.

This is what's used for the Validation monads, to allow multiple operations to be evaluated, and then all of the errors collected.

By adding Apply to Aff and Eff, we can now do the same kind of validation-logic both synchronously and asynchronously.

Contrived example

First let's create a simple asynchronous effect that delays for a period of time:

    static Aff<Unit> delay(int milliseconds) =>
        Aff(async () =>
        {
            await Task.Delay(milliseconds);
            return unit;
        });

Now we'll combine that so we get an effect that parses a string into an int, and adds a delay of 1000 milliseconds (the delay is to simulate calling some external IO).
:

    static Aff<int> parse(string str) =>
        from x in parseInt(str).ToAff(Error.New("parse error: expected int"))
        from _ in delay(1000)
        select x;

Notice how we're converting the Option<int> to an Aff, and providing an error value to use if the Option is None

Next we'll use the applicative behaviour of the Aff to run two operations in parallel. When they complete the values will be applied to the function that has been lifted by SuccessAff.

    static Aff<int> add(string sx, string sy) =>
        SuccessAff((int x, int y) => x + y) 
            .Apply(parse(sx), parse(sy));

To measure what we're doing, let's add a simple function called report. All it does is run an Aff, measures how long it takes, and prints the results to the screen:

    static async Task report<A>(Aff<A> ma)
    {
        var sw = Stopwatch.StartNew();
        var r = await ma.Run();
        sw.Stop();
        Console.WriteLine($"Result: {r} in {sw.ElapsedMilliseconds}ms");
    }

Finally, we can run it:

    await report(add("100", "200"));
    await report(add("zzz", "yyy"));

The output for the two operations is this:

Result: Succ(300) in 1032ms
Result: Fail([parse error: expected int, parse error: expected int]) in 13ms

Notice how the first one (which succeeds) takes 1032ms - i.e. the two parse operations ran in parallel. And on the second one, we get both of the errors returned. The reason that one finished so quickly is because the delay was after the parseInt call, so we exited immediately.

Of course, it would be possible to do this:

   from x in parse(sx)
   from y in parse(sy)
   select x + y;

Which is more elegant. But the success path would take 2000ms, and the failure path would only report the first error.

Hopefully that gives some insight into the power of applicatives (even if they're a bit ugly in C#!)

Beta

This will be in beta for a little while, as the changes to the Error type are not trivial.

Effect scheduling improvements

The existing Schedule type has been massively upgraded to support even more complex scheduling for repeating, retrying, and folding of Aff and Eff types.

A huge thanks to @bmazzarol who did all of the heavy lifting to make this feature a reality!

It has been refactored from the ground up, a Schedule now is a (possibly infinite) stream of durations. Each duration indicates to the retry, repeat, and fold behaviours how long to wait between each action. The generation of those streams comes from:

• Schedule.Forever - infinite stream of zero length durations
• Schedule.Once - one item stream of zero length duration
• Schedule.Never - no durations (a schedule that never runs)
• Schedule.TimeSeries(1, 2, 3 ...) - pass in your own durations to build a bespoke schedule
• Schedule.spaced(space) - infinite stream of space length durations
• Schedule.linear(seed, factor) - schedule that recurs continuously using a linear back-off
• Schedule.exponential(seed, factor) - schedule that recurs continuously using a exponential back-off
• Schedule.fibonacci(seed, factor) - schedule that recurs continuously using a fibonacci based back-off
• Schedule.upto(max) - schedule that runs for a given duration
• Schedule.fixedInterval(interval) - if that action run between updates takes longer than the interval, then the action will run immediately
• Schedule.windowed(interval) - a schedule that divides the timeline into interval-long windows, and sleeps until the nearest window boundary every time it recurs.
• Schedule.secondOfMinute(second) - a schedule that recurs every specified second of each minute
• Schedule.minuteOfHour(minute) - a schedule that recurs every specified minute of each hour
• Schedule.hourOfDay(hour) - a schedule that recurs every specified hour of each day
• Schedule.dayOfWeek(day) - a schedule that recurs every specified day of each week

These schedules are mostly infinite series, and so to control their 'length' we compose with ScheduleTransformer values to create smaller series, or to manipulate the series in some way (jitter for example). The following functions generate ScheduleTransformer values.

• Schedule.recurs(n) - Clamps the schedule durations to only recur n times.
• Schedule.NoDelayOnFirst - Regardless of any other settings, it makes the first duration zero
• Schedule.RepeatForever - Repeats any composed schedules forever
• Schedule.maxDelay(max) - limits the returned delays to max delay (upper clamping of durations).
• Schedule.maxCumulativeDelay(Duration max) - keeps a tally of all the delays so-far, and ends the generation of the series once max delay has passed
• Schedule.jitter(minRandom, maxRandom, seed) - adds random jitter to the durations
• Schedule.jitter(factor, seed) - adds random jitter to the durations
• Schedule.decorrelate(factor, seed) - transforms the schedule by de-correlating each of the durations both up and down in a jittered way.
• Schedule.resetAfter(max) - resets the schedule after a provided cumulative max duration
• Schedule.repeats(n) - not to be confused with recurs, this repeats the schedule n times.
• Schedule.intersperse(schedule) - intersperse the provided schedule between each duration in the schedule.

Schedule and ScheduleTransformer can be composed using | (union) or & (intersection):

    var schedule = Schedule.linear(1 * sec) | Schedule.recurs(3) | Schedule.repeat(3);

    // [1s, 2s, 3s, 1s, 2s, 3s, 1s, 2s, 3s]

Union | will take the minimum of the two schedules to the length of the longest, intersect & will take the maximum of the two schedules to the length of the shortest.

One thing remaining to-do is to bring HasTime<RT> back into the Core and allow these schedules to use injectable time. Some of the functions already take a Func<DateTime> to access 'now', this will be expanded so time can be sped up or slowed down, with the schedules 'just working'. That'll be in the next few weeks I'm sure, and is related to this issue.

Check out the API documentation to see what's what. And again, thanks to @bmazzarol for the hard work 👍

BREAKING: New LanguageExt.Transformers package

The transformers extensions, which are a big set of T4 templates for generating extension methods for nested monadic types have now been broken out into their own package: LanguageExt.Transformers

If you use the following functions: BindT, MapT, FoldT, FoldBackT, ExistsT, ForAllT, IterT, FilterT, PlusT, SubtractT, ProductT, DivideT, SumT, CountT, AppendT, CompareT, EqualsT, or ApplyT - then you will get compile errors, and will need to add a reference to the LanguageExt.Transformers package.

I've done this for a couple of reasons:

• There's been an ongoing concern from a number of users of this library about the size of the LanguageExt.Core library. This change takes the Core package from 3,276 kb to 2,051 kb.
  • The Core library will always be quite chunky because of the sheer amount of features, but the transformer extension methods definitely aren't always needed, so breaking them out made sense
• I suspect issues around the .NET ReadyToRun usage will be alleviated somewhat by this change.
  • I can't prove this, but the C# tooling has had a hard time with those 10,000s of generated extension methods before - so rather than wait for Microsoft to fix their tooling, I'm trying to be proactive and see if this will help.

The main transformer extensions that remain in the Core library are:

• Traverse
• Sequence

These are so heavily used that I believe moving them out into the Transformers library would mean everyone would be obliged to use it, and therefore it wouldn't achieve anything. There may be an argument for bringing BindT and MapT back into the core at some point. I will see how this plays out (it wouldn't be a future breaking change if that were the case).

Any problems, please report via the Issues in the usual way.

Aff, Eff, AtomHashMap, TrackingHashMap + fixes [RTM]

Language-ext had been in beta for a few months now. Today we go back to full 'RTM' mode.

The recent changes are:

AtomHashMap and Ref change-tracking

A major new feature that allows for tracking of changes in an AtomHashMap or a Ref (using STM). The Change event publishes HashMapPatch<K, V> which will allow access to the state of the map before and after the change, as well as a HashMap<K, Change<V>> which describes the transformations that took place in any transactional event.

In the LanguageExt.Rx project there's various observable stream extensions that leverage the Change event:

• OnChange() - which simply streams the HashMapPatch<K, V>
• OnEntryChange() - which streams the Change<V> for any key within the map
• OnMapChange() - which streams the latest HashMap<K, V> snapshot

Ref which represents a single value in the STM system, has a simpler Change event that simply streams the latest value. It also has an Rx extension, called Change().

Documented in previous beta release notes

TrackingHashMap<K, V>

This is a new immutable data-structure which is mostly a clone of HashMap; but one that allows for changes to be tracked. This is completely standalone, and not related to the AtomHashMap in any way other than it's used by the AtomHashMap.Swap method. And so, this has use-cases of its own.

Changes are tracked as a HashMap<K, Change<V>>. That means there's at most one change-value stored per-key. So, there's no risk of an ever expanding log of changes, because there is no log! The changes that are tracked are from the 'last snapshot point'. Which is from the empty-map or from the last state where tracking HashMap.Snapshot() is called.

Documented in previous beta release notes

Fixes to the Aff and Eff system

The Aff and Eff system had some unfortunate edge-cases due to the use of memoisation by-default. The underlying system has been simplified to be more of a standard reader-monad without memoisation. You can still memoise if needed by calling: ma.Memo().

Future changes will make the Aff and Eff into more of a DSL, which will allow for certain elements of the system to be 'pure', and therefore safely memoisable, and other elements not. My prototype of this is in too early a stage to release though, so I've taken the safer option here.

Breaking change: Both Clone and ReRun have been removed, as they are now meaningless.

New package LanguageExt.SysX - for .NET5.0+ features

The LanguageExt.Sys package is a wrapper for .NET BCL IO functionality, for use with the runtime Eff and Aff monads. This is going to stay as netstandard2.0 for support of previous versions of the .NET Framework and .NET Core. This new package adds features that are for net5.0+.

The first feature to be supported is the Activity type for Open-Telemetry support.

ScheduleAff and ScheduleEff usage was inconsistent

Depending on the monad they were used with, you might see a 'repeat' that was 1 greater than it should have been. This is now fixed.

Lazy Seq equality fix

Under certain rare circumstances it was possible for the equality operator to error with a lazy Seq. Fix from @StefanBertels - thanks!

No more empty array allocations

It seams c# isn't smart enough to turn a new A[0] into a non-allocating operation. And so they have all been replaced with Array.Empty(). Fix from @timmi-on-rails - thanks!

Any problems, please report them in the Issues as usual. Paul 👍

Change events for AtomHashMap and the STM system (via Ref)

As requested by @CK-LinoPro in this Issue. AtomHashMap and Ref now have Change events.

Change<A>

Change<A> is a new union-type that represents change to a value, and is used by AtomHashMap and TrackingHashMap.

You can pattern-match on the Change value to find out what happened:

public string WhatHappened(Change<A> change) =>
    change switch
    {
        EntryRemoved<A> (var oldValue)      => $"Value removed: {oldValue}",
        EntryAdded<A> (var value)           => $"Value added: {value}",
        EntryMapped<A, A>(var from, var to) => $"Value mapped from: {from}, to: {to}",
        _                                   => "No change"
    };

EntryMapped<A, B> is derived from EntryMappedFrom<A> and EntryMappedTo<B>, so for any A -> B mapping change, you can just match on the destination value:

public string WhatHappened(Change<A> change) =>
    change switch
    {
        EntryRemoved<A> (var oldValue)      => $"Value removed: {oldValue}",
        EntryAdded<A> (var value)           => $"Value added: {value}",
        EntryMappedTo<A>(var to)            => $"Value mapped from: something, to: {to}",
        _                                   => "No change"
    };

That avoids jumping through type-level hoops to see any changes!

There are also various 'helper' properties and methods for working with the derived types:

Member	Description
HasNoChange	true if the derived-type is a NoChange<A>
HasChanged	true if the derived-type is one-of EntryRemoved<A> or EntryAdded<A> or EntryMapped<_, A>
HasAdded	true if the derived-type is a EntryAdded<A>
HasRemoved	true if the derived-type is a EntryRemoved<A>
HasMapped	true if the derived-type is a EntryMapped<A, A>
HasMappedFrom<FROM>()	true if the derived-type is a EntryMappedFrom<FROM>
ToOption()	Gives the latest value from the Change, as long as the Change is one-of EntryAdded or EntryMapped or EntryMappedTo

There are also constructor functions to build your own Change values.

AtomHashMap<K, V> and AtomHashMap<EqK, K, V>

The two variants of AtomHashMap both now have Change events that can be subscribed to. They emit a HashMapPatch<K, V> value, which contains three fields:

Field	Description
From	HashMap<K, V> that is the state before the change
To	HashMap<K, V> that is the state after the change
Changes	HashMap<K, Change<V>> that describes the changes to each key

There are three related Rx.NET extensions in the LanguageExt.Rx package:

AtomHashMap Extension	Description
OnChange()	Observable stream of HashMapPatch<K, V>
OnMapChange()	Observable stream of HashMap<K, V>, which represents the latest snapshot of the AtomHashMap
OnEntryChange()	Observable stream of (K, Change<V>), which represents the change to any key within the AtomHashMap

Example

var xs = AtomHashMap<string, int>();

xs.OnEntryChange().Subscribe(pair => Console.WriteLine(pair));
            
xs.Add("Hello", 456);
xs.SetItem("Hello", 123);
xs.Remove("Hello");
xs.Remove("Hello");

Running the code above yields:

(Hello, +456)
(Hello, 456 -> 123)
(Hello, -123)

Swap method (potential) breaking-change

The implementation of Swap has changed. It now expects a Func<TrackingHashMap<K, V>, TrackingHashMap<K, V>> delegate instead of a Func<HashMap<K, V>, HashMap<K, V>> delegate. This is so the Swap method can keep track of arbitrary changes during the invocation of the delegate, and then emit them as events after successfully committing the result.

Ref<A>

Refs are used with the atomic(() => ...), snapshot(() => ...), and serial(() => ...) STM transactions. Their changes are now tracked during a transaction, and are then (if the transaction is successful) emitted on the Ref<A>.Change event. These simply publish the latest value.

As before there's Rx extensions for this:

Ref Extension	Description
OnChange()	provides an observable stream of values

Example

var rx = Ref("Hello");
var ry = Ref("World");

Observable.Merge(rx.OnChange(), 
                 ry.OnChange())
          .Subscribe(v => Console.WriteLine(v));

atomic(() =>
{
    swap(rx, x => $"1. {x}");
    swap(ry, y => $"2. {y}");
});

This outputs:

1. Hello
2. World

TrackingHashMap

This is a new immutable data-structure which is mostly a clone of HashMap; but one that allows for changes to be tracked. This is completely standalone, and not related to the AtomHashMap in any way other than it's used by the AtomHashMap.Swap method. And so, this has use-cases of its own.

Changes are tracked as a HashMap<K, Change<V>>. That means there's at most one change-value stored per-key. So, there's no risk of an ever expanding log of changes, because there is no log! The changes that are tracked are from the 'last snapshot point'. Which is from the empty-map or from the last state where trackingHashMap.Snapshot() is called.

Example 1

var thm = TrackingHashMap<int, string>();

Console.WriteLine(thm.Changes);

thm = thm.Add(100, "Hello");
thm = thm.SetItem(100, "World");

Console.WriteLine(thm.Changes);

This will output:

[]
[(100: +World)]

Note the +, this indicates an Change.EntryAdded. And so there has been a single 'add' of the key-value pair (100, World). The "Hello" value is ignored, because from the point of the snapshot: a value has been added. That's all we care about.

If I take a snapshot halfway through, you can see how this changes the output:

var thm = TrackingHashMap<int, string>();

thm = thm.Add(100, "World");
thm = thm.Snapshot();
thm = thm.SetItem(100, "Hello");

Console.WriteLine(thm.Changes);

This outputs:

[(100: World -> Hello)]

So the snapshot is from when there was a (100, World) pair in the map.

Hopefully that gives an indication of how this works!

v4.0.2 ROLLBACK - Reverting change to Seq1

@StefanBertels has rightly highlighted some possible edge-cases with yesterday's rollout. These edge cases could turn into very real bugs in your application, and so I am:

• Rolling back the change to Seq1: Seq1 will continue to be the way you create singleton Seq<A> collections.
• Keeping the migration of the Seq conversion functions to toSeq. This will be consistently be a compilation error, which you can fix up by just renaming the Seq(blah) to toSeq(blah) or blah.ToSeq().

It isn't ideal that Seq1 is still hanging around, being all inconsistent with the rest of the library, but neither is putting a burden of tracking all possible edge cases. And so, this is the right thing to do.

What I will do is pause the migration of Seq1 to Seq until June 2022. That should give everyone enough time to get to version 4 and migrate their Seq conversion function usage to toSeq. Then the subsequent change will have zero impact.

For those that don't upgrade in that time and want the safest possible migration, they can upgrade to a pre-June 2022 version, fix the conversion functions and then upgrade to the latest.

Apologies to anyone who started the migration yesterday.

Breaking change: Seq1 to Seq

This is the backstory: When language-ext was still in version 1, there was no Seq<A> collection type. There were however Seq(...) functions in the Prelude. Those functions coerced various types into being IEnumerable, and in the process protected against any of them being null.

These are they:

	Seq <A> (A? value)
	Seq <A> (IEnumerable<A> value)
	Seq <A> (A[] value)
	Seq <A> (Arr<A> value)
	Seq <A> (IList<A> value)
	Seq <A> (ICollection<A> value)
	Seq <L, R> (Either<L, R> value)
	Seq <L, R> (EitherUnsafe<L, R> value)
	Seq <A> (Try<A> value)
	Seq <A> (TryOption<A> value)
	Seq <T> (TryAsync<T> value)
	Seq <T> (TryOptionAsync<T> value)
	Seq <A> (Tuple<A> tup)
	Seq <A> (Tuple<A, A> tup)
	Seq <A> (Tuple<A, A, A> tup)
	Seq <A> (Tuple<A, A, A, A> tup)
	Seq <A> (Tuple<A, A, A, A, A> tup)
	Seq <A> (Tuple<A, A, A, A, A, A> tup)
	Seq <A> (Tuple<A, A, A, A, A, A, A> tup)
	Seq <A> (ValueTuple<A> tup)
	Seq <A> (ValueTuple<A, A> tup)
	Seq <A> (ValueTuple<A, A, A> tup)
	Seq <A> (ValueTuple<A, A, A, A> tup)
	Seq <A> (ValueTuple<A, A, A, A, A> tup)
	Seq <A> (ValueTuple<A, A, A, A, A, A> tup)
	Seq <A> (ValueTuple<A, A, A, A, A, A, A> tup)

When Seq<A> the type was added, I needed constructor functions, which are these:

    	Seq <A> ()
	Seq1 <A> (A value)
	Seq <A> (A a, A b)
	Seq <A> (A a, A b, A c, params A[] ds)

The singleton constructor needed to be called Seq1 because it was clashing with the original single-argument Seq functions from v1 of language-ext.

This has been bothering me for a long time. So, it's time to take the hit.

All of the legacy coercing functions (the first list) are now renamed to toSeq. And Seq1 has been renamed to Seq (well, Seq1 still exists, but it's marked [Obsolete].

The breaking change is that if you use any of those legacy Seq functions, you'll either need to change, this:

    Seq(blah);

Into,

    toSeq(blah);

Or,

    blah.ToSeq();

There were quite a lot of the coercion functions used in language-ext, and it took me about 10 minutes to fix it up. The good thing is that they all turn into compilation errors, no hidden behaviour changes.

Apologies for any inconvenience with this. I figure that as we've switched into v4 that this would be a good time for a clean break, and to tidy up something that I have actively seen cause confusion for devs. Seq is the most performance immutable list in the .NET sphere, so it shouldn't really be confusing to use!

Language-Ext Version 4.0.0 (finally out of beta!)

Here it is, version 4.0 of language-ext - we're finally out of beta. I had hoped to get some more documentation done before releasing, but I have ever decreasing free time these days, and I didn't want to delay the release any further. A lot has happened since going into beta about a year or so ago:

Effects

How to handle side-effects with pure functional code has been a the perennial question. It is always coming up in the issues and it's time for this library to be opinionated on that. Language-ext already has Try and TryAsync which are partial solutions to this problem, but they only really talk to the exception catching part of IO. Not, "How do we inject mocked behaviours?", "How do we deal with configuration?", "How do we describe the effects that are allowed?", "How do we make async and sync code play well together?" etc.

Language-Ext 4.0 has added four new monadic types:

• Eff<A> - for synchronous side-effects. This is a natural replacement for Try<A>
• Aff<A> - for asynchronous side-effects. This is a natural replacement for TryAsync<A>
• Eff<RT, A> - for synchronous side-effects with an injectable runtime. This is like the Reader<Env, A> monad on steroids!
• Aff<RT, A> - for asynchronous side-effects with an injectable runtime. This is like the ReaderAsync<Env, A> monad that never existed!

These four monads have been inspired by the more bespoke monads I'd been building personally to deal with IO in my private projects, as well as the Aff and Eff monads from PureScript, and also a nod to ZIO from the Scala eco-system. Together they form the Effects System of language-ext, and will be the key area of investment going forward.

These four monads all play well together, they can be used in the same LINQ expression and everything just works.

There is now a whole section of the wiki dedicated to dealing with side-effects from first principles. I ran out of steam to complete it, but I hope to get back to it soon. It does go into depth on the Aff and Eff monads, and how to build runtimes.

On top of that, there's a project called EffectsExamples in the Samples folder. It is a whole application, with an effects driven menu system, for showing what's possible.

Finally, there's a new library called LanguageExt.Sys that wraps up the core side-effect producing methods in .NET and turns them into Aff or Eff monads. So far it covers:

• Time
• Environment
• Encoding
• Console
• IO.Directory
• IO.File
• IO.Stream
• IO.TextRead

This library will grow over time to cover more of .NET. You are not limited to using these: the runtime system is fully extensible, and allows you to add your own side-effecting behaviors.

Pipes

Pipes is a C# interpretation of the amazing Haskell Pipes library by Gabriella Gonzalez. It is a streaming library that has one type: Proxy. This one type can represent:

• Producers
• Pipes
• Consumers
• Client
• Server

Each Proxy variant can be composed with the others, and when you've got it correct they all fuse into an Effect that can be run. For example, a Producer composed with a Pipe, and then composed with a Consumer will fuse into a runnable effect. Each component is standalone, which means it's possible to build reusable effects 'components'

It also supports either uni or bi-directional streaming and can work seamlessly with the effects-system.

It's actually been in language-ext for a long time, but it was borderline unusable because of the amount of generic arguments that were needed to make it work. Since then I've augmented it with additional monads and 100s of SelectMany overrides that make it easier to use. I have also removed all recursion from it, which was a problem before.

The original Haskell version is a monad transformer, meaning that it can wrap other monads. Usually, you'd wrap up the IO monad, but it wasn't limited to that. However, the C# version can't be as general as that, so it only wraps the Aff and Eff monads. However, I have managed to augment the original to also support:

• IEnumerable
• IObservable
• IAsyncEnumerable

This means we can handle asynchronous and synchronous streams. All of which play nice with the effects system. Here's a small example:

    public class TextFileChunkStreamExample<RT> where RT : 
        struct, 
        HasCancel<RT>,
        HasConsole<RT>,
        HasFile<RT>,
        HasTextRead<RT>
    {
        public static Aff<RT, Unit> main =>
            from _ in Console<RT>.writeLine("Please type in a path to a text file and press enter")
            from p in Console<RT>.readLine
            from e in mainEffect(p)
            select unit;
        
        static Effect<RT, Unit> mainEffect(string path) =>
            File<RT>.openRead(path) 
               | Stream<RT>.read(80) 
               | decodeUtf8 
               | writeLine;

        static Pipe<RT, SeqLoan<byte>, string, Unit> decodeUtf8 =>
            from c in awaiting<SeqLoan<byte>>()         
            from _ in yield(Encoding.UTF8.GetString(c.ToReadOnlySpan()))
            select unit;
        
        static Consumer<RT, string, Unit> writeLine =>
            from l in awaiting<string>()
            from _ in Console<RT>.writeLine(l)
            select unit;
    }

• This example asks the user to enter a filename, then reads the file in 80 byte chunks, converts those chunks into strings, and then writes them to the console
• mainEffect is where the Effect is fused by composing the File.openRead producer, the Stream.read pipe, the decodeUtf8 pipe, and the writeLine consumer.
• You may notice the new type SeqLoan. This is a version of Seq that has a borrowed backing array from an ArrayPool. The Pipes system automatically cleans up resources for you. And so, this stream doesn't thrash the GC

NOTE: The client/server aspect of this still has the generics problem, and is difficult to use. I will be doing the same work I did for Producers, Pipes, and Consumers to make that more elegant over time.

Concurrency

Although this library is all about immutability and pure functions, it is impossible (at least in C#) to get away from needing to mutate something. Usually it's a shared cache of some sort. And that's what the concurrency part of language-ext is all about, dealing with that messy stuff. We already had Atom for lock-free atomic mutation of values and Ref for use with the STM system.

In this release there have been some major improvements to Atom and Ref (no more deadlocks). As well as the addition of three new lock-free atomic collections:

• AtomHashMap
• AtomSeq
• VersionHashMap

And two other data-structures:

• VersionVector
• VectorClock

All but VectorClock, VersionVector, and VersionHashMap were detailed in previous release notes, so I won't cover those again. VectorClock, VersionVector, and VersionHashMap allow you to easily work with distributed versioned changes. Vector-clocks comes from the genius work of Leslie Lamport amongst others, and allow for tracking of causal or conflicting events; when combined with some data in VersionVector it's possible to resolve many clients wanting to change the same piece of data, knowing who did what first. When VersionVector is combined with an atomic-hash-map, you get VersionHashMap. This manages multiple keys of VersionVectors, making it a short hop to a distributed database.

There's some good blog posts on the Riak site that explain vector clocks.

when and unless

These two functions make working with alternative values easier in LINQ expressions.

    from txt1 in File<RT>.readAllText(path)
    from txt2 in when(txt1 == "", SuccessEff("There wasn't any text in the file, so use this instead!"))
    select txt;

The dual of when is unless, it simply reverses the predicate.

guard and guardnot

These are similar to when and unless in that they take a predicate, but instead they return the monad's alternative value (Left for Either, Error for Fin, and Aff, for example). This makes it easy to catch problems and generate contextual error messages.

Pretty

Another Haskell library I've taken inspiration from is Pretty. It's synopsis is:

Pretty is a pretty-printing library, a set of API's that provides a way to easily print out text in a consistent format of your choosing. This is useful for compilers and related tools.
It is based on the pretty-printer outlined in the paper 'The Design of a Pretty-printing Library' by John Hughes in Advanced Functional Programming, 1995

It's a fully composable text document building system that can understand layouts and can do lovely reformatting functions depending on context (like the desired width of text, the tabs, etc.). If you've ever had to manage printing of text with indented tabs, this is the feature for you.

It's pretty new in the library, I needed it for some compiler work I was working on, and it seemed generally useful - so I've added it. However, there's no documentation [other tha...

Concurrency improvements (AtomHashMap, Atom, Ref)

This release features improvements to the concurrency elements of language-ext:

• Atom - atomic references
• STM / Ref - software-transactional memory system
• AtomHashMap - new atomic data-structure

Atom and STM

The atomic references system (Atom) that wraps a value and allows for atomic updates; and the STM system that allows many Ref values (that also wrap a value, but instead work within an atomic sync transaction), have both been updated to never give up trying to resolve a conflict.

Previously they would fail after 500 attempts, but now any conflicts will cause the conflicting threads to back-off and eventually yield control so that the other thread(s) they were in conflict with eventually win and can update the atomic reference. This removes the potential time-bomb buried deep within the atomic references system, and creates a robust transactional system.

AtomHashMap<K, V>

One pattern I noted I was doing quite a lot was wrapping HashMap in an atom, usually for shared cached values:

   var atomData = Atom(HashMap<string, int>());

   atomData.Swap(static hm => hm.AddOrUpdate("foo", 123));

It struck me that it would be very useful to have atomic versions of all of the collection data-structures of language-ext. The first one is AtomHashMap. And so, instead of the code above, we can now write:

    var atomData = AtomHashMap<string, int>();

    atomData.AddOrUpdate("foo", 123);

All operations on AtomHashMap are atomic and lock-free. The underling data structure is still an immutable HashMap. It's simply the reference to the HashMap that gets protected by AtomHashMap, preventing two threads updating the data structure with stale data.

The main thing to understand with AtomHashMap is that if a conflict occurs on update, then any transformational operation is re-run with the new state of the data-structure. Obviously conflicts are rare on high-performance CPUs, and so we save processing time from not taking locks on every operation, at the expense of occasional re-running of operations when conflicts arise.

Swap

AtomHashMap also supports Swap, which allows for more complex atomic operations on the underlying HashMap. For example, if your update operation relies on data within the AtomHashMap, then you might want to consider wrapping everything within a Swap call to allow for fully idempotent transformations:

    atomData.Swap(data => data.Find("foo").Case switch
                          {
                              int x => data.SetItem("foo", x + 1),
                              _     => data.Add("foo", 1)
                          });

NOTE: The longer you spend inside a Swap function, the higher the risk of conflicts, and so try to make sure you do the bare minimum within swap that will facilitate your idempotent operation.

In Place Operations

Most operations on AtomHashMap are in-place, i.e. they update the underlying HashMap atomically. However, some functions like Map, Filter, Select, Where are expected to process the data-structure into a new data-structure. This is usually wanted, but we also want in-place filtering and mapping:

    // Only keeps items with a value > 10 in the AtomHashMap
    atomData.FilterInPlace(x => x > 10);

    // Maps all items in the AtomHashMap 
    atomData.MapInPlace(x => x + 10);

The standard Map, Filter, etc. all still exist and work in the 'classic' way of generating a new data structure.

ToHashMap

At any point if you need to take a snapshot of what's in the AtomHashMap you can all:

    HashMap<string, int> snapshot = atomData.ToHashMap();

This is a zero allocation, zero time (well in the order of nanoseconds), operation. And so we can easily take snapshots and work on those whilst the atomic data structure can carry on being mutated without consequence.

The Rest

As well as AtomHashMap<K, V> there's also AtomHashMap<EqK, K, V> which maps to HashMap<EqK, K, V>.

This is just the beginning of the new Atom based data-structures. So watch this space!

Code-gen improvements - discriminated unions

This release features updates to the [Union] feature of language-ext (discriminated union generator) as well as support for iOS Xamarin runtime code-gen

• Generates a Match function for each union
• Generates a Map function for each union.
• Unions ignore ToString and GetHashCode - allowing for your own bespoke implementations
• iOS Xamarin doesn't allow usage of IL, and so for platforms that don't support IL the runtime code-gen (for Record<A>, Range, and other types) now falls back to building implementations with Expression.

Match

Unions can already be pattern-matched using the C# switch statement and switch expression. Now you can use the generated Match function like other built-in types (such as Option, Either, etc.). This match function enforces completeness checking, which the C# switch can't do.

There are two strategies to generating the Match function:

1. If the [Union] type is an interface, then the generated Match function will be an extension-method. This requires your union-type to be a top-level declaration (i.e. not nested within another class). This may be a breaking change for you.

It looks like this:

public static RETURN Match<A, RETURN>(this Maybe<A> ma, Func<Just<A>, RETURN> Just, Func<Nothing<A>, RETURN> Nothing) =>
    ma switch
    {
        Just<A> value => Just(value), 
        Nothing<A> value => Nothing(value), 
        _ => throw new LanguageExt.ValueIsNullException()
    };

2. If the [Union] type is an abstract partial class, then the generated Match function will be an instance-method. This doesn't have the limitation of the interface approach.

Then the Match instance methods look like this:

// For Just<A>
public override RETURN Match<RETURN>(Func<Just<A>, RETURN> Just, Func<Nothing<A>, RETURN> Nothing) => 
    Just(this);

// For Nothing<A>
public override RETURN Match<RETURN>(Func<Just<A>, RETURN> Just, Func<Nothing<A>, RETURN> Nothing) => 
    Nothing(this);

And so, if you still need unions nested within other classes, switch the interface to an abstract class:

This is an example of an interface based union:

    [Union]
    internal interface Maybe<A>
    {
        Maybe<A> Just(A value);
        Maybe<A> Nothing();
    }

This is the equivalent as an abstract class:

    [Union]
    public abstract partial class Maybe<A>
    {
        public abstract Maybe<A> Just(A value);
        public abstract Maybe<A> Nothing();
    }

Map

One thing that is true of all algebraic-data-types (of which category discriminated-unions fall into); is that there is exactly one way to generate a functor Map function (Select in LINQ) it's known as a theorem for free. And so the implementation for Map and Select are now provided by default by the code-gen.

So, for the example below:

    [Union]
    public abstract partial class Maybe<A>
    {
        public abstract Maybe<A> Just(A value);
        public abstract Maybe<A> Nothing();
    }

This code is auto-generated:

public static Maybe<B> Map<A, B>(this Maybe<A> ma, System.Func<A, B> f) =>
    ma switch
    {
        Just<A> _v => new Just<B>(f(_v.Value)), 
        Nothing<B> _v => new Nothing<B>(), 
        _ => throw new System.NotSupportedException()
    };

public static Maybe<B> Select<A, B>(this Maybe<A> ma, System.Func<A, B> f) =>
    ma switch
    {
        Just<A> _v => new Just<B>(f(_v.Value)), 
        Nothing<B> _v => new Nothing<B>(), 
        _ => throw new System.NotSupportedException()
    };

This will currently only support single generic argument unions, it will be expanded later to provide a Map function for each generic argument. The other limitation is that if any of the cases have their own generic arguments, then the Map function won't be generated. I expect this to cover a large number of use-cases though.

Any problems please report in the repo Issues.

Paul