Repository navigation
Description
Lowering of extensions
Extensions are "transparent wrappers", that allow types to be augmented with additional members and (eventually) interfaces.
This outlines how we can implement extensions by lowering them to structs, and applying an erasure approach to signatures and generic instantiation.
In this document, base extensions and interfaces are ignored, except for brief consideration at the end.
Extension declarations
An extension declaration like this:
public extension E for U
{
public static U Create() { ... }
public static U operator +(U e1, U e2) { ... }
public int M() { ... this ... }
public string this[int i] { ... this ... }
}is lowered to a struct declaration, which contains a private field of the underlying type, as well as the function members from the extension declaration, modified to indirect through the field as necessary:
public struct E
{
private U __this;
public static U Create() { ... }
public static U operator +(U e1, U e2) { ... }
public int M() { ... __this ... }
public string this[int i] { ... __this ... }
}In addition an attribute or other marker may be used to designate that the struct represents an extension.
Extensions in generic instantiations
Extensions used as type arguments or array element types are erased to their underlying type:
List<E>is lowered to
List<U>This is the main mechanism ensuring that extensions and their underlying types are interchangeable, even through generic instantiation. In this way, a collection of the underlying type can be freely reinterpreted in terms of an extension, and the elements thus gain the extra members afforded by the extension.
Within member bodies, the compiler can keep track of the fact that a List<U> is "really" a List<E>, and provide appropriate conversions.
Extensions in signatures
Extensions are erased from signatures, and are instead communicated through metadata (attributes).
public E M(E e, E[] a, List<E> l) { ... }is lowered to something like
public U M([Extension(...)] U e, [Extension(...)] U[] a, [Extension(...)] List<U> l) { ... }The exact encoding scheme for the attributes is TBD, but will probably resemble those for nullable reference types and tuple element names, which are similarly type elements that are erased by the compiler.
This encoding scheme means that methods cannot be overloaded by different extensions for the same underlying type.
Extension member access
In order to provide access to extension members, the compiler is able to freely convert between the extension type and its underlying type.
This conversion relies on the fact that an extension always has exactly the same shape in memory as the underlying type. This makes it safe for the compiler to utilize the Unsafe.As(...) method.
The method
void M(E e)
{
var i = e.M();
}is lowered to
void M([Extension(...)] U e)
{
ref var __e = Unsafe.As<U, E>(ref e); // Creates a ref of type E to e
var i = __e.M();
}Conversions to and from extensions
The bi-directional implicit identity conversion between extensions and their underlying type can likewise be supported through the use of Unsafe.As:
U u = ...;
E e = u; // identity conversion
e.M();which lowers to
U u = ...;
E e = Unsafe.As<U, E>(ref u);
e.M();If the underlying type is a value type, the user may need to make e a ref to u, so that the value isn't copied in the conversion, and mutations occurring within extension members apply back to the underlying value:
U u = ...;
ref E e = ref u; // identity conversion
e.M(); // Mutations apply to uwhich lowers to
U u = ...;
ref E e = ref Unsafe.As<U, E>(ref u); // identity conversion
e.M(); // Mutations apply to uImplicit extensions
Implicit extensions are automatically used as "fallback" extensions for their underlying type in a given static scope. The compiler uses lookup machinery similar to today's extension methods to find where extension members are invoked, and implicitly inserts a conversion to the appropriate extension type. This is described in detail in the Extensions specification.
From there, the lowering proceeds as described above.
Base extensions
It is TBD how base extensions are encoded in the lowered extension declaration. When converting to base extensions, the compiler can use the same approach as between extensions and underlying types, since the underlying representation in memory is unchanged.
Extensions with interfaces
The lowering of an extension declaration which implements interfaces is in and of itself simple: Simply add the interfaces to the lowered struct declaration.
However, semantics become complicated. In particular, the implicit conversions between generic instantiations over extensions vs underlying types may no longer apply when the extension implement interfaces, and those interfaces are material to satisfying constraints in the generic instantiation itself.
A compiler-only approach to this would be to simply not have such conversions when an extension implements an interface, but that means adding an interface to an extension is highly breaking. A more permissive and situational approach would likely require significant new runtime feature work however.
Activity
kyoyama-kazusa commentedon Dec 16, 2023
KennethHoff commentedon Dec 16, 2023
That means you can't do something like this, which is unfortunate:
Instead you'd have to do this:
(I'm also assuming
public extension X for Ain this context is anexplicit extension- meaningexplicitis implicit, and you have to explicitly state if you want it to be an implicit extension - something I agree with, but can be confusing to developers at first glance).The thing I think/thought I will be using
explicit extensionsfor the most is for typed values like this, where I want something to be functionally identical to a backing type, but has a different meaning in my domain. Currently I do this by making something like this, but that has a lot of problems - especially with IO (Serialization/Deserialization)En3Tho commentedon Dec 16, 2023
I really wonder how this can be properly done without runtime support.
For example:
KennethHoff commentedon Dec 17, 2023
Adding to what @En3Tho and I said yesterday. I consider being unable to overload based on extensions to be a fairly huge problem, so I really hope this will be considered when working out the implementation details.
I think it's time to consider updating the IL metadata. Alongside this issue (extensions) - which I presume would be able to bypass the "According to IL it is the underlying type, so you can't overload based on them" problem with an updated IL representation - there are a few other language features that I believe would benefit from an update to the IL. dotnet/runtime#89730 being the first one that came to mind, but there are a few others that might, like dotnet/runtime#94620 and #5556, and I'm sure many more.
I can't really think of an example of
explicit extensionsthat don't fit into the use-case mentioned aboveΒΉ (#7771 (comment)), and because of that I think the conceptual model of "TeacherIdis a type that coincidentally have the exact same underlying data-structure as aGuid" is more useful than "TeacherIdis aGuidwith a different compiler-enforced name" (Which seemingly is what this lowering strategy will give us). While I'm sure the latter has some use-cases, I consider the former to be much more useful.Personally - and this might be taking it to the extreme - I would want to be able to disable implicit upcasting of explicit extensions (say that three times fast..), such that the following code would be illegal (I believe it currently is legal):
I realize this can be achieved through Roslyn Analyzers, but I want the language designers to be aware of this use-case.
Additionally, I hope the following is illegal by default, because if it isn't, then this feature is practically worthless for my use-case:
ΒΉ I am in no way saying there aren't any, just that I can't think of any.
CyrusNajmabadi commentedon Dec 17, 2023
At that point, just wrap the type manually with a new nominal type. This can also be done easily and cheaply
The point of an extension is that it's new functionality, but the same type (just like an extension method). It's just broadening that to all members.
If you actually want me types, that's the easy thing that is already supported today :-)
KennethHoff commentedon Dec 17, 2023
The point is that that brings a lot of baggage, like requiring custom JsonSerializers,
ToStringoverloads, Value Converters (EF Core),IParseableimplementations, and so on..I agree that
implicit extensionsoffers a lot of new useful functionality in this regard, butexplicit extensionsseemingly doesn't (But I'd love to be proven wrong)CyrusNajmabadi commentedon Dec 17, 2023
All that could be done with generators. :-)
If you want distinct types, that's the easy case. The challenge has always been in adding functionality to existing types.
CyrusNajmabadi commentedon Dec 17, 2023
It definitely does. It just doesn't allow things like overloading. But so what? Just name the methods differently. :-)
KennethHoff commentedon Dec 17, 2023
Assuming you still need nominal types to do real differentiation, then I fail to see what
explicit extensionsadds to my repertoire that my previous solution and/orimplicit extensionsdoesn't.HaloFour commentedon Dec 17, 2023
As extensions are themselves nominal types, it seems unfortunate to not be able to treat them as nominal types and to do stuff like overloading on them. Not being able to do so I think makes it awkward to use
explicitextensions as a mechanism for type-safe aliases. Having to use two distinct names and cast to the explicit extension seems quite redundant:The above proposal doesn't mention
implicit/explicitextensions at all, either because that aspect of extensions remains up in the air or because it's assumed that it would not impact encoding. I'd like to posit the argument thatexplicitextensions should further blur the lines between aliases and types, and the encoding should bare that out. Otherwise, I don't understand the purpose of having the extension be a nominal type at all, or to allow types to be declared as that extension in signatures of generic type arguments.agocke commentedon Dec 17, 2023
This discussion has me convinced that people want two different features and are calling both of them "extensions."
If you want the thing called "type classes", then the concept of being able to "overload" on extension in non-sensical. The definition of a type class is to define an interface or signature and then define how a type conforms to that interface. Giving a name to that particular implementation has some meaning, although now you are defining something closer to ML modules, but defining a new type is nonsensical. An implementation is, by definition, not a type. A type class is a type (or constraint bound). To say that a consumer of a type class can overload on implementation definitions is as nonsensical as saying that a method taking an interface gets to decide which interface implementation gets chosen. The whole point of the abstraction is that the caller has the flexibility, not the callee.
If a new type is defined, then the extensions feature isn't really about type classes at all, it's about subtyping. That is, an extension is really just a new type that is inheriting the components of the base type.
While we could have a single base syntax that allows for both features, I'm not sure that's a good idea.
KennethHoff commentedon Dec 17, 2023
I think the term
extensionmake little sense for the concept that I was describing earlier in this conversation. It's not extending anything, it's just repackaging something else, giving it a new name and pretending like they are not the same thing.In my domain,
TeacherId,StudentId, andGuidshould never be used interchangeably. It is, for the purpose of my domain, a coincident that they have the same underlying implementation.I don't think anybody would call what I just described as "an extension to Guid", as I'm more likely to change the
for Guidpart than theextension StudentIdpart in a refactor.I don't know what I would call what I want to achieve howeverΒΉ, but
extensionain't it.Still confused as to what an
explicit extensionis though - if the use-case I've described is invalid - except for having yet another level of obfuscation; Needing both a using directive and a cast operator to use it.ΒΉ Type classes? @agocke mentioned that, but my googling gave me the most abstract answer I've ever seen, so I've no idea if that's it or not.
HaloFour commentedon Dec 17, 2023
@agocke
If these aren't intended to be actual types, it also makes absolutely no sense to be able to define variables or parameters of them either, yet that's exactly how you're expected to use explicit extensions. Given the enormous amount of overlap I don't think it makes any sense to make them two completely different language features either. That feels like making them separate features for the virtue of adhering to some strict textbook definition of "type class" rather than providing solutions to solve problems.
HaloFour commentedon Dec 18, 2023
I think the problem is that "explicit extensions" are already a gray area.
I'm not super versed on typeclasses, what little understanding I have of them is through Scala which probably takes some liberties in order to make them make sense in the JVM. In Scala, the given implementation or witness is not a nominal type and thus can't be used as a nominal type. As long as the given implementation is in scope, the members of the typeclass are available on the underlying type directly, and via generics you'd only ever reference the witness implementation through the typeclass itself. This also appears to be the case with Rust traits and Swift protocols, where the implementation is either only named for the sake of scoping, or not named at all, respectively.
Explicit extensions feel like something completely different in that the witness is a nominal type that you can use as a type in local variables and method signatures. If they aren't a part of the type system, I would argue that it doesn't make sense to allow that. If it's unpalatable to make them a part of the type system because they're called "extensions", then I would argue that maybe we should change "implicit extension" to just "extension" and "explicit extension" to "type".
The recent changes to aliasing in C# generated a lot of feedback from people wanting a way to define strict aliases in the language that behaved like types. It feels like extensions gets us something like 95% of the way there, both in terms of syntax and capabilities.
alrz commentedon Dec 19, 2023
I have to say that I find this confusing:
struct E { private U __this; }why wrap then erase? that looks like a "newtype" imo. Is it only to track the type throughout the code? I can imagine the compiler could do that without actually emitting a wrapper type.This issue is already apparent in the declaration:
public extension TeacherId for Guid;. That example is probably best represented by something liketype TeacherId = Guid;, because no actual "extension" is being defined. For meextensionis like a container for extension members on a particular type aka "extension everything" which would be lowered to simple static methods, and so it wouldn't be useful to be empty.Aside: even with that definition I'm not sure how the extension name would be used, e.g. how to call an instance extension property where it would be otherwise ambiguous? The fact that it can also implement interfaces is where things get conflated in my mind. I'm looking at
impl .. fordeclarations in Rust, those would never need a name because it just adds a "conversion".20 remaining items
hez2010 commentedon Dec 22, 2023
A problem here is that
tc1.GetType()will now returnListDeeplyEquatable<Int32DeeplyEquatable>instead ofList<int>: SharpLab, which makes it non-transparent. We definitely need some runtime support here to make the type (i.e.ListDeeplyEquatable<Int32DeeplyEquatable>) compatible withList<int>while still behaves exactly asList<int>.Ideally, the following things should be supported:
tc1.GetType()yieldsList<int>, notListDeeplyEquatable<Int32DeeplyEquatable>tc1.GetType().GetInterfaces()containsIDeeplyEquatable<List<int>>, notIDeeplyEquatable<ListDeeplyEquatable<Int32DeeplyEquatable>>tc1.GetType().GetMethods()containsDeepEqualsIDeeplyEquatable<List<int>>here is derived fromIDeeplyEquatable<ListDeeplyEquatable<Int32DeeplyEquatable>>ds5678 commentedon Dec 22, 2023
I like the idea of allowing overloads for extensions, but only if the null checking problem is solved.
There could be special casing (similar to
Nullable<T>), but I doubt all the edge cases can be handled.Edit: I'm very excited for the extensions feature. Huge thanks to the LDT for working on it. Happy holidays!
agocke commentedon Dec 22, 2023
@HaloFour Right, so let's dive into type witnesses.
From the previous description of type classes you can see that the module system basically creates a dichotomy between types and classes. Types are data. Classes are behavior (functions). If you start messing with the data, you need a different type. But behavior is fluid, flexible, context-dependent. The entire concept is predicated on the idea that types are fundamental, but how you view a type changes depending on the context. Sometimes a List can be compared with another List, sometimes it can't. Sometimes a List is serializable, sometimes it isn't. You want to be able to swap the available behavior on types in and out, automatically, depending on the context.
This starts to suggest certain implementation decisions. And this is where type witnesses come in. In type theory, a witness just means that the system gains evidence that a certain evaluation is sound, where it might not be so automatically. Implementation-wise, we might as well also carry the implementation of said evaluation in the witness, since we're carrying it around anyway. And this whole thing implies a fundamental design decision: types and classes (read: interfaces) are stored separately. If your function takes an argument with some sort of interface constraint, the type and the interface implementation would be stored separately. That allows the caller to swap out the appropriate implementation based on context. This is what languages like Scala call a "type witness."
The trouble with this approach in C# is, like you've noted, the interface definitions in C# implicitly produce instance invocations. Even though there's nominally an implicit
thisparameter for the function, for a C# type witnessthiswould have the type of the witness, not the original type.The other observation is that the witness doesn't carry any state. As established, state is part of the type. Only the behavior is part of the class. So all witness are function-definitions-only, no state. This is important in two ways. First, it explains my hesitancy at mixing "new type" with "type class." In conventional type class implementations, there's an important divide between the two. Second, it carries some implementation considerations.
When I mentioned that the type witness is an extra argument, one way to do that would be an actual extra function argument. This is the way Scala implements things with implicit parameters. The other way is as a type argument, which is how "type witness" is usually constructed in type theory. Since we know the witness is stateless, there's no need to carry around an extra parameter. We only need type information (dispatch information). You could imagine this as being an empty struct in C#. Consider,
Now we're carrying the witness along as a type argument instead of a function argument. But we still have the problem that the signatures are being defined in terms of the witness type.
So maybe some sort of runtime-transparent wrappers are the answer here. Or maybe some specialty type arguments that let you override the implicit receiver.
But it all still comes down to what problem we're trying to solve, and what kind of module system we want to attach. Some of these implementation choices are perfectly fine if we want to narrowly solve some specific type-class-focused problems. But once the scope starts expanding, potential problems appear quickly.
HaloFour commentedon Dec 22, 2023
Not to mention trying to make it work with a 22 year old ecosystem.
Either way, I think the only real concern I have is what "explicit extensions" are intended to mean, given that they are nominal types in some way. There have been numerous requests for type-safe aliases in the language and it's been suggested that explicit extensions might be a way to accomplish that (if only by me?). Perhaps type erasure doesn't preclude that functionality, but it might create warts. It's not terribly uncommon to run into this with Java and generics, and it's super annoying.
Sergio0694 commentedon Jan 6, 2024
Posting another scenario here too in case anyone is able to share how would something like this (hypothetically speaking) work. I'd be interested in understanding what possible ways could there be to make this work when lowered. Consider this:
And then you'd call it like so:
The part I'm not really following is:
Testhas absolutely no idea about any extension methods for type arguments forT. There is nothing in its signature that indicates any of that: it just has someTtype parameter that's constrained to bothIFooandIBar<T>.TestwithFooas type argument. So I fully expect that myTwill be exactlyFoo. Meaning that eg. ifTestdid anything liketypeof(T), that would returnFoo. Eg. say, if it used that type for caching or whatever.How would that new interface and implementation for
IBar<Foo>be wired up throughTestin cases like this one? π€For context, this is exactly the kind of setup I have in ComputeSharp, and the idea would be that once implicit extensions became a thing, assuming this "static interface implementation through an extension" scenario was possible, I could have my source generator emit these implicit extension types for all user defined types, so that they would then be able to call all APIs constrained to both those interfaces (but users would only manually implement the first one). This is pretty much the same kind of setup that eg. @eiriktsarpalis mentioned he'd like to have in System.Text.Json as well. Just trying to get a better understanding at what would possible lowering strategies for this case look like π
BlinD-HuNTeR commentedon Jan 9, 2024
I suppose one way of doing that could be the following:
struct FooExtensionthat implementsIBar<Foo>Test<T>method, the struct will be used instead. For this to be possible, however, the generated struct must also implementIFoo, so the compiler should generate this implementation with stub methods that forward the call to the original Foo object.typeof(T), it should returnFooinstead ofFooExtension. This would require runtime changes, but I suppose it shouldn't be hard to make the JIT recognize theldtokeninstruction that references an extension type, and generate native code that references the original type instead.Also, since
FooExtensiona struct, it means that any generic types/methods that are called with that struct as type parameter, will have exclusive implementations generated, so there will be no conflict with other extensions, nor with calls involving the original Foo.hez2010 commentedon Jan 9, 2024
This would regress the runtime performance for all
ldtokencalls. Even if we ignore the performance issue here, how would you deal withobject.GetType?There's no way for the compiler to distinguish whether an argument passed into the
Fis an extension wrapper or not.TahirAhmadov commentedon Jan 9, 2024
@hez2010 I think they hope to be able to do some magic which allows this to work at no cost.
However my perspective was that when a method receives
IBar, the only promise there is thatIBarcan be invoked; yet, many others insist on it being possible to simulate the actual instance being passed in, which meansGetTypeand alsoobject.ReferenceEqualsand what not.I'm still not sure what the use case is for this, but oh well - if they can pull it off using "magic", I'm all for it - but in the meantime it does seem to be complicating the feature - I can see how it could have been delivered sooner without this added requirement.
PS. IMO the "pay to play" approach would be new
object.ReferenceEqualsWithExtensionandType.GetTypeWithExtensionmethods...Sergio0694 commentedon Jan 9, 2024
Just to clarify, in my example I was specifically only talking about extensions implementing static interfaces (ie. only static members), because my understanding is that could be simpler to do than having extensions implementing interfaces with instance members as well. Eg. in the case of just static members, you should be able to get all the info you need just via eg. some special generic type context. Not saying this makes it easy, but just saying it should at least be a subset of the more generalized "some interface" case, and possibly (hopefully?) with a simpler implementation π€
BlinD-HuNTeR commentedon Jan 9, 2024
This question involves a broader scope. Basically there are 2 different scenarios for extension with interfaces: one scenario is calling a non-generic method that simply takes the interface as parameter, and the other is calling a generic method with a generic parameter constrained to the interface.
For the first scenario, there are still many questions unanswered. First of all, boxing is unavoidable here, which means that the extension and the underlying instance are going to be two different objects. And if they are different objects, what should
object.GetTypereturn? What aboutReferenceEquals? And also, what should be the behaviour ofisinst/castclassinstructions? Should the receiver be able to transparently check the type of underlying object and cast to it? I suppose that would increase the complexity a little bit.For the generic method scenario, it can be done without boxing, which means that extension and underlying object have the same memory layout. But the behaviour of
object.GetTypeis still to be decided.hamarb123 commentedon Jan 29, 2024
Not sure if it's been asked yet, but is there any reason we're not using custom modifiers to enable overloading by extension type (for
explicit extensions anyway)?->
or similar.
Thaina commentedon Oct 3, 2024
I am not so sure about current use case of
explicit extensionbut I have an idea I wish it could supportI think
explicit extensionshould have conditional checking for casting asis. If it fail the condition it should be nullSuppose we have
Alternative syntax might be
whenand expression blockvladd commentedon Oct 3, 2024
@Thaina
I would expect
LoadFromDB(pid1) as Adultbe null is a person is under 18, but the code like(Adult)LoadFromDB(pid1)should IMHO throw in this case.