Skip to content

EnvironmentVariablesConfigurationProvider should support all types of connectionstrings Β #36123

Closed

Description

@onybo

When setting a connection string on an Azure Webapp using Azure CLI the type must be specified.
The following types are supported:
ApiHub, Custom, DocDb, EventHub, MySql, NotificationHub, PostgreSQL, RedisCache, SQLAzure, SQLServer, ServiceBus
az webapp config connection-string set

It seems that Microsoft.Extensions.Configuration.EnvironmentVariables.EnvironmentVariablesConfigurationProvider only support four of these: MySql, SQLAzure, SQLServer and Custom

This means that if I use Configuration.GetConnectionString("MyServiceBusConnectionString") and I have added a connection string of type ServiceBus on my Azure Website with the same name the code will unexpectedly fail. I will have to define the type of the connectionstring in azure to be Custom for this to work.

It would be much better if EnvironmentVariablesConfigurationProvider and the method AzureEnvToAppEnv supported the same types as the Azure CLI does.

These are the environment variable prefixes that I think should be added:
APIHUBCONNSTR_
DOCDBCONNSTR_
EVENTHUBCONNSTR_
NOTIFICATIONHUBCONNSTR_
POSTGRESQLCONNSTR_
REDISCACHECONNSTR_
SERVICEBUSCONNSTR_

I would be happy to make a PR for this should you agree that this is something that should be supported.

Activity

  1. Tratcher commented on May 9, 2019

    @Tratcher
    Member

    Wow, that layering is really bizarre. I'd have expected there to be a library that read from config and dealt with this, not having it embedded in the provider itself.

  2. poke commented on May 28, 2019

    @poke
    Contributor

    Do we actually use those provider names for anything? I did a quick search across Extensions, AspNetCore, and EF Core, and outside of configuration tests, the _ProviderName config doesn’t appear to be used. What are those for?
    And why would this configuration provider decide about what database provider I use? For example, it’s just by chance that MySqlConnector uses the same namespace for their MySQL provider (for migration and compatibility reasons). But they didn’t have to do it like that.

  3. divega commented on Sep 25, 2019

    @divega
    Contributor

    Adding a note because we got a report via email that a customer hit the fact that we don't read the connection string correctly for PostgreSQL when it is configured in app services. If this issue was fixed, the simple code they used would work as they expect and they would get the connection string from the value configured in Azure App Services:

    var cs = Configuration.GetConnectionString("MyDatabase");

    Possibly the simplest workaround is to use the "custom" option instead.

    To answer to previous comments:

    @poke No framework code that I am aware currently uses _ProviderName , but applications can do it if the wish to. For example an application could support using different kinds of databases with the same code and resolve the DbProviderFactory from DbProviderFactories using the invariant name.

    @Tratcher I am not sure I like the layering either but the logic needs to go somewhere. My main complaint is that the logic is bound to become stale anytime Azure App Services adds new connection string types. Ideally they would publish a package that encapsulates the logic. But this way we own it and it is easier for us to make it right.

    @bradygaster, I was told by @glennc that you will be following up on this. I will add you to the email thread about the customer issue.

  4. Tratcher commented on Sep 25, 2019

    @Tratcher
    Member

    @divega can we at least look at properly re-layering this in 5.x? E.g. move the connection strings out of the environment variable provider to their own API that consumes config and is called from the templates. At the very least it would be easier to maintain and ship an out of band leaf node dependency.

  5. divega commented on Sep 25, 2019

    @divega
    Contributor

    @Tratcher I think you can make a proposal, check with @bradygaster, @DamianEdwards that the resulting experience (and possible breaking change) is worth it.

  6. DamianEdwards commented on Oct 1, 2019

    @DamianEdwards
    Member

    Eeeek, having strings that are hard-coded to an Azure service's set of supported things in this provider is making me feel poorly. We shouldn't definitely look at this in 5.0

    @davidfowl @glennc

  7. bradygaster commented on Oct 3, 2019

    @bradygaster
    Member

    absolutely agree with @DamianEdwards on this - I don't see those prefixes adding value, and have been confused by their existence for some time. I'd like to re-evaluate this, and hopefully find a way to stop doing it. IMO, a "connection string is a connection string," and shouldn't contain any "triggering behavior" for the consumer or configurator. Screen shot from the AZ CLI below for reference. Adding @btardif here from App Service team as he's on the email thread in which we're discussing this internally. @btardif is this a requirement of the web apps side or something imposed by the CLI side?

    Al - is there any reason why we can't agree "a connection string is a connection string" and figure out if unwinding the prefixes is possible?

  8. davidfowl commented on Oct 3, 2019

    @davidfowl
    Member

    I believe the original reason for the prefix was because those got projected into the .NET Framework configuration system as provider specific connection strings.

    <connectionStrings>
        <add name="ConnStringDb1" connectionString="Data Source=localhost;Initial Catalog=YourDataBaseName;Integrated Security=True;" providerName="System.Data.SqlClient" />
    </connectionStrings>

    There's no first class concept like that in .NET Core so we mapped it internally (in a hacky way) to ConnectionString:{Name} or ConnectionString:{Name}_{Provider}. So the options here are to:

    • Change AppService to match our conventions or add our conventions to the list natively when connection strings appear.
    • Create an app service specific configuration provider and have users add that (or light it up on app service)
    • Keep hacking and add more prefixes
  9. roji commented on Oct 3, 2019

    @roji
    Member

    Just to mention the point discussed offline, having a provider classification on a connection string could allow us to do interesting things in the UI, e.g. a provider-specific UI for building the string, validation, or whatever. There's definitely value in that, especially for people starting out and not familiar with what options are supported in the connection string of which provider, etc.

  10. khellang commented on Mar 12, 2020

    @khellang
    Member

    Wow, this was a bizarre experience.

    I created a stock "Web App with Postgres" in Azure and deployed my application, only to spend hours trying to debug why the app wasn't picking up my connection string (which had been automatically added to the web app using the "PostgreSQL" type).

    That lead me to eventually find this issue and switch the connection string type to "Custom" instead, and suddenly the app worked fine. What a mess 😞

  11. tnc1997 commented on May 8, 2020

    @tnc1997

    I would like to see this issue addressed if possible, or perhaps an explicit warning for those of us who select an unsupported connection string type such as PostgreSQL if that'd be easier.

  12. transferred this issue fromdotnet/extensionson May 8, 2020
  13. 34 remaining items

  14. pk-ot commented on Jan 28, 2024

    @pk-ot

    Fact: The auto-generated (semantically correct) configuration by the Azure Web App wizard that creates a Postgres database and web app in one step is just not working.

    Why the template for that wizard is not simply forcing a type of "Custom" for the connection string and adds a short note somewhere on this known issue, is beyond me. Either nobody of the responsibles has ever tested this with even just a "hello world"-type of application, or they have and simply don't care. Microsoft rather lets everyone figure it out by themselves (since 2019!). Losing hours to a problem like this is frustrating.

    But hey, now it's scheduled for .NET 9. πŸ™

  15. sqbiak commented on Nov 23, 2024

    @sqbiak

    Hello from a 2024.... from a S&P 500 where I wanted to fire a guy for this code:

            string connectionString = null;
            IDictionary environmentVariables = Environment.GetEnvironmentVariables();
            foreach (DictionaryEntry envVar in environmentVariables)
            {
                if (envVar.Key.ToString().IndexOf("POSTGRESQLCONNSTR_x", StringComparison.OrdinalIgnoreCase) >= 0)
                {
                    connectionString = envVar.Value.ToString();
                    break;
                }
            }
    

    but then a simple

             builder.Services.AddDbContext<DatabaseContext>(options =>
             {
                 var connectionString = builder.Configuration.GetConnectionString("POSTGRESQLCONNSTR_x");
                 if (string.IsNullOrEmpty(connectionString))
                 {
                     throw new InvalidOperationException("Database connection string 'POSTGRESQLCONNSTR_x' not found.");
                 }
                 options.UseNpgsql(connectionString);
             }); 
    

    didn't work.

  16. am11 commented on Nov 23, 2024

    @am11
    Member

    var connectionString = builder.Configuration.GetConnectionString("POSTGRESQLCONNSTR_x");

    nit: with GetConnectionString() API, it should be just "x"; the full form"POSTGRESQLCONNSTR_x" can be used with indexer ([]).

    Meanwhile, we can inherit from EnvironmentVariablesConfigurationProvider and add the missing prefixes.

    Source
    using System;
    using System.Collections;
    using Microsoft.Extensions.Configuration;
    using Microsoft.Extensions.Configuration.EnvironmentVariables;
    
    internal static class CustomEnvironmentVariablesExtensions
    {
        public static IConfigurationBuilder AddCustomEnvironmentVariables(this IConfigurationBuilder builder, string? prefix = null)
        {
            return builder.Add(new CustomEnvironmentVariablesConfigurationProvider.Source(prefix));
        }
    }
    
    file class CustomEnvironmentVariablesConfigurationProvider(string? prefix) :
        EnvironmentVariablesConfigurationProvider(prefix)
    {
        private const string PostgreSqlPrefix = "POSTGRESQLCONNSTR_";
        private const string NotificationHubPrefix = "NOTIFICATIONHUBCONNSTR_";
        private const string ServiceBusPrefix = "SERVICEBUSCONNSTR_";
        private const string EventHubPrefix = "EVENTHUBCONNSTR_";
        private const string DocumentDbPrefix = "DOCDBCONNSTR_";
        private const string RedisCachePrefix = "REDISCACHECONNSTR_";
    
        public override void Load()
        {
            base.Load();
    
            LoadAdditionalPrefixes(Environment.GetEnvironmentVariables());
        }
    
        private void LoadAdditionalPrefixes(IDictionary envVariables)
        {
            foreach (DictionaryEntry entry in envVariables)
            {
                string key = (string)entry.Key;
                string? value = (string?)entry.Value;
    
                if (key.StartsWith(PostgreSqlPrefix, StringComparison.OrdinalIgnoreCase))
                {
                    HandleConnectionStringPrefix(PostgreSqlPrefix, key, value, "Npgsql");
                }
                else if (key.StartsWith(NotificationHubPrefix, StringComparison.OrdinalIgnoreCase))
                {
                    HandleConnectionStringPrefix(NotificationHubPrefix, key, value);
                }
                else if (key.StartsWith(ServiceBusPrefix, StringComparison.OrdinalIgnoreCase))
                {
                    HandleConnectionStringPrefix(ServiceBusPrefix, key, value);
                }
                else if (key.StartsWith(EventHubPrefix, StringComparison.OrdinalIgnoreCase))
                {
                    HandleConnectionStringPrefix(EventHubPrefix, key, value);
                }
                else if (key.StartsWith(DocumentDbPrefix, StringComparison.OrdinalIgnoreCase))
                {
                    HandleConnectionStringPrefix(DocumentDbPrefix, key, value);
                }
                else if (key.StartsWith(RedisCachePrefix, StringComparison.OrdinalIgnoreCase))
                {
                    HandleConnectionStringPrefix(RedisCachePrefix, key, value);
                }
            }
        }
    
        private void HandleConnectionStringPrefix(string prefix, string fullKey, string? value, string? provider = null)
        {
            string normalizedKey = Normalize(fullKey.Substring(prefix.Length));
    
            Data[$"ConnectionStrings:{normalizedKey}"] = value;
    
            if (provider is not null)
            {
                Data[$"ConnectionStrings:{normalizedKey}_ProviderName"] = provider;
            }
        }
    
        private static string Normalize(string key) => key.Replace("__", ConfigurationPath.KeyDelimiter);
    
        internal class Source(string? prefix) : IConfigurationSource
        {
            public IConfigurationProvider Build(IConfigurationBuilder builder)
            {
                return new CustomEnvironmentVariablesConfigurationProvider(prefix);
            }
        }
    
    }

    usage:

    using System;
    using Microsoft.Extensions.Configuration;
    
    IConfigurationRoot configuration = new ConfigurationBuilder().AddCustomEnvironmentVariables().Build();
    
    Console.WriteLine($"PostgreSQL Connection String with GetConnectionString: {configuration.GetConnectionString("x")}");
    Console.WriteLine($"PostgreSQL Connection String with indexer: {configuration["POSTGRESQLCONNSTR_x"]}");

    run with dotnet 9.0:

    $ export POSTGRESQLCONNSTR_x="Server=myServer;Database=myDb;User=myUser;Password=myPass;"
    $ dotnet add package Microsoft.Extensions.Configuration.EnvironmentVariables
    $ dotnet run
    PostgreSQL Connection String with GetConnectionString: Server=myServer;Database=myDb;User=myUser;Password=myPass;
    PostgreSQL Connection String with indexer: Server=myServer;Database=myDb;User=myUser;Password=myPass;
  17. cliedeman commented on Nov 29, 2024

    @cliedeman

    Just ran into this using Cosmos db with the Document Db type

  18. SebastianToet commented on Feb 18, 2025

    @SebastianToet

    Wow, just spent half a day trying to pinpoint this issue for the "Redis Cache" connection string type in Azure App Services.
    Pretty frustrating that now for more than 5 years so many people must be running into this!

  19. spaasis commented on Apr 24, 2025

    @spaasis

    Six years later...

  20. txnnr commented on Apr 25, 2025

    @txnnr

    ran into this issue using PostgreSQL as well

  21. trullock commented on Apr 25, 2025

    @trullock

    I love being subscribed to this issue still 6 years later, knowing how much time i wasted on this, seeing how noone cares, and being glad i migrated away from this diabolical stack to other platforms with their own cryptic and less documented quirks that make me hate them equally

  22. Estyn commented on Apr 25, 2025

    @Estyn

    We also just ran into this issue.

  23. davidfowl commented on May 2, 2025

    @davidfowl
    Member

    OK lets just add them, we can hopefully remove them in the long run. This is a VERY unsavory hack, but maybe we just embrace it at this point.

  24. ericstj commented on May 3, 2025

    @ericstj
    Member

    So long as we change in sync, I guess it's OK. Who adds the upstream usage of these? Can we drop a breadcrumb there to remind them to update this library whenever something new is added?

  25. added
    in-prThere is an active PR which will close this issue when it is merged
    on May 27, 2025
  26. tarekgh commented on May 27, 2025

    @tarekgh
    Member

    I have opened the PR #116037 if anyone interested to review.

  27. locked and limited conversation to collaborators on Jul 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions