dotnet:8.v88 runtime breaks functions bundling Amazon.Lambda.Core < 3.0.0 (ReflectionTypeLoadException for ILambdaResponseStream)

[jonathanbcsouza]

Describe the bug

The dotnet:8.v88 managed runtime ships Amazon.Lambda.RuntimeSupport 2.1.1.0, which contains ImplLambdaResponseStream : ILambdaResponseStream in ResponseStreamLambdaCoreInitializerIsolated.cs. When a function bundles Amazon.Lambda.Core < 3.0.0 (which dotnet publish does by default), the DLL in /var/task takes assembly resolution precedence over the runtime's copy in /var/runtime. The older DLL lacks ILambdaResponseStream.

Any code that calls Assembly.GetTypes() on the Amazon.Lambda.RuntimeSupport assembly (standard in ASP.NET Core DI, AutoMapper, MediatR) throws ReflectionTypeLoadException at INIT. The function never starts.

GetTypes() fails on the Amazon.Lambda.RuntimeSupport assembly (not Amazon.Lambda.Core) because ImplLambdaResponseStream implements ILambdaResponseStream. .NET tries to resolve the interface from the loaded Amazon.Lambda.Core (the older copy from /var/task), which doesn't have it.

The existing try/catch in LambdaBootstrap.cs:244 protects InitializeCore() but does not protect downstream code scanning RuntimeSupport's types.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

Functions bundling older Amazon.Lambda.Core (< 3.0.0) should continue to initialize successfully on dotnet:8.v88, as they did on dotnet:8.v84. The runtime should be backward compatible with existing deployment packages.

Current Behavior

Functions fail at INIT with:

INIT_START Runtime Version: dotnet:8.v88
INIT_REPORT Init Duration: 443.99 ms  Phase: init  Status: error  Error Type: Runtime.Unknown

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
---> System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types.
Could not load type 'Amazon.Lambda.Core.ResponseStreaming.ILambdaResponseStream'
from assembly 'Amazon.Lambda.Core, Version=1.0.0.0, Culture=neutral, PublicKeyToken=885c28607f98e604'.
   at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
   at Startup.ConfigureServices(IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.UseStartup(...)
   at Microsoft.Extensions.Hosting.HostBuilder.Build()
   at Amazon.Lambda.AspNetCoreServer.AbstractAspNetCoreFunction`2.Start()
   at Amazon.Lambda.RuntimeSupport.Bootstrap.UserCodeLoader.Init(...)  UserCodeLoader.cs:140
   at Amazon.Lambda.RuntimeSupport.LambdaBootstrap.InitializeAsync()   LambdaBootstrap.cs:358

Reproduction Steps

1. Create Function.cs:

using System;
using System.Linq;
using System.Reflection;
using Amazon.Lambda.AspNetCoreServer;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.DependencyInjection;

namespace ReprodFunction;

public class LambdaEntryPoint : APIGatewayProxyFunction
{
    protected override void Init(IWebHostBuilder builder)
    {
        builder.UseStartup<Startup>();
    }
}

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        var allTypes = AppDomain.CurrentDomain.GetAssemblies()
            .SelectMany(a => a.GetTypes())
            .Where(t => t.IsClass && !t.IsAbstract)
            .ToList();
    }

    public void Configure(IApplicationBuilder app, IWebHostEnvironment env) { }
}

2. Create ReprodFunction.csproj:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
    <OutputType>Library</OutputType>
    <NoWarn>NU1605</NoWarn>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Amazon.Lambda.Core" Version="2.2.0" />
    <PackageReference Include="Amazon.Lambda.Serialization.SystemTextJson" Version="2.3.1" />
    <PackageReference Include="Amazon.Lambda.AspNetCoreServer" Version="8.1.0" />
  </ItemGroup>
</Project>

3. Build, zip, deploy, invoke:

dotnet publish -c Release -r linux-x64 --self-contained false -o ./publish
cd publish && zip -r ../repro.zip .

aws lambda create-function \
  --function-name v88-repro-test \
  --runtime dotnet8 \
  --handler "ReprodFunction::ReprodFunction.LambdaEntryPoint::FunctionHandlerAsync" \
  --role <any-lambda-role-arn> \
  --zip-file fileb://repro.zip \
  --region us-east-1

aws lambda invoke --function-name v88-repro-test \
  --cli-binary-format raw-in-base64-out --payload '{}' \
  --region us-east-1 output.json

Result on v88: INIT crash (ReflectionTypeLoadException)
Result on v84 (pinned via put-runtime-management-config): INIT succeeds

Possible Solution

Remove the direct ILambdaResponseStream interface implementation from the Amazon.Lambda.RuntimeSupport assembly metadata. Use reflection-based initialization in ResponseStreamLambdaCoreInitializerIsolated.cs so no type in RuntimeSupport directly references ILambdaResponseStream, preventing GetTypes() from triggering the type load.

Alternatively, move ImplLambdaResponseStream to a separate assembly that is not subject to GetTypes() scanning by customer code.

Additional Information/Context

• PR Add support for Lambda Response Streaming #2288 (Add support for Lambda Response Streaming) introduced the type
• PR Merge response streaming feature branch #2354 (Merge response streaming feature branch) merged it to dev
• dotnet publish bundles Amazon.Lambda.Core.dll by default, so virtually every .NET 8 Zip Lambda has it
• The issue affects any function using ASP.NET Core hosting, AutoMapper, MediatR, or any DI framework that scans types at startup
• Workaround: pin runtime to v84 via Manual runtime management, or update Amazon.Lambda.Core to >= 3.0.0

AWS .NET SDK and/or Package version used

Amazon.Lambda.Core 2.2.0
Amazon.Lambda.AspNetCoreServer 8.1.0
Amazon.Lambda.Serialization.SystemTextJson 2.3.1
Amazon.Lambda.RuntimeSupport 2.1.1.0 (shipped in dotnet:8.v88 managed runtime)

Targeted .NET Platform

.NET 8

Operating System and version

Amazon Linux 2023

[normj]

Thanks for reporting the issue and apologize for the inconvenience. The versioning of Amazon.Lambda.Core has its challenges.

I'm looking through your PR. In the meantime, if you update to the latest version of Amazon.Lamba.AspNetCoreServer that will pull in the version 3.1.0 of Amazon.Lambda.Core and should get you past your issue. Obviously, our ideal solution is existing deployed functions would not be affected when we deploy new versions of the runtime out. We did test with old versions of Amazon.Lambda.Core with new versions of Amazon.Lambda.RuntimeSupport but missed this use case of iterating through the types like this.

[normj]

Out of curiosity what library are you using that triggers this? Looks like AutoMapper would potentially do a scan across all assemblies by default by MediatR only scans what assemblies you tell it to scan.

The challenge with the PR is it requires another deployment of the managed runtimes which by the time that is out user's who ran into this issue would have had to deal with it sooner by updating their dependencies.

[jonathanbcsouza]

In my replication the trigger was a custom functino calling GetAssemblies().SelectMany(a => a.GetTypes()) during Startup.ConfigureServices and not a 3rd party lib.

The PR was an attempt to prevent this class of issue from happening with RuntimeSupport changes... If that doesn't apply, please feel free to close the PR.

[normj]

So to be clear you don't have a function that is down or went down this was a speculation issue? I do appreciate that and the PR gives good tips for the next time we have to add a new type.

[normj]

@jonathanbcsouza I closed your PR because it causes Native AOT issues. I have a new PR out that solves the problem without any new reflection. #2435