Lambda Invocation from Aurora MySQL: Setup
After creating an instance of Aurora MySQL it must get privileges assigned in order for it to be able to invoke AWS Lambda. Giving Aurora MySQL access to AWS Lambda is described in the following page. I followed that process to the last detail: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/ AuroraMySQL.Integrating.Lambda.html
Lambda: First Invocation from Aurora MySQL
A first invocation of my AWS Lambda function is implemented as simple SQL select statement – mostly for testing. It looks like the following:
SELECT lambda_sync('arn:aws:lambda:us-west-2:<...>: function:fibonacci', '{"fib": 5}') AS 'result';
First Trouble: Input JSON Document Inconsistency
The return value of the select statement is a BLOB and it contains an error message:
{ "body" : "{\"output\":\"Event body is null\", \"input\":\"{\\\"fib\\\":5}\"}", "isBase64Encoded" : false, "statusCode" : "400" }
Now that is curious as the invocation with the exact same payload works when going through the API Gateway:
POST /test/ HTTP/1.1 Host: eflqvnlyoj.execute-api.us-west-2.amazonaws.com content-type: application/json Cache-Control: no-cache Postman-Token: c5102b2c-bd89-4d89-af37-2fd46124a55a { "fib": 5 }
Returns
{ "output": "5", "input": <...> }
Error Analysis
The error message “Event body is null” is coming from my own function implementation. This error message is returned when the function cannot find a property called “fib” in the input payload. It looks like it finds “fib” when invoked through the API Gateway, but it does not find “fib” when invoked through Aurora MySQL. Why is that?
Based on the CloudWatch log the input parameter value (InputStream) to the Lambda function is different when coming from the API Gateway and the Aurora MySQL trigger. The value of the InputStream parameter of the HandleRequest method differs. The difference is as follows.
In case of the API Gateway the InputStream contains a JSON document consisting of the path, path parameters, headers, etc., providing the context of the REST invocation. The actual input payload is contained in a JSON top-level property called “body”.
In case of the invocation from Aurora MySQL the input is the JSON document as defined in the SQL select statement above (without further context like path, path parameters, etc. as it is in context of the API Gateway). Specifically, it is not wrapped within a JSON document with a property “body”.
Disappointing Inconsistency
First of all, this inconsistency is software-engineering-wise disappointing as a part of the function implementation now has to be conditional on the client invoking it. The payload from different clients is structured differently by AWS. Not good.
What should have happened is that the payload should be at the same location, no matter the invocation context or client. In addition, the context should have been qualified and separated out as it would have a different content depending on the client. For example, the InputStream value structure could have been like this:
{ “body”: <payload as sent>, “context”: <a generic context object with a context type> }
The value of the context type could be the kind of possible invocation clients (environments) of functions like “API Gateway”, “Aurora MySQL 5.6”, etc., with details following outlining the specifics of the invoking client. This approach would have allowed a uniform function invocation as well as a uniform function implementation.
Error Resolution
There are different ways to deal with the inconsistent situation. One would be to refactor the function into three parts:
- One function to be used when invoked from the API Gateway
- One function to be used when invoked from the Aurora MySQL
- One “core” function that implements the “raw” functionality and is reused by the previous two functions that are client specific
While this would be a possible approach, this would mean limited reuse of the client specific functions and a proliferation of the number of functions to manage within the AWS Lambda implementation and environment (in my example it would be three instead of one).
My idea, however, is to be able to use the same single function for both clients (in my case). This would be possible if there were a way to check from within the function implementation where the invocation is originating from. Or, if it is possible to probe the InputStream for indications about the invoking client.
Note, there is a context being passed into the function invocation as well (com.amazonaws.services.lambda.runtime.Context), however, this context does not provide a reliable way to distinguish the type of client. It is not fully populated, especially it does not contain a client type designation.
Therefore my chosen solution is
- Probe for “body” (assuming that every API Gateway invocation adds this property to the InputStream)
- If found then it must be a API Gateway call
- If not found then it is a Aurora MySQL invocation
Clearly, this is not a general solution as functions can be invoked from many more types of clients, however, for the current implementation this approach is sufficient. In a complete implementation all possible clients of functions would have to be supported.
Modified Function Implementation
The function implementation now tests for “body”. If “body” is found (API Gateway), then this is extracted to be the “real” input to the function. If “body” is not found (Aurora MySQL), then the value of InputStream is taken as the “real” input.
So far, so good. Now the same function can be used for invocations from the Aurora MySQL trigger as well as an invocation from the API Gateway.
But wait!
More Trouble: Result Output Requirements
There is more trouble. For the API Gateway to work, the output has to contain specific JSON top level properties, and that output would not be the same response to the invocation from Aurora MySQL as it is API Gateway specific. Search for “api gateway function internal server error” and there is a variety of discussions on what has to be included.
So not only the input, but also the output is client specific. Second bummer!
The error resolution for this case (my goal) is to make the output the same for both clients by creating it accordingly. In context of the API Gateway and Aurora MySQL this is possible. The output is as the API Gateway requires it, and the Aurora MySQL trigger will have to process it as well (requiring logic to do so).
In my case a function output example is
{ "isBase64Encoded": false, "body": "{\"output\":\"5\", \"input\":\"{<the whole input is contained here>}\", \"error\":null}", "statusCode": "200" }
This results in the following when the function is invoked via the API Gateway:
{ "output": "5", "input": "{<the whole input is contained here>}\"}", "error": null }
However, at this point it is not clear to me if that would work for all possible AWS Lambda clients, or if AWS actually would force me to have different outputs for different clients.
Even More Trouble: Error Handling Behavior
The error handling behavior is different for API Gateway and the Aurora MySQL as well. If the payload has a JSON syntax error when invoking the function via the API Gateway, the API Gateway will not complain (neither will AWS Lambda) and the error handling is left to the function implementation itself.
When invoking the function via Aurora MySQL lambda_sync() and the payload has a JSON syntax error, the following error message is returned:
Error Code: 1873. Lambda API returned error: Invalid Request Content.
Could not parse request body into json: Unexpected character (';' (code 59)):
was expecting comma to separate Object entries at [Source: [B@413d13c7; line: 1, column: 11]
The payload with the JSON syntax error is not passed on to the function, but caught before the function is being invoked. This means that AWS does some error handling itself before the function invocation and some reaction to the possible errors have to be implemented in the client code in context of Aurora MySQL in addition to the function implementation.
Trouble Keeps On Coming: MySQL JSON Function Support
Aurora MySQL version 5.6 has AWS Lambda invocation support (e.g. the lambda_sync() function), but no MySQL JSON function support. Aurora MySQL 5.7 has JSON function support, but not AWS Lambda invocation support. Go figure.
(As a side note, this is probably the reason why the AWS Lambda invocation examples from Aurora MySQL 5.6 do not show how a function result is processed in a synchronous function invocation).
Destination: Not Reached Yet
With all the trouble to go through and having had to work through the various inconsistencies in behavior, the blog did not reach its final destination to have the same function be used from both, the API Gateway and Aurora MySQL 5.6.
What is left to be done (as it is known to me so far) is functionality to interpret the returned JSON result in context of Aurora MySQL. I probably have to implement a MySQL function that takes the AWS Lambda function result JSON document and picks out the return value.
Once the return value is extracted, it can be further processed. The Aurora MySQL function invocation logic (in my case via MySQL database triggers) will described in the next blog as the second part of the journey.
Summary
In summary, reusing AWS Lambda functions by different AWS technologies that can act as function invocation clients should be straightforward and not require specific coding in the function implementation.
It is unclear why AWS did not ensure this as it would have been probably easier (at least from a user’s perspective) to add additional clients in the future without having to go through extensive testing and probably additional coding.
Another observation is that the various behaviors and constraints are not documented by AWS. The ecosystem is your friend in this context.
Go Serverless!
Disclaimer
The views expressed on this blog are my own and do not necessarily reflect the views of Oracle.
r e a l - p r o g r a m m e r 