Serverless Computing: MySQL as Function Trigger (Preparation)

In the previous blog an AWS Lambda function is invoked through the API Gateway by a client. How do I invoke (trigger) the same AWS Lambda function by a relational database? This blog sets up MySQL first as preparation, including a database trigger.

Create MySQL RDS Instance

I followed CHAP_GettingStarted.CreatingConnecting.MySQL.html to create an RDS (Relational Database Service) MySQL instance.

In order to connect to it from the MySQL Workbench you need to

  • find the connectivity information on the details page. Look for the “Connect” tile to find the connection endpoint and port
  • use the master username and password and try to connect. However, the connection must fail as no security group is created yet that allows access (I am correctly getting MySQL error 10060)
  • create a second security group and create a rule with custom TCP, port 3306 (in my case) and the IP address from which you are accessing the database (the AWS UI determines that automatically). The rule has to be assigned to the database instance via modification for it to take effect
  • alternatively you could create a rule that allows access by all traffic from anywhere. However, I prefer a more restrictive rule, even though I might have to update it when the IP address changes in the system from where I am accessing the MySQL instance

Now logging in from the MySQL Workbench on your laptop using the master user name and master user password is possible – that is going to make development for MySQL easier.

Create Aurora MySQL Instance

Turns out, creating a MySQL RDS Instance was a wasted effort for the goal that I set myself. It is not possible to call a AWS Lamda function from an MySQL RDS instance. Bummer. However, it is possible from Aurora MySQL.

So, I started over and created an Aurora MySQL. Once I had Aurora MySQL setup and running, I could continue with the MySQL user, table, trigger and function definition specification inside Aurora MySQL. I’ll refer to Aurora MySQL as MySQL for short in the following.

Setup MySQL User

For development I setup a user in the database instance as follows:

  • CREATE USER ‘cbmysqluser’@’%’ IDENTIFIED BY ‘cbmysqluser’;
  • GRANT CREATE ON fib.* TO ‘cbmysqluser’@’%’;

Additional commands are necessary during development to provide additional permissions. Here is the collection I ended up with over time. You might not have to use every single one of those, but anyway. Here is the list of grants

  • GRANT DROP ON fib.* TO ‘cbmysqluser’@’%’;
  • GRANT SELECT ON fib.* TO ‘cbmysqluser’@’%’;
  • GRANT INSERT ON fib.* TO ‘cbmysqluser’@’%’;
  • GRANT TRIGGER ON fib.* TO ‘cbmysqluser’@’%’;
  • GRANT CREATE ROUTINE ON fib.* TO ‘cbmysqluser’@’%’;
  • GRANT ALTER ROUTINE ON fib.* TO ‘cbmysqluser’@’%’;
  • GRANT EXECUTE ON fib.* TO ‘cbmysqluser’@’%’;
  • GRANT UPDATE ON fib.* TO ‘cbmysqluser’@’%’;
  • GRANT DELETE ON fib.* TO ‘cbmysqluser’@’%’;

Here some revocations

  • REVOKE CREATE ON *.* FROM ‘cbmysqluser’@’%’;
  • REVOKE ALL PRIVILEGES, GRANT OPTION FROM ‘cbmysqluser’@’%’;

And some management commands

  • SHOW GRANTS FOR ‘cbmysqluser’@’%’;
  • FLUSH PRIVILEGES;

Create MySQL Table

Having the user setup, now a table can be created by that user.

The idea is to use the table as the invocation interface. A value is inserted into a column (representing a function parameter), and after the function is executed the result is stored in another column into the same row. Each row therefore has the function parameter value as well as the corresponding return value.

In my case the function is the Fibonacci function. For example, calling it with 0 will return 0. Calling it with 10 will return with 55.

| fib | value |
+-----+-------+
|   0 |     0 |
|  10 |    55 |

Here the table creation statement

CREATE TABLE fibonacci (
  fib INT,
  value INT
);

Create MySQL Trigger

Now moving on to the trigger. I split the trigger into the trigger itself, and a separate function. The rigger, when fired, passes the input parameter to the function and stores the value computed by the function into the table. The trigger is specified as follows

DELIMITER $$
CREATE TRIGGER fibcomp 
  BEFORE INSERT ON fibonacci
  FOR EACH ROW
    BEGIN
      SET NEW.value = fibcomp(NEW.fib);
    END$$
  DELIMITER ;

The function computes the Fibonacci number as follows

DELIMITER $$
CREATE FUNCTION fibcomp (fib INT) 
  RETURNS INT 
BEGIN
  DECLARE f1 INT DEFAULT 0;
  DECLARE f2 INT DEFAULT 1;
  DECLARE sum INT;
  DECLARE i INT DEFAULT 2; 
  
  IF fib <= 0 THEN RETURN f1;
  ELSE
    WHILE i <= fib DO
      SET i = i + 1;
      SET sum = f1 + f2;
      SET f1 = f2;
      SET f2 = sum;
    END WHILE;
    RETURN f2;
  END IF;
END $$
DELIMITER ;

When issuing the following two insert statements, the above table content is the result

INSERT INTO fibonacci VALUES (0, null);
INSERT INTO fibonacci VALUES (10, null);

At this point I have the whole infrastructure available in context of MySQL and the functionality in place. This is the basis for incorporating the AWS Lambda implementation of the function implementing the Fibonacci computation next.

Summary

Once it was clear that Aurora MySQL is required, the setup of a database user, table, trigger and function was easy as this is done within MySQL not using any additional AWS infrastructure. With the ability to connect to the database instance using MySQL Workbench the development environment familiar to me was available, and that’s great.

Go Serverless!

Disclaimer

The views expressed on this blog are my own and do not necessarily reflect the views of Oracle.

Advertisements

Serverless Computing: My First Lambda Function

This blog describes to some level of detail how I created my first AWS Lambda function (https://aws.amazon.com/lambda/).

Overview

You are about to embark on a steep learning curve. Defining and executing a AWS Lambda function (in short: function) is not only typing in the function specification, its implementation and then invoking it. A lot more is involved – the below blog provides a general flow (not a command-by-command tutorial – for that several references are provided).

From a high level, the architecture looks as follows:

+--------+  +-------------+  +-----------------+  +----------------+
| Client |->| Api Gateway |->| Lambda Function |->| Implementation |
+--------+  +-------------+  +-----------------+  +----------------+
                  |                  |
            +--------------------------------+
            | Identity and Access Management |
            +--------------------------------+

A client invoking a function does so via the API Gateway. For the function to be executed, its implementation has to be provided. When using Java for the implementation, the implementation has to be uploaded in a specific packaging. Identity and Access Management (IAM) governs various access points from a security perspective.

Super briefly, as a summary, in order for a function invocation to be successful the following has to be in place (a lot of moving parts):

User

  • needs to know API Gateway URL (it is shown when selecting a stage within the Stages link on the API Gateway page)
  • Must have an access key and secret key for key-based authentication (configured in IAM)

Api Gateway

  • API definition as well as resource specification

Lambda Function

  • Function specification
  • Function implementation uploaded
  • Function policy allowing API Gateway access

Identity and Access Management (IAM)

  • User, group and role definition
  • Access policy definition assigned to invoking user (directly or indirectly) for API gateway

Aside from the function implementation everything can be specified on the AWS web page. The function implementation is uploaded by means of a defined packaging.

AWS Account

All specification and definition activity takes place in context of an AWS account. If you don’t have one then you need to create one. Chances are you purchased an item on Amazon before; in this case you have an AWS account already.

Identity and Access Management (IAM)

Initially I setup two users. One called “apiDev”, and a regular user.

Then I created two groups “apiDevelopers” and “apiUsers”. apiDevelopers has the policy AdministratorAccess assigned. This allows apiDev to create all artifacts necessary to implement and to invoke a function. I logged in as apiDev for creating the function and all necessary artifacts.

The group apiUsers has no policy assigned initially, however, it will get a (function execution) policy assigned that is going to be specifically created in order to access the function. This establishes a fine-grained permissions allowing the users of the group to execute the function.

Function Definition

The function definition is separate from the function implementation. A function is created without it having an implementation necessarily at the same time. In my case I am using Java and the implementation has to be uploaded in a specific packaging format; and that upload is distinct from specifying the function in AWS Lambda.

A function definition consists of a name, the selection which language (runtime) is going to be used as well as an execution role. The latter is necessary for a function to write e.g. into the Amazon CloudWatch logs. However, a function specification does not include the function parameters or return values. A function specification does not contain its signature. Any input/output signature specification is absent and only the code will contain the authoritative information.

The phrase “creating a function” can therefore refer to different activities, e.g., just the function specification in AWS Lambda, or including its implementation via e.g. an upload.

The instructions for creating the function and its implementation is here: https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-create-api-as-simple-proxy-for-lambda.html. I chose the execution role lambda_basic_execution.

As a side note, AWS Lambda is different from anonymous lambda functions (https://en.wikipedia.org/wiki/Anonymous_function).

Function Implementation

Being an Intellij user I created a separate project for implementing the function. It turns out the easiest way approaching the function implementation was to create a gradle project from scratch using the Intellij project creation option for gradle, and then fill in the AWS function implementation (rather starting with the function implementation and trying to turn it into a gradle project afterwards).

Once the function is developed it has to be uploaded to AWS Lambda in form of a specific packaging. The process of creating the corresponding zip file is here: https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-create-api-as-simple-proxy-for-lambda.html#api-gateway-proxy-integration-lambda-function-java and here: https://docs.aws.amazon.com/lambda/latest/dg/lambda-java-how-to-create-deployment-package.html.

The upload only happens when pressing the “save” button on the AWS Lambda page and it’ll take a while as the package tends to be several GB. Once uploaded one or more tests can be defined on the web page and executed. While this is not a practical unit test approach, it allows to execute the function without an API Gateway integration in place.

After the function implementation (I choose to implement a function computing Fibonacci numbers) the AWS Lambda user interface looks like this:

Note: this screen dump was taken after I integrated the function with the API Gateway; therefore the API Gateway trigger it is displayed in the UI.

API Gateway

One way invoking (“triggering”) a function is via the API Gateway. This requires the specification of an API and creating a reference to the function. The simplest option is using the proxy integration that forwards the invocation from the API Gateway to the function (and its implementation).

The instructions for creating the API in the API Gateway and its implementation is here: https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-create-api-as-simple-proxy-for-lambda.html. I chose the Lambda Function Proxy integration.

An API specification must be deployed into a stage in order for it to be accessible. Stages are a mechanism to e.g. implement the phases of development, testing, or production.

Analogous to AWS Lambda, the API Gateway also allows direct testing within the web user interface and this can be used for some initial testing (but is not feasible for integration testing as it is manual).

Once the relationship between the API definition in the API Gateway and the function is established via resource specifications, the function can be invoked from external to Amazon using e.g. Postman.

After the implementation the API Gateway user interface looks like this:

Note on security: by default the API is not secured, meaning, everybody who knows the URL is free to call it and invoke the associated function.

Securing the Function

There are two different locations in the invocation chain that require security consideration and policy setup:

  • First, the API in the API Gateway needs to be protected
  • Second, the function in AWS Lambda needs to be protected

Securing an API is outlined here:

https://docs.aws.amazon.com/apigateway/latest/developerguide/permissions.html This differentiates accessing the API definition and invoking an API.

For invoking an API I created a policy and attached it to the group apiUsers. Any user within this group is allowed to invoke the API that I created. In addition, I set the authorization to AWS_IAM (see above figure) and that means that the invoking client has to provide the access and secret key when invoking the API.

The function in AWS Lambda is secured using a function policy that can be seen when clicking on the symbol with the key in the AWS Lambda user interface. In my case is states that the API Gateway can access the function when invoked through a specific API.

Note (repeat from earlier): when defining an API in the API Gateway access is open, aka, anybody knowing the URL can execute the function behind the API. While the URL contains the API identifier (and that is randomly generated) and highly unlikely to be guessed, still, access is open.

Once the access policy is defined and put in place, access will be limited according to the policy. However, access restriction is not immediate, it takes some (short) time to become effective.

Function Invocation

In order to invoke the function, a client (in my case Postman) requires the URL. The URL can be found when clicking a stage in the Stages link in the API Gateway UI.

I opted for the IAM authorization using access key and secret key. That needs to be configured in the authorization setting of Postman (it also requires the AWS Region to be specified). No additional headers are required.

As I have defined a POST method, the payload has to be added as well. In my case this is a simple JSON document with one property.

POST /test/ HTTP/1.1
Host: <API URL>
content-type: application/json
Cache-Control: no-cache

{
"fib": 7
}

Once the invocation is set up, and once a few invocations took place, the API Gateway Dashboard will show the number of invocations, errors, etc., separated for API Gateway as well as Lambda functions.

Summary

Defining the first function is an effort as many pieces have to fall in place correctly for it to work out and many mistakes will happen most likely along the way. However, the ecosystem is quite large and has many questions already answered; in addition, AWS has a lot of documentation, which is mostly accurate (but not quite 100%).

The first function, as defined above, now gives me a jump-off platform to investigate and to experience AWS Lambda functions further. Stay tuned for many more blogs that explore a huge variety of aspects and concepts of serverless distributed computing.

Go Serverless!

Disclaimer

The views expressed on this blog are my own and do not necessarily reflect the views of Oracle.

 

Serverless Computing: What is it?

In a nutshell: “Just upload code and execute it”.

Serverless Computing

From my viewpoint as an engineer, serverless computing means that I can implement as well as use cloud functionality without having to establish and to manage server deployments.

A quote from Amazon states: “Serverless computing allows you to build and run applications and services without thinking about servers. Serverless applications don’t require you to provision, scale, and manage any servers. You can build them for nearly any type of application or back-end service, and everything required to run and scale your application with high availability is handled for you” (https://aws.amazon.com/serverless/).

For example, if business logic has to be executed, I develop one or more functions (procedures), deploy those “into a serverless cloud” and invoke them without having to worry, for example, about initialization, containers, web or application servers, deployment descriptions or scaling.

Or, when I need database access, I create a database service instance “in the cloud” and use it. Of course, I have to possibly (maybe – maybe not) reason about capacity, but I don’t have to find hardware, find, install and maintain database software images, scale the instances, and so on.

There are many explanations and discussions of serverless computing, for example, https://martinfowler.com/articles/serverless.html or https://martinfowler.com/bliki/Serverless.html, among many others. A clear-cut technical definition of “serverless computing” is still missing.

Serverless Computing: Why is it Interesting?

There are many reasons why serverless computing is interesting and appealing. Cost is one factor, hardware utilization is another. The above references outline many, many more.

However, from a software or service engineering perspective there are many important reasons why this relatively new concept is very much worth exploring and considering for future development and possibly migration from traditional approaches like, for example, Kubernetes.

For one, the focus and the effort on managing server deployments and hardware environments is extremely reduced or removed altogether, including aspects like scaling or failure recovery. This frees up engineering, QA, dev ops and ops time and resources to focus on the core business functionality development.

More importantly, a serverless development and execution environment restricts the software architecture, implementation, testing and deployment in major ways. This reduction in variance allows execution optimization and enables a significant increase in development quality and dependability.

Serverless Computing: What are the Choices?

As in the case of any new major development, several providers put forward their specific implementations, and this is the case for serverless computing as well. I don’t attempt to provide a comprehensive list here, but instead refer to the following page as one example that collected some providers: http://www.nkode.io/2017/09/12/serverless-frameworks.html.

One observation is that there are vendor specific as well as vendor unspecific implementations of serverless computing. It is in the eye of the beholder (based on use cases and requirements) to determine the most applicable environment.

Serverless Computing: Changing Planets

It is easy to label serverless computing as “The Next Big Thing”. However, I believe that a real fundamental shift is taking place that in a major ways breaks with the historical development and engineering of distributed computing. Remote procedure call (RPC – https://en.wikipedia.org/wiki/Remote_procedure_call), Distributed Computing Environment (DCE – https://en.wikipedia.org/wiki/Distributed_Computing_Environment), CORBA (https://en.wikipedia.org/wiki/Common_Object_Request_Broker_Architecture), REST (https://en.wikipedia.org/wiki/Representational_state_transfer), just to name a few, have all in common that the design and engineering has to take hardware and resource topology and deployment into consideration, including scaling, and recovery. Furthermore, it had to reason about “local” and “remote”.

Serverless computing, as it currently is implemented, takes away the “distribution” and deployment aspect to very large extent or even completely. This will change significantly how software engineering approaches system and service construction. Time will tell the real impact, of course, but I’d expect major shifts from a software engineering perspective.

What’s Next: Journey Ahead

From a very pragmatic viewpoint, serverless computing is an alternative software development and execution environment for (distributed) services. And this defines the journey ahead: figuring out how the various aspects of software engineering are realized, like for example

  • Function implementation
  • Procedure implementation
  • Error handling, exception handling and logging
  • Failure recovery
  • Scaling (out and in)

The blogs following this will explore these and other aspects over time.

Summary

Serverless computing, and serverless functions in particular, are very appealing developments, especially from the viewpoint of software development/engineering as well as scaleable execution.

Let’s get hands-on and see what the upside potentials are as well as where the limits and issues lie.

Go Serverless!

Disclaimer

The views expressed on this blog are my own and do not necessarily reflect the views of Oracle.