In this video we are going to learn kinesis streams using a lambda to process kinesis data streams.
Carney says the themes are very similar with the dynamic DBC teams when it comes to process on Lambda
functions.
We can use a lambda function to process records in Amazon Kinesis data streams.
A Kinesis data stream is a set of charts.
Each chart contains a sequence of data records.
A consumer is an application that processed the data from the clinician's data stream.
We can map a lambda function to consuming this kinase data stream.
So there is also two iterator in the kinesis four standard iterator lambda pulse.
Each chart in our Kinesis data stream for the records using the HTTP protocol, the event source mapping
shares throughput with other consumers of the chart.
Lambda reads the records from the data streams and invokes our function synchronously with an event
that contains stream records.
Lambda reads records in a batch and invoke our function to process record from the batch.
Each batch contains the records from a single chart data stream, so you can see the example Kinesis
record that is triggered to event JSON in the lambda function.
So this is the A including the record array is the same way with the Dynamo DB, but this time the service
is kinesis.
Here you can see the attributes comes from the Kinesis data charts and Kinesis data items.
So basically we will use this Kinesis event source and also you can see the event name and you can see
the other details in the Kinesis data items.
By default, Lambda invokes our function as soon as records are available.
If the batch that Lambda reads from the event source has only one record, Lambda sends only one record
to the function.
In order to avoid invoking function.
With a small number of records, we can tell the event source to buffer records up to 5 minutes by configuring
in the baking window before invoking the function.
Lambda continues to read records from the event source until it has gathered the full batch.
The batch window expires or the batch reached the payload limit.
We can change the batch configurations.
So when it comes to integration from Lambda to Kinesis, we should consider the batch window and batch
configurations.
So how we can configure our data stream and lambda function on our lambda function is a consumer application
for our kinesis data stream.
It is processed one batch of records at a time from the each chart.
So in order to consume kinesis streams from the lambda function, we have two prerequisites.
The first one is required permissions for pulling kinesis data shard from lambda, and the second one
is the create event source mapping between Lambda and the Kinesis.
So let's focus on the first part, which is the required permissions.
We should create the execution role in the lambda function that gives our function permission to access
ADA resources and in our case, Amazon kinase streams.
For that purpose, we will create a role which includes these permissions.
You can see the permissions in here.
Kinesis describes the stream discussed in summary get record, get short iterator list, short list,
stream, and so on.
These are the kinase role permissions that we are going to use in the lambda execution role.
So all these permissions are already predefined policy, including from ADA, which name is ADA Lambda
Kinase execution role that we will attach the policy when we create our lambda function and execution
load.
And also we have second prerequisites, which is the configuring stream from the event source.
That means we will create an event source mapping to tell Lambda to send records from the our kinesis
data stream to lambda function so we can create multiple event sources mapping to process the same data
with multiple lambda functions.
But in our case we will use one lambda function and create event source mapping invocation from lambda
to Kinesis data stream.
In order to configure our function read from the Kinesis, we will go to Lambda Console and we will
add trigger to our lambda function is a kinesis data stream that we did before in the Dynamo DB stream.
It is the same process.
What we have done so far very good when we add a trigger to Lambda is a kinesis stream as an event source,
it is required some of the complications like kinesis, stream name, consumer size, batch window and
so on.
These are the operations do consume volume pulling or pulling from the lambda function for that purpose.
When it comes to pole based communication, we should provide to batch related configurations to get
records from the kinesis streams.
We will leave as it is and continue our operations.
As you can see that we have understood to kinesis streams and how we can use ADA Lambda to process kinesis
streams.