1 00:00:00,120 --> 00:00:07,050 In this video we are going to learn kinesis streams using a lambda to process kinesis data streams. 2 00:00:07,840 --> 00:00:13,330 Carney says the themes are very similar with the dynamic DBC teams when it comes to process on Lambda 3 00:00:13,330 --> 00:00:14,080 functions. 4 00:00:14,500 --> 00:00:20,110 We can use a lambda function to process records in Amazon Kinesis data streams. 5 00:00:21,040 --> 00:00:24,460 A Kinesis data stream is a set of charts. 6 00:00:24,490 --> 00:00:27,670 Each chart contains a sequence of data records. 7 00:00:27,760 --> 00:00:32,520 A consumer is an application that processed the data from the clinician's data stream. 8 00:00:32,530 --> 00:00:37,450 We can map a lambda function to consuming this kinase data stream. 9 00:00:38,020 --> 00:00:44,630 So there is also two iterator in the kinesis four standard iterator lambda pulse. 10 00:00:44,650 --> 00:00:50,920 Each chart in our Kinesis data stream for the records using the HTTP protocol, the event source mapping 11 00:00:50,930 --> 00:00:54,280 shares throughput with other consumers of the chart. 12 00:00:54,550 --> 00:00:59,980 Lambda reads the records from the data streams and invokes our function synchronously with an event 13 00:00:59,980 --> 00:01:01,900 that contains stream records. 14 00:01:01,930 --> 00:01:07,150 Lambda reads records in a batch and invoke our function to process record from the batch. 15 00:01:07,180 --> 00:01:14,080 Each batch contains the records from a single chart data stream, so you can see the example Kinesis 16 00:01:14,080 --> 00:01:18,100 record that is triggered to event JSON in the lambda function. 17 00:01:18,100 --> 00:01:24,400 So this is the A including the record array is the same way with the Dynamo DB, but this time the service 18 00:01:24,400 --> 00:01:25,390 is kinesis. 19 00:01:25,480 --> 00:01:32,440 Here you can see the attributes comes from the Kinesis data charts and Kinesis data items. 20 00:01:32,440 --> 00:01:39,880 So basically we will use this Kinesis event source and also you can see the event name and you can see 21 00:01:39,880 --> 00:01:43,330 the other details in the Kinesis data items. 22 00:01:43,510 --> 00:01:47,830 By default, Lambda invokes our function as soon as records are available. 23 00:01:47,860 --> 00:01:53,500 If the batch that Lambda reads from the event source has only one record, Lambda sends only one record 24 00:01:53,500 --> 00:01:54,400 to the function. 25 00:01:54,400 --> 00:01:56,380 In order to avoid invoking function. 26 00:01:56,380 --> 00:02:02,290 With a small number of records, we can tell the event source to buffer records up to 5 minutes by configuring 27 00:02:02,290 --> 00:02:05,260 in the baking window before invoking the function. 28 00:02:05,260 --> 00:02:09,880 Lambda continues to read records from the event source until it has gathered the full batch. 29 00:02:09,880 --> 00:02:14,230 The batch window expires or the batch reached the payload limit. 30 00:02:14,590 --> 00:02:16,900 We can change the batch configurations. 31 00:02:16,900 --> 00:02:23,410 So when it comes to integration from Lambda to Kinesis, we should consider the batch window and batch 32 00:02:23,410 --> 00:02:24,400 configurations. 33 00:02:24,610 --> 00:02:30,730 So how we can configure our data stream and lambda function on our lambda function is a consumer application 34 00:02:30,730 --> 00:02:32,950 for our kinesis data stream. 35 00:02:33,160 --> 00:02:36,970 It is processed one batch of records at a time from the each chart. 36 00:02:36,970 --> 00:02:42,430 So in order to consume kinesis streams from the lambda function, we have two prerequisites. 37 00:02:42,430 --> 00:02:48,970 The first one is required permissions for pulling kinesis data shard from lambda, and the second one 38 00:02:48,970 --> 00:02:52,360 is the create event source mapping between Lambda and the Kinesis. 39 00:02:52,480 --> 00:02:56,800 So let's focus on the first part, which is the required permissions. 40 00:02:56,800 --> 00:03:02,590 We should create the execution role in the lambda function that gives our function permission to access 41 00:03:02,640 --> 00:03:07,090 ADA resources and in our case, Amazon kinase streams. 42 00:03:07,150 --> 00:03:12,310 For that purpose, we will create a role which includes these permissions. 43 00:03:12,310 --> 00:03:14,140 You can see the permissions in here. 44 00:03:14,170 --> 00:03:21,580 Kinesis describes the stream discussed in summary get record, get short iterator list, short list, 45 00:03:21,580 --> 00:03:23,080 stream, and so on. 46 00:03:23,080 --> 00:03:29,260 These are the kinase role permissions that we are going to use in the lambda execution role. 47 00:03:29,260 --> 00:03:36,100 So all these permissions are already predefined policy, including from ADA, which name is ADA Lambda 48 00:03:36,100 --> 00:03:42,250 Kinase execution role that we will attach the policy when we create our lambda function and execution 49 00:03:42,250 --> 00:03:42,700 load. 50 00:03:43,210 --> 00:03:50,170 And also we have second prerequisites, which is the configuring stream from the event source. 51 00:03:50,170 --> 00:03:56,680 That means we will create an event source mapping to tell Lambda to send records from the our kinesis 52 00:03:56,680 --> 00:04:02,650 data stream to lambda function so we can create multiple event sources mapping to process the same data 53 00:04:02,680 --> 00:04:04,210 with multiple lambda functions. 54 00:04:04,210 --> 00:04:09,880 But in our case we will use one lambda function and create event source mapping invocation from lambda 55 00:04:09,880 --> 00:04:11,350 to Kinesis data stream. 56 00:04:11,590 --> 00:04:18,220 In order to configure our function read from the Kinesis, we will go to Lambda Console and we will 57 00:04:18,220 --> 00:04:25,060 add trigger to our lambda function is a kinesis data stream that we did before in the Dynamo DB stream. 58 00:04:25,060 --> 00:04:26,260 It is the same process. 59 00:04:26,260 --> 00:04:35,140 What we have done so far very good when we add a trigger to Lambda is a kinesis stream as an event source, 60 00:04:35,140 --> 00:04:42,760 it is required some of the complications like kinesis, stream name, consumer size, batch window and 61 00:04:42,760 --> 00:04:43,360 so on. 62 00:04:43,360 --> 00:04:50,800 These are the operations do consume volume pulling or pulling from the lambda function for that purpose. 63 00:04:50,800 --> 00:04:56,500 When it comes to pole based communication, we should provide to batch related configurations to get 64 00:04:56,500 --> 00:04:58,450 records from the kinesis streams. 65 00:04:58,450 --> 00:05:01,450 We will leave as it is and continue our operations. 66 00:05:01,480 --> 00:05:07,930 As you can see that we have understood to kinesis streams and how we can use ADA Lambda to process kinesis 67 00:05:07,930 --> 00:05:08,440 streams.