1
00:00:00,210 --> 00:00:06,810
In this video, we are going to do hands on up process kinesis streams using a lambda.

2
00:00:07,380 --> 00:00:13,410
So kinesis streams are very similar with dynamic DB streams when it comes to a process from a lambda

3
00:00:13,410 --> 00:00:16,200
function, you can see the image on the slide.

4
00:00:16,200 --> 00:00:18,930
So basically there is client application.

5
00:00:18,930 --> 00:00:24,600
These are the client application, mobile client, traditional server users, Iot device and so on.

6
00:00:24,600 --> 00:00:31,470
All these client application produced the data and these data comes to Amazon Kinesis data streams and

7
00:00:31,470 --> 00:00:38,580
Amazon Kinesis provides the streams and Adobe is lambda received streams with pole based event source

8
00:00:38,580 --> 00:00:43,590
mapping in location according to batch size and run the function itself.

9
00:00:43,680 --> 00:00:48,870
So there are several steps that we can follow when we provide this architecture.

10
00:00:48,900 --> 00:00:53,010
For example, first of all, we will create a kinesis data streams.

11
00:00:53,040 --> 00:00:58,320
After that, we will create a lambda function, and after that we will add required permissions for

12
00:00:58,320 --> 00:01:01,110
pulling kinase charts from the lambda function.

13
00:01:01,290 --> 00:01:06,750
And after that we will create event source mapping, pulling in location type between Lambda and Kinesis.

14
00:01:06,750 --> 00:01:11,280
And after that we will develop our lambda function code with incoming Kinesis records.

15
00:01:11,280 --> 00:01:14,490
And lastly, we will perform end to end test.

16
00:01:14,490 --> 00:01:16,020
So let's get started.

17
00:01:16,230 --> 00:01:21,450
Remember that everyone hands on loop in the serverless project we have two steps.

18
00:01:21,450 --> 00:01:25,110
The first step is creating infrastructure on a cloud environment.

19
00:01:25,110 --> 00:01:29,670
So that means we will create these hands on lab architecture in a cloud environment.

20
00:01:29,670 --> 00:01:35,520
And the second step is developing lambda function code with interacting kinesis data streams.

21
00:01:35,580 --> 00:01:40,260
So now let me start with the first phase, which is the create infrastructure.

22
00:01:40,260 --> 00:01:45,030
So that means we will create kinesis data streams and Lambda functions for that purpose.

23
00:01:45,030 --> 00:01:52,080
Please open the development console, go to Kinesis, see that Kinesis is collect process and analyze

24
00:01:52,080 --> 00:01:53,940
data stream in the real time.

25
00:01:53,940 --> 00:02:01,200
So we will use Kinesis data stream collecting streaming data and click the create data stream button.

26
00:02:01,380 --> 00:02:05,610
So now I'm going to give the stream name, for example, test stream.

27
00:02:06,430 --> 00:02:11,920
And if you scroll down, you can see the cup estimate we select on demand and you can see the capabilities

28
00:02:11,920 --> 00:02:17,680
of these kinesis data stream and the right capacity and capacity in here.

29
00:02:17,710 --> 00:02:20,700
If you scroll down, you can see the data stream settings.

30
00:02:20,710 --> 00:02:26,470
All these settings are coming from the default, which is the on demand we use in the capacity mode

31
00:02:26,530 --> 00:02:30,670
and leave as it is for all configurations and click to create data stream.

32
00:02:31,930 --> 00:02:39,640
So as you can see that we have created Amazon Kinesis data stream and if you scroll down, you can see

33
00:02:39,640 --> 00:02:40,990
the details in here.

34
00:02:41,110 --> 00:02:46,060
And so to becomes active, if you scroll down, you can see the applications and these applications

35
00:02:46,060 --> 00:02:46,930
split by two.

36
00:02:46,960 --> 00:02:51,220
That means there is a producer applications and also consumer applications.

37
00:02:51,220 --> 00:02:57,040
Since we didn't integrate with any kind of data stream, now there is no producer and consumer, but

38
00:02:57,040 --> 00:03:03,130
we can create producers and consumers using the SDK and other tools you can see in here.

39
00:03:03,380 --> 00:03:03,740
Okay.

40
00:03:03,820 --> 00:03:07,950
If you scroll up and see that there is also enhanced fanart.

41
00:03:07,960 --> 00:03:13,030
If you scroll down, you can see that we here specify consumers using enhanced fanart.

42
00:03:13,060 --> 00:03:18,870
Maybe you remember in the last section we saw an article that explained how we can perform find out

43
00:03:18,880 --> 00:03:22,720
architecture with using Amazon Kinesis data streams and lambda function.

44
00:03:23,470 --> 00:03:30,550
After we have successfully define test stream of Amazon Dynamo data streams, now I'm going to create

45
00:03:30,550 --> 00:03:34,420
a lambda function to perform the create infrastructure phase.

46
00:03:34,450 --> 00:03:41,500
Go to lambda function and create function please and give the name of the function as a kinesis function.

47
00:03:43,230 --> 00:03:44,490
And scroll down.

48
00:03:44,760 --> 00:03:50,880
So this time we are going to pull the records from the kinesis before creating with default.

49
00:03:50,910 --> 00:03:57,510
Let me check, is there any predefined policy in here if I search for the kinesis?

50
00:03:58,600 --> 00:04:05,620
There is no any predefined a list policy for now, so please leave as it is and create any role with

51
00:04:05,620 --> 00:04:11,350
basic learner permission and give the name as a stage function and click the create function button.

52
00:04:11,560 --> 00:04:17,470
So this will take some time and after we have created the lambda function we will check several things,

53
00:04:17,920 --> 00:04:22,390
as you remember, that we will follow the steps into the architecture.

54
00:04:22,390 --> 00:04:26,860
We have finished the first two item and continue with the other ones.

55
00:04:26,860 --> 00:04:30,820
So let's go back to our management console and see what happened.

56
00:04:30,940 --> 00:04:31,600
Okay.

57
00:04:31,600 --> 00:04:38,320
We have successfully define kinase function and now I'm going to follow the other steps, which is the

58
00:04:39,040 --> 00:04:43,780
third step, add required permissions to polling kinase charts from the lambda.

59
00:04:43,780 --> 00:04:49,090
So for that purpose, please go to lambda function, go to configuration and go to permissions.

60
00:04:49,090 --> 00:04:52,150
See that we don't have any kinase permission.

61
00:04:52,150 --> 00:04:58,360
I would like to add kinase related permissions in order to receive items from the kinases, please go

62
00:04:58,360 --> 00:05:02,080
to execution role page and go to permissions.

63
00:05:03,800 --> 00:05:06,950
And if you scroll down, you can see the permission policies.

64
00:05:07,040 --> 00:05:14,780
Basically, we will add a policy which required Kinesis rules if you open the or we use to do code.

65
00:05:14,810 --> 00:05:21,530
You can see that these kinesis items and permissions should be required in order to receive item from

66
00:05:21,530 --> 00:05:28,400
the kinesis, from the lambda function and all these permissions, including a Lambda Kinesis execution

67
00:05:28,400 --> 00:05:30,410
rule predefined policy from the ADA.

68
00:05:31,070 --> 00:05:39,050
Please copy this rule and come here, paste in here and you can see that if you click the add permission

69
00:05:39,050 --> 00:05:45,380
and attach the policy, we can search for the attached policy and there is a management permission and

70
00:05:45,380 --> 00:05:48,520
policy which name is a lambda kinase exceptional?

71
00:05:48,530 --> 00:05:54,680
And if you expand this permission, you can see that all the required stream, required kinesis stream

72
00:05:54,680 --> 00:05:58,790
required permissions actions included in this policy.

73
00:05:58,790 --> 00:06:06,950
And we can select this policy and attach the policy into our item role, which is our execution of lambda

74
00:06:06,950 --> 00:06:07,580
function.

75
00:06:07,580 --> 00:06:13,670
If you scroll down, you can see in here and also if you refresh the page of the lambda, we will verify

76
00:06:13,670 --> 00:06:20,990
that the permissions come here and you can see the kinesis and see that all these actions are allowed

77
00:06:20,990 --> 00:06:22,460
into our lambda function.

78
00:06:22,460 --> 00:06:28,130
So that means our lambda function now capable to receive items from the Amazon Kinesis.

79
00:06:28,340 --> 00:06:34,040
But how we can receive items from the Kinesis with pull based invocations.

80
00:06:34,040 --> 00:06:37,160
For that purpose, we will follow the fourth step.

81
00:06:37,160 --> 00:06:42,020
If you come back to our architecture, you can see that we should create event source mapping, pulling

82
00:06:42,020 --> 00:06:44,540
in location type between London Kinesis.

83
00:06:44,780 --> 00:06:49,910
In order to do that, we are going to add a trigger to our Kinesis function.

84
00:06:49,910 --> 00:06:57,020
If you come here and search for the kinesis, you can see that we can select event source as a kinesis

85
00:06:57,020 --> 00:07:01,400
of lambda function and give the kinesis stream name, which is a test stream.

86
00:07:01,490 --> 00:07:04,520
And then if there is a consumer, we can set the consumer.

87
00:07:04,520 --> 00:07:10,490
We don't have consumer and I'm going to scroll down, see that these are the pull based configuration

88
00:07:10,490 --> 00:07:12,500
that we don't change anything.

89
00:07:12,500 --> 00:07:15,200
I'm leaving as it is and click the red button.

90
00:07:15,200 --> 00:07:21,350
So we are successfully create event source mapping communication between Lambda Function and Kinesis.

91
00:07:21,440 --> 00:07:27,410
So if you come back to our architecture, you can see that the last thing we should do that develop

92
00:07:27,410 --> 00:07:30,170
our lambda function code with incoming kinesis record.

93
00:07:30,170 --> 00:07:38,120
So if you go back to our main console, how we can understand kinesis event JSON to trigger lambda function

94
00:07:38,150 --> 00:07:45,800
the easiest way, go to test window of the lambda function and create a new event which is a test kinesis.

95
00:07:47,440 --> 00:07:49,570
And if you scroll down, you can select the template.

96
00:07:49,570 --> 00:07:56,650
If you search for the Kinesis and search for the Kinesis data streams, if you search for the Kinesis

97
00:07:56,680 --> 00:08:03,910
data streams, you can see that the last item is Kinesis data stream and see the example event JSON

98
00:08:03,910 --> 00:08:06,340
object that will trigger the lambda function.

99
00:08:06,340 --> 00:08:13,750
So if you copy all these codes and go to Visual Studio code, I'm going to create one JSON file and

100
00:08:13,750 --> 00:08:15,520
you can copy paste in here.

101
00:08:15,940 --> 00:08:16,360
Okay.

102
00:08:16,360 --> 00:08:16,840
Very good.

103
00:08:16,840 --> 00:08:20,110
So this is the example event object theory.

104
00:08:20,110 --> 00:08:21,850
Go to lambda function again.

105
00:08:21,850 --> 00:08:29,650
There is a records array and every record item has kinesis and these are the attributes and key properties

106
00:08:29,650 --> 00:08:31,990
that we can define about the kinesis.

107
00:08:31,990 --> 00:08:38,410
You can see the event source is a Kinesis event ID event name and event source and these are the Kinesis

108
00:08:38,410 --> 00:08:41,920
detailed attributes that we can see in the UN JSON object.

109
00:08:42,100 --> 00:08:42,940
Okay, very good.

110
00:08:42,940 --> 00:08:48,760
So if you come back to our lambda function now, we can develop our lambda function according to this

111
00:08:48,760 --> 00:08:49,630
test event.

112
00:08:49,630 --> 00:08:55,720
So if you go back to our pseudocode, I'm going to develop our lambda function in the index case.

113
00:08:55,720 --> 00:09:01,780
See that I have created separate lecture and section folder and go to index and see the implementation

114
00:09:01,870 --> 00:09:06,490
I have already implement before because this is very similar with the dynamic DB systems.

115
00:09:06,490 --> 00:09:12,470
First of all, we log incoming event and after that we will iterate the records and get the record into

116
00:09:12,470 --> 00:09:15,010
the our kinesis stream.

117
00:09:15,010 --> 00:09:22,990
But this time the only difference is this record kinase is stored as a base 64 and we need to decode

118
00:09:22,990 --> 00:09:26,560
this record because it comes with the encoded way.

119
00:09:26,560 --> 00:09:34,140
So for example, if you go to record Kinesis data in the incoming event, JSON example called event

120
00:09:34,150 --> 00:09:42,910
is an example go to record Kinesis data, you can see that these value is encoded base 64 value and

121
00:09:42,910 --> 00:09:47,200
it is required to decode when we consume this data.

122
00:09:47,200 --> 00:09:55,180
So that's why if we when we develop this lambda function, we are basically perform the decode operation

123
00:09:55,180 --> 00:09:58,180
according to base 64 and get the payload.

124
00:09:58,180 --> 00:10:00,610
So after that, I'm going to log the payload.

125
00:10:00,640 --> 00:10:05,860
This is very easy business logic, but you can increase or you can change the business logic.

126
00:10:05,860 --> 00:10:08,620
Applying the final pattern as we talked before.

127
00:10:08,650 --> 00:10:09,910
Okay, very good.

128
00:10:09,910 --> 00:10:17,890
Please develop these indexes and copy these codes and go back to our estimation console and go to Lambda,

129
00:10:17,890 --> 00:10:22,750
select the code and remove the indexes and paste in here.

130
00:10:22,750 --> 00:10:23,980
Click the deploy button.

131
00:10:26,070 --> 00:10:31,200
Look, we have successfully deployed our lambda function and now we are ready for the received data

132
00:10:31,200 --> 00:10:32,400
from the Kinesis.

133
00:10:32,580 --> 00:10:38,880
If you open the architecture, as you can see that we have created the Kinesis data streams and create

134
00:10:38,880 --> 00:10:44,400
lambda function and develop our lambda function and create event source mapping connection between kinase

135
00:10:44,400 --> 00:10:45,600
and lambda function.

136
00:10:45,630 --> 00:10:52,560
The only difference with the dynamic DC stream is that Kinesis using Base64 when transferring data into

137
00:10:52,560 --> 00:10:52,940
it.

138
00:10:52,950 --> 00:10:55,650
So let's test the whole process in the next video.