1 00:00:00,090 --> 00:00:06,590 In this video, we are going to learn how a list functions orchestrate distributed transactions with 2 00:00:06,600 --> 00:00:07,800 using Python. 3 00:00:08,430 --> 00:00:14,130 I will give some examples an implementation with self function and showing these architecture diagram 4 00:00:14,130 --> 00:00:14,870 on the slide. 5 00:00:14,880 --> 00:00:21,030 So basically you can think about that when we place an order to order, fulfillment process is starting. 6 00:00:21,120 --> 00:00:26,040 So think that we have to microservices, order and fulfillment microservices. 7 00:00:26,160 --> 00:00:28,750 When we place an order order microservices. 8 00:00:28,780 --> 00:00:33,120 Note five fulfillment microservices to start fulfillment processes. 9 00:00:33,900 --> 00:00:39,930 We call this communication is an event notification pattern, but order fulfillment process has several 10 00:00:39,930 --> 00:00:40,540 steps. 11 00:00:40,560 --> 00:00:46,230 For example, if the payment service is still pending, when the order arrives to fulfillment, that 12 00:00:46,230 --> 00:00:47,190 will be a problem. 13 00:00:47,190 --> 00:00:51,570 Or it could be a short delay in the payment status update in the database. 14 00:00:51,840 --> 00:00:56,100 This problem can be solved with the wait state in a list step functions. 15 00:00:56,190 --> 00:00:59,070 Even notification is file and forget pattern. 16 00:00:59,100 --> 00:01:04,620 It emits an event and execute the process but doesn't allow the users to control the timing. 17 00:01:05,010 --> 00:01:09,420 So how we can manage delays by using event driven microservice architectures. 18 00:01:10,620 --> 00:01:13,500 This can very easily solved with a list of functions. 19 00:01:13,560 --> 00:01:15,870 Edible functions has weight state. 20 00:01:15,870 --> 00:01:22,530 When the source services emits an event, the workflow waits for 10 seconds in this case and then starts 21 00:01:22,530 --> 00:01:24,450 the fulfillment processing steps. 22 00:01:24,780 --> 00:01:30,240 As you can see that with step functions, we can get more control over the service integrations by creating 23 00:01:30,240 --> 00:01:33,420 state machines and workflows on a list of functions. 24 00:01:34,390 --> 00:01:39,370 So microservices usually communicate through events with asynchronous communications to indicate the 25 00:01:39,370 --> 00:01:40,610 process change. 26 00:01:40,630 --> 00:01:47,410 So in our ecommerce case, customer placed an order in our eCommerce application API gateway next to 27 00:01:47,410 --> 00:01:49,320 request to order microservices. 28 00:01:49,330 --> 00:01:55,840 We can see in here and order microservices get processed and forward to inventory service and then payment 29 00:01:55,840 --> 00:01:58,210 service and provide to fulfillment process. 30 00:01:58,750 --> 00:02:02,790 This is the happy path of place order use case in ecommerce domain. 31 00:02:02,800 --> 00:02:07,360 There are no failures on the workflow and order get placed successfully. 32 00:02:07,480 --> 00:02:13,150 You got to place order request and you can continue to follow these microservices and perform the fulfillment 33 00:02:13,150 --> 00:02:14,860 process and success to order. 34 00:02:16,250 --> 00:02:21,710 So this is an example of distributed transaction with good persistence to all the transaction. 35 00:02:21,710 --> 00:02:26,780 Data gets stored across different databases and each service writes its own databases. 36 00:02:26,900 --> 00:02:31,260 What if we faced a network failure and the payment gateway is timeout? 37 00:02:31,280 --> 00:02:36,130 We can see in here what if the payment gateway is done and has timeout? 38 00:02:36,140 --> 00:02:41,220 So the last step of this place order workflow has failed at the time of failure. 39 00:02:41,240 --> 00:02:47,180 The order database and the inventory database are all in the updated to order status is set to fail 40 00:02:47,180 --> 00:02:50,450 in order to indicate failure to the downstream services. 41 00:02:50,630 --> 00:02:56,900 But if you look at the our order data, it is now inconsistent and need to roll back updates at that 42 00:02:56,900 --> 00:02:57,530 pace. 43 00:02:57,560 --> 00:03:04,940 Order status is set as ordered and inventory has been documented, so there is no way to correct data 44 00:03:04,940 --> 00:03:08,330 and there is no option to retry the process that have failed. 45 00:03:08,780 --> 00:03:11,470 So how we can fix this problem in our workflow? 46 00:03:11,540 --> 00:03:17,320 If you design monolithic application with relational databases, we can provide transactional automaticity 47 00:03:17,330 --> 00:03:23,750 consistency with for in case if a transaction pays in a relational database the transaction rollback. 48 00:03:24,350 --> 00:03:30,110 But when it comes to distributed transaction in microservice architectures, we can't use database commits 49 00:03:30,110 --> 00:03:35,360 and transactions due to transaction data is distributed across various databases. 50 00:03:36,610 --> 00:03:41,510 In this case, the solution is use the circle pattern with using circle pattern. 51 00:03:41,530 --> 00:03:47,770 If any transaction pace in the workflow, the circle executes a series of compensating steps that roll 52 00:03:47,770 --> 00:03:51,040 back the chains that were made by the preceding transactions. 53 00:03:51,520 --> 00:03:58,030 So if you think about our main use case, when we place order case in the ecommerce application, customer 54 00:03:58,030 --> 00:04:05,620 placed an order and when the item was deducting in stock, another customer order gets fulfilled. 55 00:04:05,620 --> 00:04:10,630 In the meantime, it goes out of stock and after the order has been placed. 56 00:04:10,900 --> 00:04:17,200 So as we all know that this can happen when multiple customers are trying to purchase the same item 57 00:04:17,200 --> 00:04:18,280 at the same times. 58 00:04:18,790 --> 00:04:22,960 So in this case, the update inventory step will fail in the workflow. 59 00:04:22,990 --> 00:04:26,550 The orchestrator will now execute the compensatory steps. 60 00:04:26,560 --> 00:04:28,930 It will run rollback inventory. 61 00:04:29,940 --> 00:04:32,850 And remove the order steps and return status. 62 00:04:32,850 --> 00:04:34,080 The A's failed. 63 00:04:34,470 --> 00:04:41,700 These compensatory steps are provided to ensure the data, integrity and consistency is maintained. 64 00:04:41,850 --> 00:04:45,900 The event rewards the original level and the orders reverted back. 65 00:04:47,300 --> 00:04:53,120 As you can see that the every action that makes a change to the databases there is an opposite action 66 00:04:53,120 --> 00:04:56,090 to compensate the change in case of a failure. 67 00:04:56,120 --> 00:05:01,430 So a failure that happens in the last step, the payment service in this case, we saw that in this 68 00:05:01,430 --> 00:05:03,010 case, the payment is done. 69 00:05:03,020 --> 00:05:09,440 This will trigger all the compensatory transactions to be executed before returning to failure state. 70 00:05:09,680 --> 00:05:14,280 We can implement this algorithm flow with using a step function workflows. 71 00:05:14,300 --> 00:05:17,780 The individual steps invoke lambda function to perform the task. 72 00:05:17,810 --> 00:05:24,620 The step transitions are defined in the Amazon State language, so it is very easy to set up and configure 73 00:05:24,620 --> 00:05:27,380 the compensatory transactions in case of failures. 74 00:05:27,650 --> 00:05:32,990 We will use such orchestration to implement e-commerce place or the use case for our happy and failure 75 00:05:33,350 --> 00:05:34,010 scenarios. 76 00:05:34,010 --> 00:05:41,840 So we will create an API gateway to trigger or add step function when place order request comes from 77 00:05:41,840 --> 00:05:48,890 customer and saga orchestrator pattern implementation with the ADA step functions in order fulfillment 78 00:05:48,890 --> 00:05:55,490 orchestration, we will use success and failure pets in a distributed transactions, restoring data 79 00:05:55,490 --> 00:05:57,970 consistency in Amazon dynamic databases. 80 00:05:57,980 --> 00:05:59,690 So now let's get started.