Brook Preloader

Blog

Data Flow Among Microservices

At MOURI Tech, we’ve started development of a new product with clear requirements and microservices architecture. It is a great start but the next question is what about data flow among services?

Know your requirements, understand and proceed!!

Our requirement is to accept the request in order, enrich data with the required information and send it to the next service. If the service is unable to handle the request then try again. This process should be asynchronous.

Ways of data flow among services

Traditional API relationship: Send request and receive a response. If service goes down there is always a chance of losing request-response. This is a tightly-coupled synchronous mechanism.

PUB-SUB relationship: Publisher sends a message to a topic, subscriber creates the subscription and receives a message from it. This is a loosely-coupled asynchronous mechanism.

PUB-SUB message flow fits our requirement as it handles a large number of events asynchronously.

1. A publisher/producer application sends messages to the topic. A message contains a payload and optional attributes that describe the payload content.

2. A queue is a buffer that stores all messages.

3. Pub/Sub forwards messages from a topic to its subscriptions individually. It depends on the way in which the subscriber has subscribed to the topic.

4. The subscriber receives pending messages from his subscription and acknowledges each one to the Pub/Sub service.

5. When a message is acknowledged by the subscriber, it is removed from the subscription’s message queue.

There are numerous message-oriented middleware available such as ActiveMQ/Apollo, RabbitMQ, ZeroMQ, Kafka, IronMQ, Apache Qpid and so on. Most of these follow the same or similar flow.



FEATURE
Active MQ/Apollo RabbitMQ ZeroMQ Kafka IronMQ Apache Qpid
Brokerless/ Decentralized No No Yes Distributed Distributed/ Cloud-based No
Clients C++, Java, Others C++, Java, Ruby, Node JS, Others C++, Java, Others C++, Java, Others C++, Java, Others C++, Java, Others
Transaction Yes Yes No No, but can be implemented with a plugin No Yes
Persistence/ Reliability Yes (Configurable) Yes (Built-in) No (Persistence requiring higher layers to manage persistence) Yes (Build-in
File System)
Yes (Built-in) Yes (Additional plugin required)
Routing Yes (Easier to implement) Yes (Easier to implement) Yes (Complex to implement) No (Can be implemented) No No
Users CSC etc. Mozilla, AT&T, UIDAI etc. No LinkedIn, Twitter, Mozilla etc. No No
Failover/HA Yes Yes No Yes Yes Yes
Unlimited Queue Yes Yes Yes Yes Yes Yes
Scalability Yes Yes Yes Yes Yes Yes
License/ Community Apache (open source) SpringSource (Licensed under Mozilla public license) iMatrix – Licensed Apache (open source) Iron.io (Commercial) Apache (open source)

Table: Comparison of various features in different middleware

Finally, we decided to go with RabbitMQ. Why?

  • RabbitMQ is a traditional messaging broker which comes with open source
  • In-built simple management UI & helps in monitoring and managing message queue easily
  • Easy to understand and implement on different languages
  • Supports messages over multiple protocols
  • Easy to trace messages and transactional history
  • Easy to deploy on Cloud
  • No message partitions – sends messages on prioritized order

This is the most important factor in our requirement. For instance, there are 2 producers (A and B) and 1 consumer (C). Events from A has to be picked up first then events from B by C. This is possible in RabbitMQ with priority (1-255, a larger number indicates higher priority).

Source: CloudAMQP (https://www.cloudamqp.com/img/blog/exchanges-topic-fanout-direct.png)

How to implement? Happy Coding!!

  1. Create your instance on https://customer.cloudamqp.com/instance/create
    • Signup on URL.
    • Follow the steps and setup instance. Once an instance is up, we can manage all in RabbitMQ dashboard.

2. Manage Vhost, Users and permissions on the Admin page.

3. Create an application on any language such as C++, Java, Ruby, Node js, Python, PHP etc.
We have used Node js. Follow the detailed steps mentioned below or go to
https://www.rabbitmq.com/tutorials/tutorial-three-javascript.html

4. Use amqplib to establish a connection with RabbitMQ with amqp connection string.

 const conn = await amqp.connect(connectionString, {  
servername: hostname,
clientProperties: {
connection_name: connectionname
}
});
conn.createChannel()

Ex connection string: amqps://{user}:{userpassword}@{hostname}/{vhost}

5. Create exchanges, queues and bind queues:

const entity = {    
 name:  ‘FirstService’,    
 exchangeType: 'direct',   
  exchangeOptions:  { durable: true },   
  queueOptions:     { exclusive: false , maxPriority: 10 }  
 } 
await channel.assertExchange(     
  entity.name,     
  entity.exchangeType,   
 entity.exchangeOptions    
 ); 
await channel.assertQueue(     
entity.name,       
entity.queueOptions     
);
await channel.bindQueue(      
 entity.name,     
  entity.name,       ""   
  ); 

// similarly create other service exchanges, queues and bind queues to exchanges.  

6. Now we are ready to publish and consume events.

  • Producer: On initialization connect and set up RabbitMQ with channel, exchange, queues and bind queue. Then keep publishing an event to consumer’s queue.
const msg = ("Hello Word"  
await channel.publish(targetExchange, "", Buffer.from(msg))  
  • Consumer: On initialization connect and set up RabbitMQ with channel, exchange, queues and bind queue. Then keep consuming incoming events.
channel.consume(q.queue, function(msg) {
        if(msg.content) {
            console.log(" [x] %s", msg.content.toString());
          }
      }, {
        noAck: true
      }); 

We are thus able to achieve our requirement with the help of RabbitMQ.

References

Contact for further details

Lalitha Chukka
Technology Specialist
lalithac.in@mouritech.com
MOURI Tech

0 0 vote
Rating
guest
0 Comments
Inline Feedbacks
View all comments