Twitter system design | twitter Software architecture | twitter interview questions

Is usertimeline also in redis cache? Does that mean that in redis we are maintaining two timelines for every user, one is usertime and other is homeline? Thanks

Hi Naren, Thanks for the video and your explanation. I had one question though , could you please explain why do we need Web socket connection here?

How expensive it would be to keep a copy of timelines in a redis cache for 20 million active users?

Great video! Thank you!

As I know, Redis Cluster does not require Apache ZooKeeper. Redis Cluster has its own built-in mechanisms for node coordination, failure detection, and failover. Please correct me if I am wrong.

Great content. Could you discuss news channels with multiple subscribers system design? Was asked in one of the interviews recently

great video!!

Absolutely love your work man. Gives so much more confidence when you mention what these companies are actually using. Highly, highly appreciated, thank you!

Sir, can you please explain why websocket push is used for mobile applications, why they can't call timeline service? why do we need to create another service like timeline service ?

very good compilation Naren, really like your videos as they are detailed and good quality :)

Great Video. The only part missing is storage estimation, at least for cache..since we are using it extensively in this design. Can you provide some data on storage please?

Woww. I mean, simply wow.

Good stuff brother

what if redis shuts down then how we'll handle that?

do you have any resource about the FANOUT algorithm?

This is so thorough and specific yet explained in a great way! Shows true mastery of the topic. Thank you very much for your work, really appreciated :D

Thank you also for mentioning the concepts involved (eventual consistency, gather & scatter, fan out, ifti)

Thank you for putting up this tutorial! Study vidoes like this and then practice at Meetapro with mock interviews will help you land multiple offers.

But if the Redis only has the list of the Users that a particular user 'A' is following, then how would the fanout service know which twitter users need to recieve the tweets made by user 'A' ?
Anyone?
Shouldnt it be the other way around?
Shouldnt there be a list of which users follow user 'A' or maybe both the lists because we also have to show which users follow whom?

Thank you!!!!

Thank you Narendra, your videos really helpful

Amazing work, crystal clear concepts. Thank you!

Have someone told you that you look like an NBA player Kyrie Irving?

[Question]
for the home timeline section. basically it's possible for a user to follow lots of celebrity users right? so when fetching the hometimeline data from celebrities, do we limit the number of celebrities? lets say this user follows hundreds of celebrities. thx

Thanks a lot for explaining it very clearly....

I wish there was a Korean translation.

is each square box a microservice ?

great job

Hello Sir,

For Searching tweets, I think Elastic Search which uses Inverted Index can be used.

the best video on twitter system design I've ever seen before in my life

I agree with the comment from Atul.K.Yadav. Great Job Narendra. You are exceptional in your explanation.

Amazing video , I like how you explained the fanout concept and how you explained that some of the daemon processes are delegated to streaming technologies like kafka. Would you recommend for a start-up that is trying to save on hosting , to use background jobs/workers instead of using Kafka or Apache Storm?

Amazing videos! Congrats! One suggestion could be to use a different microphone to reduce echo and noise and make your voice more clear since for me it seems that every word you say is important! Keep the good work!

What's the exact use-case for WebSocket here?
Is it for -
1. Send a response to the query for search timeline results?
2. Send notifications for new tweets?

to my understanding, requests are just sent and then all responses are sent using a WebSocket connection

Brilliant. Well done managing the details+clarity in 35 mins!

You might have aware about LinkedIn feed algorithm, they moved from "fanout write" -> "fanout read". Can you please explain about"fanout read" and the technology behind it?

Why do we use MySQL to store tweets instead of a NoSQL database like Cassandra? NoSQL databases offer better scalability, and we have to store millions of tweets every day.

You mentioned picking the recent tweets by a a celeb and putting it in the response but how are we going to query for that from Celebrity timeline? How to make sure we are not putting duplicate tweets of the celebrity?

Super work bro. Thanks a lot

One quick way I can think of fetching all Tweet related data for the users to display on the user timeline is to shard the user and tweet databases based on geo-location and building indexes on those shards to fetch the user data and tweet data associated with the user to display on the user timeline. Only when we have this I can guess then it makes sense to put this data into a Redis cache, otherwise for the Redis cache to hold this data you would first need to execute a huge Select query on top of the database. Please correct me if I am wrong in my thinking.

Wonderful explanation, keep it up. Had a question RE Celebrity Tweet - When a celebrity tweet, you mentioned that it will be a dynamic query from User to Celebrity User Home to fetch the latest tweet, what happens if a million users sending this query at the same time? Would the design work seamlessly as you mentioned? Could it be a good design to use a combination of Fan-out/in based on the Users active state (from latest to say a day latest, to weeks latest)

I understand that querying followers etc to create the Home time-line is expensive, and that REDIS can be the solution. However, how does the initial data enter the redis cache in the first place? You have to first do the slow query for everyone?

How do you deduplicate fetching celebrity tweets?

Great Content

Hi Narendra, great video! I have a question about storing the data in Redis:
Lets say user X has the hometimeline stored in the Redis database under 'user_x_hometimeline'. What kind of data is stored here? Only an array of all tweet ids, or also the metadata and the tweet itself? Or is the tweet content only stored in the main DB?
And a rethorical question: how would you change the system design when you could change the content of a tweet? Facebook/LinkedIn offer the feature to change the content of a post

Good video, most comprehensive. Kindly add subsection/bookmarks.