Designing the architecture of Twitter and similar social networks is a popular engineering interview question asked at companies like LinkedIn, Microsoft, Google, Snapchat, NVidia and others. This interview question is extremely broad but gives you the opportunity to talk about technologies like in-memory databases, replication, sharding etc. It's important to give a clear high level overview of the problem, ask clarifying questions and talking confidently about strengths and weaknesses of the proposed solution. Every architecture has trade-offs and interviewers want to hear you talk about them.
Follow SuccessInTech on Facebook: https://www.facebook.com/SuccessInTech/
Follow SuccessInTech on Twitter:
Details taken from a presentation of the VP of Engineering at Twitter: https://www.infoq.com/presentations/Twitter-Timeline-Scalability
Nice effort and really explain the things. Please mention the sources you have referred. This would allow us to go in-depth of a topic.
Can you please also add low level design?
In some orgs they asks HDL and LLD. How would you classify this? I think these boxes you drew were HLDs but the tables and ER diagrams would be part of LLD right?
Hi, Its an awesome video on twitter Architecture. Just a qn on, when you had said that when user tries to access his Home Timeline, if you are a follower of big celebrities, their latest tweets would be fetched from DB and inserted along with Redis data. You had missed this feature while explaining the Home timeline feature towards the end of video. Please clarify.
I don't think it is fair to ask this question in an interview and expect the candidate to come up with this Redis solution. If the candidate can explain why fan out is a challenge and come up with some reasonable solutions/suggestions, that must be sufficient.
Thanks for the great video! I have a question regarding timeline - does the system stores the whole user's timeline from the beginning (in Redis), or some portion of the timeline? For example, from last login?
The video is good, but instead of "How to design Twitter?" you actually talk more about "How Twitter IS designed" which is a huge difference because rather than show how to think on your feet you recall stuff you know from memory.
You did some research so you know at least how certain parts are designed and you even claim that in the video. Go watch again at 7:10 where you say that you know what Twitter is doing cause you watched a very good talk by VP of engineering from Twitter and you say that you will tell us.
This is exactly my issue with the video, cause rather than solving a problem while thinking on your feet which is what one normally does during an interview you actually repeat stuff from memory and even if you mix that with your own experience (obviously one can tell from the video that you do have experience and knowledge in the area) it still is far from an interview situation. So the title is wrong and misleading.
Can you please do a system design video on 2 topics
A) how do u make sure the number of simultaneous video streams somebody watching let’s say Netflix is only 3 devices at a time.
B) windows system update, how do u stream a windows system update to client computers ?
Awesome video. Thank you. It gives me a basic idea about how to approach system design questions. This design covers a lot of things which is used in real-world huge systems. It includes relational databases, In-memory databses, hashing, load balancers and most important how to design system based on actual requirements, like eventual read consistency in case of twitter.
You're welcome mate.
Overall your algorithm sounds ok, I think that would be a good and logic approach to the problem, having in mind that it's a costly operation but not needed that often.
However there are some other moving parts, for example Bob probably won't get all tweets from all people he follow at once. Among the timestamp sort there are more criteria, like how relevant are certain accounts to Bob. He may be following a celebrity or two or N, and that tweets will for sure have more precedence that something his real-life-friend-but-sporadic-user tweeted. They doesn't have to be celebrities, maybe a certain account Bobs usually interacts a lot with will gain precedence too. When Bob starts to scroll then pagination happens and eventually he'll see the rest, "less" relevant content, including that friend's sporadic posts.
In sort, what I mean is that instead of
"1 for all the people Bob follows"
I think it would be:
"1 for the most relevant people Bob follows and leaving the rest for later"
Here is a link from a very interesting talk where the fan out thing is discussed. A bit dated but it's worth: https://www.youtube.com/watch?v=VttXHNveuwI
Alberto Carbonell thanks man, I understand and agree with you. I'm just looking for details about how that content is actually generated. I think you hinted at this by mentioning it's still done in a fan out manner. Could the algorithm be something along the lines of:
1 for all the people Bob follows
2 get latest n tweets and add to some list
3 sort this list of tweets according to some ranking algorithm
4 add to redis
5 serve to Bob
Perhaps anybody knows of a link of where I could find more into of this fan out on read scenario?
I think it could be something like this:
If Bob gets out of the active people for timeline pre-computation in Redis, then the next time he connects his timeline is generated on the fly, being this a more costly operation because is not "cached" at Redis so he may wait some seconds. Conceptually this process would be similar to the fan-out one, which builds each user's timeline but in this case only processing Bob account. The timeline gets computed and delivered to Bob. After that he probably will be added to the Redis list again in an optimistic fashion (haha) so next times he'll get the timeline from Redis as usual.
Hope this helps.
Is it true that they only use an in-memory database? I would think that they have another database that persists the tweet and then they just use the cache for timelines and update the timelines in the cache (which redis can be used as).
Really good video.. Well put up but I also see some interviews where they expect you to draw class diagrams of systems wherein it would be difficult to put across the detailing you spoke about without actually getting face-2-face with the interviewer. How do you suggest I go about it? Thanks in advance.
Thanks for a great video. I've got a couple of questions though. Q1. How do you maintain the followers' database? Is it denormalized for each user?
Q2. How often and when do you update the hashlookup (to find the correct redis instance)? What happens if that hashlookup server goes down? Should we add redundancy there?
Q1: Not necessarily. Looking up followers should be a fairly fast lookup based on an indexed key which is the userId.
Q2: Yes redundancy for loadbalancers are a must, you don‘t want a single point of failure there. By the way: There is going to be a new video this weekend about loadbalancing ;)
Thanks for sharing this content. My only question or confusion arises from how your optimized solution solved the unoptimized solution of needing to query a large tweets for reads. The optimized solution still has a user's list storing a tweet_id which to my understanding looks like it still needs to query a large tweets table here also? Do you have any clarification for my misunderstanding?
Buddies, We have created a youtube channel, a one-stop solution to Learn Data Structure and Algorithm along with associated code in GitHub. Please do subscribe to the channel and share your feedback.
Twitter could have gone with LDAP for storing the following lookup per user as it will be much faster (Followers list doesn't get updated that fast, even a MongoDB instance could have been sufficient as eventual consistency could have been suffice). Now as far as storing tweets are considered, I would have gone with LDAP for first 100 tweets and all tweets would have been stored in Relational DB for backup. For important people, I would have rather used a different store altogether even a single instance would have been sufficient. For hometime line, definitely InMemory DB is required as Home page is what almost 99% users will always hit. Let me know if all above will actually make sense or design mentioned in your video is better.
This is a great video. I have a quick question with using list in Redis. The video only mentioned store the tweet_id and sender_id for Bob's list. What about the actual tweet? Is the actual tweet store in Redis and we will need to do a look up by each tweet_id to get the actual text?
Thanks a lot for the video. it helps us to think the system design in a broader perspective.
I have two questions here. You said conventional Relational Database would be a bottleneck in this kind of systems. Does NOSQL would be the ideal one here for storage?. Also during the entire video, you have talked about In Memory Database. At what point of time, this data gets persisted into the database?
He mentioned there should be a machine between the Load balancer and the redis clusters. I would guess that machine would take care of persisting the tweet into the database (preferably in an async manner)
Hello Mr. Lopez! I loved this video. But it is always very likely to face a system design question totally out of what you had prepared for an interview. So a video on all possible system design components and how they are used for specific use cases in real life products can be very useful. So once building blocks are available, its easier from there. For example, REDIS database with its in-memory function is a good takeaway from this video which I can use in different scenarios.
Thanks Ramon such good explanations,
1. purpose of 3 cluster ? is only for - the fastest one response to be taken as result?
2. user bob table and follower table are created in radish cache only not physical db tables?
Here are few others design / architecture which i am curious to know ... would be great if you could create them in the near future:
1. Youtube architecture and design or similar video streaming websites
2. Amazon or any E-commerce website
4. Google Search Engine
Thanks for such detailed explanation :)
one question .. you started from a naive solution to store tweets in Relational DB
then moved to storing it in Redis In memory DB ... my question is .. is storing in Redis is enough ? aren't we storing it later in any Relational DB or just storing all of the information in Redis and that too in 3 Redis per user ? If yes, then will it cause inconsistency like 3 redis have some user's data while other Redis DB has other user's data but not of Alice ?
Please correct me if i am wrong or have missed something and even for silly question :)
this isn't a direct answer to your question but at 24:40 talking about search he says when the tweet reaches the load balancer it also triggers a totally separate architecture that stores the tweet. he doesn't mention if it is a rational db or not though.
Great video. Thanks for the upload. Some thoughts or questions I had was about how the data is being stored in redis. You mentioned a hash table that stores the ip address of the redis machine, but isn't that too over simplistic for an interview? A little more dive deep on data sharding and access will be helpful.
Seems like it's a big omission to just draw a line from the client to the load balancer to redis. Is the load balancer connecting directly to redis and storing that information there? Or is it connecting to a webserver which is storing the information in redis? Seems like the webserver would be worth a box on the diagram if the load balancer is.
Thanks that would help. More than once i have been asked to get into details of data sharding, consistent hashing, data replication and handling failures when a node goes down, and what happens to the sharded data, etc etc.
Yeah you could dive deep into the exact routing mechanisms or even talk about tools doing that. But in the end it all boils down to some sort of map which maps a user to it's stored timelines. That map is probably also replicated, under strict latency limits, frequently updated, thread safe etc. I would always start with the basic idea, which is: "You have a primary key (user ID) and you need to know FAST where the timeline is stored". And then build on top of that if time allows. Regarding sharding: I'm currently working on a system design video on the topic of storage mechanisms and I'll try to cover sharding there as well. Stay tuned :D
+Brijesh Jaggi Redis acts like a key/value store. No relations between data other than key/value mapping but raw speed because of it‘s in-memory characteristics. There are lots of good resources out there.
Awesome video. Such a great explanation and concept about twitter. I know you would be busy so thank you so much for making video.
I have a request.. can you talk about designing of facebook mutual friend system. It seems complecated because we don't check our friend list actively and still facebook is able to display all our mutual friend list instantly and also suggest friends based on that. Thanks again for your effort and making these awesome videos.
Thank you so much for sharing this. But I think it is just some parts of the system design. We still have a lot of things to introduce. Redis is a memory database, but what if all the replicas are down? We should store the data in disk, with Redis itself or other no-sql databases. Shall we consider how the servers work, what servers we should have, with read and post servers respectively? How we consider the security issues, shall we user a gateway, and introduce SOA theory tools like service registry and discovery?
+Yasen Zhang Yeah, nobody expects you to cover the in‘s and out‘s of such a system in a 45min interview. Architectures like this grow over years. If the interviewer wants you to cover a specific topic then you should dive deeper into it.
Great video. Thanks for taking effort to make it available for us all. A quick question if you can help me understand, wondering when is the tweet stored and where? Who can be responsible for such cases? Thanks!
One question, probably silly one.:) At the end of the video, you mentioned that LB will do hash look up to find which RADIS cluster to query for BOB's timeline, does this mean that always BOB timeline will be stored in same 3 RADIS machine? Or it depends on something.
+Nirav Purohit No silly questions, don’t worry. It means Bob’s timeline is always stored on 3 different Redis hosts (for availability and other reasons). Now when he visits the timeline in his browser, the LB (or a component behind the LB) queries the hashmap to find one of those 3 Redis machines. The content returned by the fastest of those 3 machines is returned to Bob’s browser.
Thanks for insightful videos.
Can you please give your insights for the designs of the dynamic scoring (for sports udpated scores) websites like cricinfo etc where scores keeps changing every seconds.
would really love to see that.
Hi, Can you please do a video on designing a service like Uber/Lyft? Including services like location based look-ups for cabs, computing route, fare etc. It seems to be a common interview question. Great job by the way.
+Jon Snow Hey man, glad you like them! If you want to dive deeper into real world solutions http://highscalability.com is really good. On the more theoretical side of things the books of Andrew S. Tannenbaum are classics but they get updated quite often. Definitely worth reading.
man i cant express my happiness. you are the only one on youtube(infact the internet) concentrating on high level systen design. many companies are shifting their focus from algorithms to system design now a days. it was so hard to figure out how to come up with answers to these. Your videos are a life saver sir. You people are literally changing lives. the minimum i can do is say a big thank you to you for making these vids.
+Biswajit Singh I‘m really stoked to hear that, man! Happy I can help you out. You would do me a huge favor if you could share my videos on your social networks. There will be more interesting videos to come! 👍
+Sanjeev Garg If you want to make the same tradeoff as Twitter you could propose a similar design. Although you could argue that the timeline is not as prominent in LinkedIn. The real feature of LinkedIn is a deep social graph that let‘s users explore networks (in the social sense) defined by their work history, education, skills and interests. Interesting question, could be a future video (Design FB/LinkedIn). Thank you!
Marin Community Foundation and Asian Americans/Pacific Islanders in Philanthropy.
Overall, US-born Muslims make up the largest percentage at 34% of all Muslims in the Bay Area, followed by 14% born in Pakistan, 11% in Afghanistan, 10% in India, 3% in Egypt and 2% each in Iran, Jordan, Palestine and Yemen.
Silicon Valley Pakistani-American by the Numbers:
There are 35,000 Pakistani-born Muslims in San Francisco Bay Area, or 14% of the 250,000 Muslims who call the Bay Area home, according to the study. Bay Area Muslim community constitutes 3.5 percent of the area’s total population and is one of the highest concentrations of Muslims in the country.
As of 2013, South Asian Muslims, including Pakistanis, have the highest income levels, with nearly half (49%) of them having a household income above $100,000. In comparison, those groups with the lowest proportion of household incomes above $100,000 were Hispanic Muslims (15%), Afghans (10%), and African American Muslims (10%).
The Bay Area Muslim community is very diverse in terms of race and ethnicity:
South Asians (30%)
African Americans (9%)
Asian/Pacific Islanders (7%)
Based on the survey findings, the majority of Muslims live in the following three counties:
and Contra Costa (12%)
Thousands of Pakistan-born techies are working at Apple, Cisco, Google, Intel, Oracle and hundreds of other high-tech companies from small start-ups to large Fortune 500 corporations. Pakistani-Americans are contributing to what Erik Brynjolfsson and Andrew McAfee describe as "The Second Machine Age" in a recent book with the same title.