{"id":824,"date":"2019-10-09T23:53:28","date_gmt":"2019-10-09T23:53:28","guid":{"rendered":"http:\/\/www.projectimmerse.com\/blog\/?p=824"},"modified":"2020-06-15T10:43:28","modified_gmt":"2020-06-15T10:43:28","slug":"distributed-systems-in-one-lesson","status":"publish","type":"post","link":"https:\/\/www.projectimmerse.com\/blog\/distributed-systems-in-one-lesson\/","title":{"rendered":"Distributed Systems in One Lesson"},"content":{"rendered":"\r\n<p>Alright, this is a continuation of <strong>System Architecture<\/strong>, a super broad subject I&#8217;ve been recently immersing myself in. I must admit, my last post was a bit poor in quality &#8211; I even left things hanging in the end. So I felt compelled to dive in deeper into an architecture known as &#8220;Distributed Systems&#8221;, yes in one lesson by <a href=\"http:\/\/timberglund.com\/\" title=\"Tim Berglund\">Tim Berglund.<\/a>. As the name of this blog post suggests, this is based off a youtube video I found &#8211; <a href=\"https:\/\/www.youtube.com\/watch?v=Y6Ev8GIlbxc\" title=\"Distributed Systems in One Lesson\">Distributed Systems in One Lesson.<\/a><\/p>\r\n\r\n<!--more-->\r\n\r\n<p>So I was a bit hesitant on investing roughly 45 min of my time on a video that claims to cover a vast topic in that short a time. But hey, what&#8217;s 45 minutes? So I decided to proceed and outline a few important key points when it comes to <strong>Distributed Systems<\/strong>. Quick background on Tim &#8211; Tim is Director of Developer Experience at <a href=\"https:\/\/www.atlassian.com\/software\/confluence\" title=\"Confluence\">Confluence<\/a><\/p>\r\n\r\n<h2>What is a Distributed System and why they are terrible?<\/h2>\r\n\r\n<p>In a simple system, the below actions aren&#8217;t too complicated, however as Tim explains in his experience &#8211; complications arise when the system scales and gets larger.<\/p>\r\n\r\n<p>\r\n&#8211; Store Data<br>\r\n&#8211; Computate Data<br>\r\n&#8211; Move Data<br>\r\n<\/p>\r\n\r\n<p>A formal definition of a Distributed System by Andrew Tannenbaum &#8211;<\/p>\r\n\r\n<blockquote>\r\n&#8220;A collection of independent computers that appear to its users as one computer.&#8221;\r\n<\/blockquote>\r\n\r\n<p>Examples of these include <a href=\"https:\/\/aws.amazon.com\/\" title=\"Amazon Web Services\">Amazon.com (AWS)<\/a>, <a href=\"http:\/\/cassandra.apache.org\/\" title=\"Cassandra Cluster\">Cassandra Cluster<\/a> and <a href=\"https:\/\/kafka.apache.org\" title=\"Kafka\">Kafka<\/a>.<\/p>\r\n\r\n<h3>Distributed Systems have 3 Characteristics<\/h3>\r\n<p>\r\n1. The computers operate concurrently<br>\r\n2. The computers fail independently<br>\r\n3. The computers do not share a global clock\r\n<\/p>\r\n\r\n<p>Tim says&#8230; my goal here&#8230;<\/p>\r\n\r\n<blockquote>\r\n&#8220;Don&#8217;t do this. Get an easier job. Build a smaller system. Live in the country.&#8221;\r\n<\/blockquote>\r\n\r\n<p>I couldn&#8217;t help myself here but quote those exact words from the video. I&#8217;m excited to see what Tim has to say and his reasons why.<\/p>\r\n\r\n<h3>Single-Master Storage<\/h3>\r\n<p>Yes the simple days. You have a database that exists on one server. But as the scale slider gets larger, things get busier and busier. Typically in a web system there is more read traffic than there is write traffic. In a relational database, reads are usually cheaper than writes.<\/p>\r\n\r\n<h4>How do we scale reads?<\/h4>\r\n<p>Answer. <strong>Read Replication<\/strong>. However as Tim describes, we&#8217;ve already broken something with this system, we&#8217;ve created a distributed system it&#8217;s just not intentional (yet).<\/p>\r\n\r\n<p>Well, what did we break?<\/p>\r\n\r\n<p>We broke consistency due to propagation involved in Read Replication. Also as we continue to scale up and add more follower servers, the master server will eventually max out. So what do we do?<\/p>\r\n\r\n<h3>Sharding &#8211; splitting up the database<\/h3>\r\n<p>Not a new technique. But with this system, we&#8217;ve again broken consistency and the data model. Now we can join across shards, we have completely different database from shard to shard. This works &#8220;sometimes&#8221;.<\/p>\r\n\r\n<h3>Alright, what are some viable solutions?<\/h3>\r\n<p>When indexing, removing table joins no longer become a stable solution in scaling a database we en up denormalizing, quite the opposite of what a relational database is supposed to be.<\/p>\r\n\r\n<h4>Consistent Hashing<\/h4>\r\n<p>Consistent hashing requires that the hashing algorithm stays constant and each node is assigned a hash as seen below. Then some data, key value pair is is stored on each node.<\/p>\r\n<img decoding=\"async\" src=\"\/blog\/wp-content\/uploads\/2019\/10\/distributed-systems-consistent-hashing.png\" alt=\"Distributed Systems - Consistent Hashing\">\r\n\r\n<h3>Replication<\/h3>\r\n<p>With replication, we can replicate each key value pair on other nodes. As mentioned earlier, since each system stands independently from each other should one fail (hardware failure or some anomaly), a distributed system allows for fail proof delivery.<\/p>\r\n<img decoding=\"async\" src=\"\/blog\/wp-content\/uploads\/2019\/10\/distributed-systems-replication.png\" alt=\"Distributed Systems - Consistent Hashing\">\r\n\r\n<p>But there is an issue here as we are in the same attempt to solve the issue. Yes. <strong>Consistency<\/strong>. Now that we have several copies of data, this causes a consistency issue. When we expect a successful read or write from a database, we expect all copies of data to be consistent across the board. Well what if one or two nodes fail in a read? Or let&#8217;s say one or two nodes fail in a write? Whoa, these are some major issues.<\/p>\r\n\r\n<h4>C.A.P. Theorem<\/h3>\r\n<p>\r\n1. Consistency<br>\r\n2. Availability<br>\r\n3. Partition Tolerance<br>\r\n<\/p>\r\n\r\n<p>The short version of the underlying issue using the C.A.P. method &#8211; you can&#8217;t have all three. Put simply, we can only achieve 2 of the 3 things, not all three. This presents the same issue.<\/p>\r\n\r\n<p>Alright enough with Storage. Let&#8217;s move on to the next topic &#8211; <strong>Distributed Computation<\/strong>.<\/p>\r\n\r\n<h2>Distributed Computation<\/h2>\r\n<p>Again, really easy if you only have a single processor. But&#8230; you have multiple computers and you&#8217;re trying to distribute computational tasks across multiple computers &#8211; this makes things harder.<\/p>\r\n\r\n<h3><a href=\"https:\/\/hadoop.apache.org\/\" title=\"Hadoop\">Hadoop<\/a> and MapReduce<\/h3>\r\n<p>\r\n&#8211; MapReduce API<br>\r\n&#8211; MapReduce job management<br>\r\n&#8211; Distributed Filesystem<br>\r\n&#8211; Enormous ecosystem\r\n<\/p>\r\n\r\n<p>&#8230; but this a legacy system.<\/p>\r\n\r\n<h3><a href=\"https:\/\/spark.apache.org\/\" title=\"Spark API\">Spark API<\/a>, similar to Hadoop<\/h3>\r\n<p>Instead of Map, we have Transform. Instead of Reduce we have Action.<\/p>\r\n\r\n<p>\r\n&#8211; Scatter\/gather paradigm (similar to MapReduce)<br>\r\n&#8211; More general data model(RDDs, DataSets)<br>\r\n&#8211; Provides objects. More general programming model (transform\/action)<br>\r\n&#8211; Storage agnostic\r\n<\/p>\r\n\r\n<h3><a href=\"https:\/\/kafka.apache.org\/\" title=\"Kafka\">Kafka<\/a> &#8211; distributed streaming platform<\/h3>\r\n<p>Enter Kafka<\/p>\r\n<p>\r\n&#8211; Focuses on real-time analysis,  not batch jobs<br>\r\n&#8211; Streams and streams only<br>\r\n&#8211; Except streams are also tables (sometimes)<br>\r\n&#8211; No cluster required!<br>\r\n&#8211; Streams are first class citizens<br>\r\n&#8211; Not just a messaging framework, but also a computational framework for distributed systems\r\n<\/p>\r\n\r\n<p>Alright on to your last topic &#8211; <strong>Messaging<\/strong><\/p>\r\n\r\n<h2>Messaging in Distributed Systems<\/h2>\r\n<p>\r\n&#8211; Means of loosely coupling subsystems.<br>\r\n&#8211; Build little chunks of functionality<br>\r\n&#8211; Version and release messaging functionality<br>\r\n&#8211; Messages consumed by <em>subscribers<\/em><br>\r\n&#8211; Created by one or more <em>producers<\/em><br>\r\n&#8211; Organized into <em>topics<\/em><br>\r\n&#8211; Processed by <em>brokers<\/em><br>\r\n&#8211; Usually persistent over the short term\r\n<\/p>\r\n\r\n<h3>Messaging Problems<\/h3>\r\nProducer -> Topic -> Consumer\r\n<p>\r\n&#8211; What if producer gets too big?\r\n&#8211; What if a topic gets too big &#8211; too much data retention\r\n&#8211; Or messages get too big\r\n&#8211; Transacting messages too fast (read\/write)\r\n&#8211; What if one computer isn&#8217;t reliable enough?\r\n<\/p>\r\n\r\n<h3>Revisiting the Kafka Distributed Messaging System<\/h3>\r\n<p><strong>Definitions<\/strong><\/p>\r\n\r\n<p><img decoding=\"async\" src=\"\/blog\/wp-content\/uploads\/2019\/10\/apache-kafka-distributed-system.png\" alt=\"Kafka Distributed Systems\">\r\n\r\n<p>\r\n&#8211; <strong>Message:<\/strong> an immutable array of bytes<br>\r\n&#8211; <strong>Topic:<\/strong> a feed of messages<br>\r\n&#8211; <strong>Producer:<\/strong> a process that publishes messages to a topic.<br>\r\n&#8211; <strong>Consumer:<\/strong> a single-threaded process that subscribes to a topic<br>\r\n&#8211; <strong>Broker<\/strong>: one of the servers that comprises a cluster\r\n<\/p>\r\n\r\n<p>So Kafka is a great messaging solution, but when we start getting into replication &#8211; this is where things start getting complicated.<\/p>\r\n\r\n<h2>Lambda Architecture<\/h2>\r\n<p><img decoding=\"async\" alt=\"Lambda Architecture\" src=\"\/blog\/wp-content\/uploads\/2019\/10\/lambda-architecture-distributed-systems.png\"><\/p>\r\n\r\n<p>Why was this bad though? Well, as Tim explains &#8211; this forced people to write two versions of code, one for stream and one for batch.<\/p>\r\n\r\n<h2>With Streams<\/h2>\r\n<p>Now that we have a messaging bus. Stop taking events and writing them to some place in a database. Let them be events and process them in-place through a stream processing API. Events are consumed on a as-needed basis.<\/p>\r\n\r\n<h2>Conclusion:<\/h2>\r\n<p>This was a super interesting intro to distributed systems and its pitfalls. I myself have run into this while prematurely designing a database with scaling in mind. As I planned for the future, I ended up denormalizing the database and I was slowly spiraling down into a mess I could no longer translate to my colleagues.<\/p>\r\n\r\n<p>Stay tuned for more blog topics covering System Architecture.<\/p>\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n","protected":false},"excerpt":{"rendered":"<p>Alright, this is a continuation of System Architecture, a super broad subject I&#8217;ve been recently immersing myself in. I must admit, my last post was a bit poor in quality &#8211; I even left things hanging in the end. So I felt compelled to dive in deeper into an architecture known as &#8220;Distributed Systems&#8221;, yes &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/www.projectimmerse.com\/blog\/distributed-systems-in-one-lesson\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Distributed Systems in One Lesson&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":1096,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-824","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.3 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Distributed Systems in One Lesson - Project Immerse<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.projectimmerse.com\/blog\/distributed-systems-in-one-lesson\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Distributed Systems in One Lesson - Project Immerse\" \/>\n<meta property=\"og:description\" content=\"Alright, this is a continuation of System Architecture, a super broad subject I&#8217;ve been recently immersing myself in. I must admit, my last post was a bit poor in quality &#8211; I even left things hanging in the end. So I felt compelled to dive in deeper into an architecture known as &#8220;Distributed Systems&#8221;, yes &hellip; Continue reading &quot;Distributed Systems in One Lesson&quot;\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.projectimmerse.com\/blog\/distributed-systems-in-one-lesson\/\" \/>\n<meta property=\"og:site_name\" content=\"Project Immerse\" \/>\n<meta property=\"article:published_time\" content=\"2019-10-09T23:53:28+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2020-06-15T10:43:28+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.projectimmerse.com\/blog\/wp-content\/uploads\/2019\/10\/distributed-systems-in-one-lesson-system-architecture.png\" \/>\n\t<meta property=\"og:image:width\" content=\"750\" \/>\n\t<meta property=\"og:image:height\" content=\"965\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"projectimmerse\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"projectimmerse\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.projectimmerse.com\\\/blog\\\/distributed-systems-in-one-lesson\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.projectimmerse.com\\\/blog\\\/distributed-systems-in-one-lesson\\\/\"},\"author\":{\"name\":\"projectimmerse\",\"@id\":\"https:\\\/\\\/www.projectimmerse.com\\\/blog\\\/#\\\/schema\\\/person\\\/c53f2864be524ee6fa08a7e4800dd1e5\"},\"headline\":\"Distributed Systems in One Lesson\",\"datePublished\":\"2019-10-09T23:53:28+00:00\",\"dateModified\":\"2020-06-15T10:43:28+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.projectimmerse.com\\\/blog\\\/distributed-systems-in-one-lesson\\\/\"},\"wordCount\":1184,\"commentCount\":0,\"image\":{\"@id\":\"https:\\\/\\\/www.projectimmerse.com\\\/blog\\\/distributed-systems-in-one-lesson\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.projectimmerse.com\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/10\\\/distributed-systems-in-one-lesson-system-architecture.png\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.projectimmerse.com\\\/blog\\\/distributed-systems-in-one-lesson\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.projectimmerse.com\\\/blog\\\/distributed-systems-in-one-lesson\\\/\",\"url\":\"https:\\\/\\\/www.projectimmerse.com\\\/blog\\\/distributed-systems-in-one-lesson\\\/\",\"name\":\"Distributed Systems in One Lesson - Project Immerse\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.projectimmerse.com\\\/blog\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.projectimmerse.com\\\/blog\\\/distributed-systems-in-one-lesson\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.projectimmerse.com\\\/blog\\\/distributed-systems-in-one-lesson\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.projectimmerse.com\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/10\\\/distributed-systems-in-one-lesson-system-architecture.png\",\"datePublished\":\"2019-10-09T23:53:28+00:00\",\"dateModified\":\"2020-06-15T10:43:28+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/www.projectimmerse.com\\\/blog\\\/#\\\/schema\\\/person\\\/c53f2864be524ee6fa08a7e4800dd1e5\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.projectimmerse.com\\\/blog\\\/distributed-systems-in-one-lesson\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.projectimmerse.com\\\/blog\\\/distributed-systems-in-one-lesson\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.projectimmerse.com\\\/blog\\\/distributed-systems-in-one-lesson\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.projectimmerse.com\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/10\\\/distributed-systems-in-one-lesson-system-architecture.png\",\"contentUrl\":\"https:\\\/\\\/www.projectimmerse.com\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/10\\\/distributed-systems-in-one-lesson-system-architecture.png\",\"width\":750,\"height\":965,\"caption\":\"Distributed Systems in One Lesson\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.projectimmerse.com\\\/blog\\\/distributed-systems-in-one-lesson\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.projectimmerse.com\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Distributed Systems in One Lesson\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.projectimmerse.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/www.projectimmerse.com\\\/blog\\\/\",\"name\":\"Project Immerse\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.projectimmerse.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.projectimmerse.com\\\/blog\\\/#\\\/schema\\\/person\\\/c53f2864be524ee6fa08a7e4800dd1e5\",\"name\":\"projectimmerse\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/4d06955033d6227bfdcf30014e457e4334f7deeb73907de49b65ec2484921931?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/4d06955033d6227bfdcf30014e457e4334f7deeb73907de49b65ec2484921931?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/4d06955033d6227bfdcf30014e457e4334f7deeb73907de49b65ec2484921931?s=96&d=mm&r=g\",\"caption\":\"projectimmerse\"},\"url\":\"https:\\\/\\\/www.projectimmerse.com\\\/blog\\\/author\\\/projectimmerse\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Distributed Systems in One Lesson - Project Immerse","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.projectimmerse.com\/blog\/distributed-systems-in-one-lesson\/","og_locale":"en_US","og_type":"article","og_title":"Distributed Systems in One Lesson - Project Immerse","og_description":"Alright, this is a continuation of System Architecture, a super broad subject I&#8217;ve been recently immersing myself in. I must admit, my last post was a bit poor in quality &#8211; I even left things hanging in the end. So I felt compelled to dive in deeper into an architecture known as &#8220;Distributed Systems&#8221;, yes &hellip; Continue reading \"Distributed Systems in One Lesson\"","og_url":"https:\/\/www.projectimmerse.com\/blog\/distributed-systems-in-one-lesson\/","og_site_name":"Project Immerse","article_published_time":"2019-10-09T23:53:28+00:00","article_modified_time":"2020-06-15T10:43:28+00:00","og_image":[{"width":750,"height":965,"url":"https:\/\/www.projectimmerse.com\/blog\/wp-content\/uploads\/2019\/10\/distributed-systems-in-one-lesson-system-architecture.png","type":"image\/png"}],"author":"projectimmerse","twitter_card":"summary_large_image","twitter_misc":{"Written by":"projectimmerse","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.projectimmerse.com\/blog\/distributed-systems-in-one-lesson\/#article","isPartOf":{"@id":"https:\/\/www.projectimmerse.com\/blog\/distributed-systems-in-one-lesson\/"},"author":{"name":"projectimmerse","@id":"https:\/\/www.projectimmerse.com\/blog\/#\/schema\/person\/c53f2864be524ee6fa08a7e4800dd1e5"},"headline":"Distributed Systems in One Lesson","datePublished":"2019-10-09T23:53:28+00:00","dateModified":"2020-06-15T10:43:28+00:00","mainEntityOfPage":{"@id":"https:\/\/www.projectimmerse.com\/blog\/distributed-systems-in-one-lesson\/"},"wordCount":1184,"commentCount":0,"image":{"@id":"https:\/\/www.projectimmerse.com\/blog\/distributed-systems-in-one-lesson\/#primaryimage"},"thumbnailUrl":"https:\/\/www.projectimmerse.com\/blog\/wp-content\/uploads\/2019\/10\/distributed-systems-in-one-lesson-system-architecture.png","inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.projectimmerse.com\/blog\/distributed-systems-in-one-lesson\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.projectimmerse.com\/blog\/distributed-systems-in-one-lesson\/","url":"https:\/\/www.projectimmerse.com\/blog\/distributed-systems-in-one-lesson\/","name":"Distributed Systems in One Lesson - Project Immerse","isPartOf":{"@id":"https:\/\/www.projectimmerse.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.projectimmerse.com\/blog\/distributed-systems-in-one-lesson\/#primaryimage"},"image":{"@id":"https:\/\/www.projectimmerse.com\/blog\/distributed-systems-in-one-lesson\/#primaryimage"},"thumbnailUrl":"https:\/\/www.projectimmerse.com\/blog\/wp-content\/uploads\/2019\/10\/distributed-systems-in-one-lesson-system-architecture.png","datePublished":"2019-10-09T23:53:28+00:00","dateModified":"2020-06-15T10:43:28+00:00","author":{"@id":"https:\/\/www.projectimmerse.com\/blog\/#\/schema\/person\/c53f2864be524ee6fa08a7e4800dd1e5"},"breadcrumb":{"@id":"https:\/\/www.projectimmerse.com\/blog\/distributed-systems-in-one-lesson\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.projectimmerse.com\/blog\/distributed-systems-in-one-lesson\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.projectimmerse.com\/blog\/distributed-systems-in-one-lesson\/#primaryimage","url":"https:\/\/www.projectimmerse.com\/blog\/wp-content\/uploads\/2019\/10\/distributed-systems-in-one-lesson-system-architecture.png","contentUrl":"https:\/\/www.projectimmerse.com\/blog\/wp-content\/uploads\/2019\/10\/distributed-systems-in-one-lesson-system-architecture.png","width":750,"height":965,"caption":"Distributed Systems in One Lesson"},{"@type":"BreadcrumbList","@id":"https:\/\/www.projectimmerse.com\/blog\/distributed-systems-in-one-lesson\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.projectimmerse.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Distributed Systems in One Lesson"}]},{"@type":"WebSite","@id":"https:\/\/www.projectimmerse.com\/blog\/#website","url":"https:\/\/www.projectimmerse.com\/blog\/","name":"Project Immerse","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.projectimmerse.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.projectimmerse.com\/blog\/#\/schema\/person\/c53f2864be524ee6fa08a7e4800dd1e5","name":"projectimmerse","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/4d06955033d6227bfdcf30014e457e4334f7deeb73907de49b65ec2484921931?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/4d06955033d6227bfdcf30014e457e4334f7deeb73907de49b65ec2484921931?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/4d06955033d6227bfdcf30014e457e4334f7deeb73907de49b65ec2484921931?s=96&d=mm&r=g","caption":"projectimmerse"},"url":"https:\/\/www.projectimmerse.com\/blog\/author\/projectimmerse\/"}]}},"_links":{"self":[{"href":"https:\/\/www.projectimmerse.com\/blog\/wp-json\/wp\/v2\/posts\/824","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.projectimmerse.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.projectimmerse.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.projectimmerse.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.projectimmerse.com\/blog\/wp-json\/wp\/v2\/comments?post=824"}],"version-history":[{"count":49,"href":"https:\/\/www.projectimmerse.com\/blog\/wp-json\/wp\/v2\/posts\/824\/revisions"}],"predecessor-version":[{"id":886,"href":"https:\/\/www.projectimmerse.com\/blog\/wp-json\/wp\/v2\/posts\/824\/revisions\/886"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.projectimmerse.com\/blog\/wp-json\/wp\/v2\/media\/1096"}],"wp:attachment":[{"href":"https:\/\/www.projectimmerse.com\/blog\/wp-json\/wp\/v2\/media?parent=824"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.projectimmerse.com\/blog\/wp-json\/wp\/v2\/categories?post=824"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.projectimmerse.com\/blog\/wp-json\/wp\/v2\/tags?post=824"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}