Presented at Decoupled Drupal Days 2018 in New York City, this session analyzes how Chromatic and Meredith Corporation evaluated the benefits of decoupled architecture for Meredith’s brand portfolio. Meredith, who owns and operates a slew of media brands using Drupal to serve tens of millions of users per month, needed a platform that could accommodate all of the brands’ varying needs. After close examination, we concluded that a decoupled architecture was the best solution.
This session explores Meredith’s intended goals, the proposed solution, and some unique challenges encountered along the way, as well as how we addressed them.
Transcript
[music]
Alfonso Gómez-Arzola: Well, good morning, everyone. Thank you very much for coming here. Thanks to the organizers for letting us speak today. My name is Alfonso Gómez-Arzola, and I am joined today by my partner in crime Adams Zimmerman architect Chromatic. I'm Alfonso Gómez-Arzola as I said, a front-end designer and developer for Chromatic, and we're also joined by John Reynolds, Director of Software Engineering at Meredith. Unfortunately, Joshua Stewardson couldn't be with us today, but he was instrumental in putting this together, so, thanks to him for that information. A little bit about Chromatic, we're a world-class, fully distributed digital agency. We've been helping large organizations solve work-related problems for over 10 years and we're made up of a diverse and distributed team of senior [unintelligible 00:01:02] professionals. Our areas of expertise are design development, dev-ops, and even beyond that. We're excited to enter the decoupled space and this project, the multi-tenant project has allowed us to do that.
John Reynolds: Good morning. I'm John Reynolds and Director of Software Engineering at Meredith. Let me tell you a little bit about Meredith Corporation. Meredith is a company that has its roots in print publishing. It is a magazine company, it is a digital publishing company with the acquisition of Time Inc earlier this year, we're the largest magazine publisher in the US and some of the flagship brands are Better Homes and Gardens, Martha Stewart, Entertainment Weekly, People [unintelligible 00:01:52] portfolio and in digital space, we have [unintelligible 00:01:58] which is top food site on the internet and which was spun off a new magazine, interestingly enough. I didn't mean to do that.
Okay. The way to think about Meredith is as a brand consolidator. A lot of the brands which are targeted to a female audience have been acquired over the years. For example, I've actually come from Martha Stewart originally, our operations were acquired by Meredith in 2014, and the portfolio changes frequently. Brands are acquired, brands are created, brands are sold off or brands who are can be shuttered entirely, but we are first and foremost, a brand company, as well as a media company. The basic value of a company that is consolidating brands is that we can run the brands more efficiently together than if they were run independently. Going into our problem statement.
With the variety of brands, we managed, they were acquired through various means. We ended up with a lot of different tech solutions for a lot of sites that had outwardly to the end-user a lot in common and even to the editorial experience. We had a wide divergence of features and even wider divergence of site formats on the front-end. Incompatibility with upgrades would be a problem when, for example, security patches need to be applied, it becomes a headache to have to do it to multiple individual stacks. Automated testing is always a challenge with Drupal, ended up with production bugs and a lot of inefficiencies and chasing those down.
I think there's probably a problem that any company that manages a lot of sites. Some sites take favor over the others when it comes to product development, bug fixes because frankly, they make more money. The revenue-generating sites would get more attention and other sites would tend to go to seed and this created a problem that would eventually bite us on the back-end, obviously.
The multi-tenant platform was conceived as a solution for this and what was so great was that we had the full support of the leadership team at Meredith to draw down some of our ongoing feature development on the brands and divert more of the dev resources to this amazing idea, which was, let's consolidate all of the sites under the same platform and let the brands diverge in terms of styles and fonts and brandy things, but all of the sites basically have their version of an article, they all have their version of the slideshow, they all have their version of a recipe, why do we have different content schema for all these different things?
Let's consolidate on the data schema level, let's consolidate on the page structural level, and conceive one platform that can drive all the services efficiently and be managed by one dev team. The goals of multi-tenant platform when we set out strategizing with Chromatic on how we're going to architect it. We wanted to modernize our code and our practices and be best in class, so, we had a lot of 777 in the portfolio, Better Homes and [unintelligible 00:05:25], Adobe experience manager. We had a couple of odds and ends.
We wanted to bring everything up to speed and do everything. We're committed to open source, we do everything best in class, modern, even the things that were unfamiliar, we wanted to dive into the waters and we were given the opportunity to do it. A consistent feature set to ensure long-term viability. Again, this goes to, if we've got a set of content structures that are uniform across brands, but differ only in the content within them, let's enforce that consistency, and that's going to serve us better in the long run. The support for flexibility brand to brand really comes in the form of at the front-end level.
In a decoupled model, the branding is squarely on the front-end, so, it would allow us to have a more uniform CMS platform in terms of the schema, but we could diverge at the brand level in terms of CSS, HTML, in ways that made sense and didn't violate the basic premise of consolidating into a single platform and let's bring automated testing in for real. Drupal 8 obviously afforded that because of PHP and it made it easier than simple tests ever did to testing in Drupal. Okay, this is the 3000-foot view of multi-tenant. It's a decoupled Drupal stack with node express on the front-end.
By continuous migration, what we're talking about is, as we onboarded sites into this platform, we didn't want to disrupt the end-user experience or the revenue streams coming in from ads on the sites. We needed to devise a way to iteratively move the sites onto the new platform, content type by content type was the model we chose while continuing to serve content that wasn't ready to be surfing the platform on the legacy sites. We were continuously iteratively migrating toward the new platform. It became part of the front-ends job to proxy requests that were still to be routed to the legacy platform by simply sending them to that legacy platform and passing the response straight through, back to the browser.
The first step was just to put that in place, have all the requests for a given brand route to express, and just pass them through.
Then, as we began to bring this content over onto the new platform, we would start serving that URL from the new stack [unintelligible 00:08:07] that URL from the new stack, but the default would be serve it from the old. As we begin to serve content on the new stack, what we had built was the Drupal 8 backend with varnishing front, caching API responses. Again, express in addition to actually serving the front-end to the end-users became the traffic cop that would route the request from one stack to the other and obviously, as we progressed, more and more requests would be routed to that end, that multi-tenant stack. MT means multi- tenant. We just call it things by acronyms, assume that everybody knows what they mean. Okay. This is just a picture of the whole thing. This is the 30,000-foot view of what we set out to build. I'm going to hand it back to Alfonso now to talk about that express layer.
Speaker: Right. The express application consists of two major parts. One of them is a core library, this is where the vast majority of our code and effort goes into. It's highly configurable, it provides default middleware, controllers templates, and also a component library. This component library is very useful, it allows design and development to speak the same language because they're looking at a component library that's actually published to a website and so, they can take a look at what components they have to work with when they need to work with a new content type or a new type of page or just a new element on existing pages. They can take a look at those components and work with real live components that are easy for developers to incorporate into an existing template or a new one. It also allows a lot of overrides. This is done both through config and also via convention, so if a brand site, a project that is consuming this library has a middleware directory, for example, then, any middleware in there that is named after the middleware inside of the core library just takes over and overrides that. This core library, when invoked and required, it builds an express application with all of these configurations and overrides incorporated into them and then just returns a clustered server that's already running. The other part of this, the brand applications, all they need to do is require this core library passing in their config and their overrides, and it's good to go.
They can also consume the core library styles and override and build on them if needed and most of them do because they have different colors, different variances and spacing, and other kinds of stylistic things like that. Very few of them need to override certain middleware via convention through that middleware directory that they get to write. With that, now let's hear from Adam, who's going to talk to us about the Drupal side of things.
Adam Zimmermann: Yes. We went with Drupal 8 for the backend as you know, many of us have. We used the composer base workflow to get the whole thing scaffold up and set up. We built a large library of shared modules, so, we put all of our custom functionality and each of these separate modules based on what they do. We bring those all into each brand's repo with composer. The beauty of this is just like on the front-end, on the back-end, there's very little custom code needed for each brand and we can scaffold up new brands, it helped me with Meredith business needs easily with that. Of course, we went with Jason API for our API layer.
I know many of us are talking about graph QL this week and I think honestly if we had a re-pick, it'd be a very tough decision between Jason API and Graph QL, but at the time, there was a lot of community support behind Jason API and there still is, so, I think it'd still be a very tough decision. We have done some cool things with custom normalizes in Jason API that I'm not sure you could do with Graph QL, so, six in one, half dozen in another. One of the coolest things we did, like the previous chart alluded to is, we put varnish between our [unintelligible 00:12:42] application and Drupal. That's really helped decouple it in a way.
When Drupal is going through a deployment and presenting an Air 500 or something goes wrong in Drupal, we have that protection and we use saint mode and grace mode and varnish, so, if there is a problem, the front-end will continue to just living on making those requests to Drupal and it doesn't have to know anything about it. Of course, that keeps the server load on Drupal down greatly too, which is another huge win.
Last thing that I think was really cool is because we had the brand buy-in to consolidate features and functionality in our data model, we were able to run all these brands off of one big folder of configuration [unintelligible 00:13:28] which was a huge win for us. Not having to be like, oh, did we get that field on this site, the field and this site it's all in one spot and brands all just import from that one folder of configuration and because we had the brand by-in and also, we just had-- we had the very limited set of things we truly had to override per brand, like some API keys or app IDs, or what have you, we could just stick and settings up PHP and not have to worry about which was wonderful for us because there's so much that needed to be configured.
Separation of concerns, this is something we all talk about in software development and we all like to do, and it's obviously a key tenant of decoupling. We want to separate the concerns of what handles what. Express does the web stuff and Drupal just does content and they're blissfully unaware of each other and everything's perfect, right, but, of course, that's not quite how it goes. Unfortunately, Express is trying to serve a website and Drupal happens to be really good at serving websites and express doesn't know everything it needs to know to serve a website.
There's all these gray areas like meta-tags, structured data, menu items, or data coming from external resources, where should that stuff live? So, that was something we debated a lot. Then also, URL aliases, where should that stuff live. Here's how we kind of answered that question a lot of times you said, is this something other consumers of our API aka Content or Drupal will need? If we determined the answer was yes, we could-- most of the time we could justify saying, yes, this is something Drupal should handle. It's not just truly a web thing that we're polluting by putting into Drupal, this is part of the content that Drupal needs to serve, so, we're not just serving a website or serving an application too potentially and it will need this data.
For example, we have recipe nodes in our content and we have a recipe ID on some of those-- on all of them but the recipe content isn't actually in Drupal, it's in another external system. We added a dynamic field to the recipe entities and when they're rendered out through the Jason API, it requests that data caches it so we don't have to do it every time and it just sends it all out to the front-end gracefully and the front- end doesn't know where that data from or that it wasn't there to begin with and it can just consume the content and any other consumer could also consume the content when, or if Meredith has applications to go with these brands.
Same thing with video data, just to give you another example. If we're using [unintelligible 00:16:12] for our video providers, so we have an ID, but we don't store all that stuff in Drupal because you just don't want to store all that, but we can just send pipe that through Drupal to the front-end all through Jason API with some customer normalizes and it just goes through and it's wonderful. The last thing is the content alias. This was something we talked about a lot of, where does URL alias assignment live? Is that part of our content, is that a web-only concern?
It came down to a couple of things, like, if another application wanting to use our content, they might want to link back to the web for that same piece of content, so, ultimately it needs to be in the central place. Also, what is the alternative to this? Drupal is really good at managing content aliases, It has path auto, you can use all these incredible tokens, if you're building a custom alias, you can use your field values to construct it and trying to recreate all that in some microservice outside of Drupal, it seems like a lot of work and money for the sake of content separation of concerns just to meet that. Decoupling is hard as we've all been talking about this week.
Alias to UUID, so, once we decided we were going to keep the alias in Drupal, the front-end, all it-- when a request comes in, all it says is I have an alias, I don't know what to do with it, how do I get my data, does this go with the node? I don't know. Jason API is incredible, but you can't just say, here's my alias, now, give me the piece of content. Jason API wants to know your UUID, the node ID, the node bundle, all that stuff before I can make a request. We built a custom tool called the alias lookup utility that you can send an alias in and then it'll respond back with something like this.
Now, it should be noted that after we built all of this, there's been a really cool tool that the community seems to be coming around called decoupled router which does many of these same features and I think it's the direction we would take if we did this now, but I think it's still worth discussing some of the issues around this routing in a decoupled architecture. There's also the issue of redirects, where are you going to manage your redirects? We also had a very firm mandate from SEO that all redirects had to be done in a single hop. We had to get HTTP to HTTPS, we had to enforce our trailing slashes, we had to do multi-pop path redirects all in single 301, and we couldn't do multiple pops.
Drupal handles all that for us. You send it the alias that you got and you could see from the request. That's what you send into Drupal and it looks up, it says, hey, is this associated with the node, is this associated with whatever, do I have any redirects for this? It uses the redirect module and it responds back and says, yes, here's the proper alias, you should do a 301, and here's what you need to know about this entity to look it up with Jason API. Here's another example of, if you send the proper alias, that'll give you the same information, but it'll say, yes, this is a good, this is a 200, so, go ahead and make a request and serve the data. Here's what would happen if you send it something, Drupal will say, well, I don't know what to do about this alias, so, 400. With that, I'm going to pass it on to Alfonso who was going to talk about what we do with all that JSON data when they get around the front end.
Speaker: Anybody who's worked with JSON API, can tell you it's a very powerful spec, it can quite eloquently express a wealth of data model complexity. It's very good at what it does, but it's also extremely verbose. A single request with related entities, I guess, we would say, for example, a story if you're using paragraphs module, you would get for every block in your paragraphs, in your story, you would get a related entity for it.
Then any one of those can have images, right. You have a media entity that's linking to an image, that's linking to a file. All of that can be included in the same request or in the same response but then what you have is this really massive JSON API compliant response. [clears throat] This makes things very difficult to parse to deserialize, manipulate using related objects. It requires iteration of nested arrays, which, as anybody can tell you, is very scary to do right.
Because it's very easy to put yourself in a situation where you're just going in an endless loop. All of this essentially boils down to JSON API is not a suitable application data model spec. It is an API [clear throat] data model spec. This needs to be translated into something that an application can use consistently. What we came up with was the JSON API data mapper. It's a tool we use internally and, in this diagram, we're looking at it as if it was sitting in between the front-end application and the JSON API.
In reality, it's not. It very much behaves this way because the first thing we do after we get that response is put it through this JSON API data mapper. It essentially accepts a schema and an object that would be the response from the JSON API. The schema literally just points certain property names that we define internally in the application to certain JSON API fields.
We can very easily say this deeply nested field that is from this file linked to an image, linked to a media entity, linked to a paragraph, linked to the body of the story.
We can just reach in there and grab that and just put it at the root of this object if we want to. This means that there is a single point of translation between JSON API data and your application data model, which is very powerful, because then in the future, if anything changes with the API, chances are you're going to have to change it in one place.
That is unless the nature of the data is changing and you're going from, say, like a category field that is just a string, to a category, field that's an object with an idea and a path and a bunch of other stuff.
Then, of course, the rest of your application is going to need to adjust to that, but even then, the fact that you're not having to look up things in this massive JSON API object is very, very powerful, because you no longer have to do this dance where you say story.category, does that exist? Oh, great. Okay. Then let's say, JSON.category.id, does that exist and does its name exist, and does its path exist?
You can just do all of that from the schema. The schema will just do it for you. [clears throat] Then the application only concerns itself with the application data model that you wrote. Some of the other benefits of this is that you can have typed schemas and that's what we have. Whenever a story has a paragraph, we have a schema that is dedicated to text blocks. We have a schema that's dedicated to image blocks.
An image can have up to three images, each one of those images being an entity, each one of them with an image, each one of them with a file. All of that, each one of those has its own schema. No matter what context you find an image in, it's always going to have the exact same data or data shape, it's going to have the same form. Whatever the context, you'll always have the same form.
You're able to enforce contracts. From the schema, you're able to say, well, if the JSON API data does not include property XYZ, then we're just going to feed in this default object or this default value or make sure that it's null and not undefined, whatever it is that you need to. That's the point where you get to enforce contracts so that your middleware controllers’ templates, they all get to rely on certain field objects existing and being available.
You don't have to run so many checks every time you want to use a piece of data; you don't have to say thing.property does that exist? Thing.property.sub-property, does that exist?
You don't have to do that stuff as you can force it in your schemas. You can also have virtual properties that are calculated automatically, so very much like an ORM where you get to say maybe your data has a field for a first name and a field for the last name, that’s a classic example, but you really, really want a field just full name that just [unintelligible 00:26:05] those two.
You can enforce that or implement that in the schema so that even though the API data does not contain that information, your internal application data model does. You can also do post-processing. We did this a whole lot where the data that we get from the JSON API is fine, but we also want to run this little function that's going to transform that data a little bit in one way or another due to business cases.
In other words, it really, truly decouples your application logic from the API data spec, which is a very, very positive thing because you don't want your application data model to be dictated by the APIs data model. By the way, if all of this is sounding a heck of a lot like it could be taken further, I have, inspired by this work, I decided to just take a couple of weekends of hard work and started working on this thing called JSON Monger.
This takes it to another level. This is basically an ORM for your JSON API server. It doesn't just read and allow you to decide, here's my schema, here's what my application model is going to look like. It does all of this in memory. It keeps that data mapping live, and it allows you to commit changes back to the JSON API. In the case of Meredith, we're not really doing any of this. We're just consuming this data.
Applications need to also write data persistent to the API. JSON Monger allows you to do that. It's still a work in progress, but we're very excited about it. I do recommend that you guys check it out because it's something I'd love input on. Oh, sorry. Now, we’re going to pass it to John.
John: I wanted to take not too much time just to talk about a topic that doesn't really have so much to do with the decoupling aspect of this project. This is more of a shell shock we encountered going from D7 to D8. Probably a lot of people encounter these kinds of things, but I wanted to call them out, even though it's not strictly decoupled, it's loosely coupled to the topic of decoupling.
[laughter]
Speaker: Ouch.
Speaker: What we had going in the legacy Meredith ecosystem was a lot of sites and we had Multisite groupings that were logically done at the time, bringing brands together that had common editors. We had reasons to bring certain groups of sites together. All the Martha Stewart sites were on one Multisite.
We had a particular multisite with about 11 sites on it, two of which really only ever got any attention from developers because they were the high revenue earners. Some of the headaches that arose from multi-sites, I think that we came to the conclusion and there's lots of posts online that say something like multi-sites, does it work in enterprise applications?
My sense was no. A particular headache that we always encounter had to do with updates because we would get security patches for modules and core that were really important to do on the sites that we cared about the most when we'd do them on the sites that haven't done them, we haven't touched that codebase in a while.
A lot of custom code and some of the smaller brands gets old and we would update something and it would break. Now we've got sites that need these updates that are being held hostage by the sites we care less about. In some cases, we would end up splitting the sites we cared about more onto their own Multisite platforms so that we could not update them. Anyway, it wasn't great that it didn't feel or smell right in terms of a way. When we were approaching this, we were like how can we solve this if we’re developing a common platform for sites where we avoid this problem either by making it so sites don’t break when we’re updating common elements or allowing ourselves to update certain sites and not other sites, without incurring a lot of tech data and opportunity cost later on if we have to like suddenly [unintelligible 00:30:21] we're putting all kinds of developer weight into this site that doesn't make much money, only so we can update these other sites. The promise of D8 brought with it the promise of Composer, which was very new to the team and new to me.
What we discovered right away was semantic versioning really solved this problem for us because as we develop internal code that we keep our in own repositories and source via Composer versus putting it in your site’s, all directory, multisite, we could change it, iterate on it and increase the version number on it. That custom code could be inherited by sites that we're ready for it, but not inherited by sites that weren't ready for it, simply by not incrementing a version number in the site's composer of JSON file.
Semantic versioning for the win in that respect. Also, the promise of, when we were developing modules, when Mark Dawson and I was at Martha Stewart developing custom modules, we would have nerd fights about does this code belongs in a module or should this go in a theme. We would sometimes play the game where you pretend that the module is a contributed module. If it was a contributed module, what would you do? Then you divorce yourself from all of your preconceived notions about your own site?
Well, Composer enforces that by its very nature. If your individual module or code for your platform is actually in a separate repository and you're maintaining it as someone who is maintaining an external project, it puts you in that library mindset and you're going to write more decoupled better-abstracted code. This was also perceived as a win as we started to launch into the world of Composer. Now, some of the pitfalls of all this and tangentially related to the whole Composer thing, config is not like features.
If you folks have worked with Drupal 8 and config and you've also worked with Drupal 7 and features, there's an analogy there that's very common, right? If you're socializing Drupal 8 config with developers who are familiar with Drupal 7, you're going to say, we’ll think about features. You make changes in the database; you export them into code and you've got a feature module that then you can put into another site. It's basically the data state encode so it becomes portable when you commit it to your VCS et cetera.
Well, config is like features in that respect, you make changes to the database, you can export them into code, but there's no equivalent for features update in the world of config. The reason is when you have a module that defines config. I think I'm not talking out of turn here.
Speaker: I’ll [unintelligible 00:32:58] you.
[laughter]
Speaker: Okay, Dan will tell you if you are. That module is going to provide you with an initial config. That is going to be when you do your config import is going to be poured into your database and it'll be present in your database. You can export that config; you can make changes and export it. It's going to export where to your site’s config. The config is now owned by the site, no longer owned by the module. You can't get that config change back into your module, not gracefully. We ended up doing things like in the service of trying to maintain those internal libraries.
We would export the code into the database, make changes and then go into the config directory of the site and manually take those config changes, put them back into the module, commit the module back to the repo.
It became an unsustainable developer flow and it became known as the config thing. It's like we're going to talk about the config thing again, but a task force for the config thing. I could tell you where we ended up. I don’t think it would work for everybody, but we did the most dead simple thing after looking at solutions like Nimbus, which is something you can search for if you have this problem. Seems like a nascent project that looks to be able to make config exportable back to modules.
We're talking about running our own config filter to accomplish the same thing. We ended up just having a module that was in its own library. It's called Base, which is called the base module, and it has the config for all the sites in it. When we bring it in via Composer into each site, we just configure the sites to use that modules folder as the site's config folder. It makes sense? As we're making changes in the database we export to config, to the site's config folder.
It happens to be the one used by the module so we can commit the changes right there and push them back up to the repo. That way we can truly share the config around all the different sites, which frankly, I don't think Drupal 8 really wants you to do because Drupal 8 config has a site [unintelligible 00:34:54] that itself is a problem, but in our particular case, it works because we truly want to lock all these sites down to the same config.
Any exceptions we do, we do in settings.php. We have a lot more settings.php config overrides than we would otherwise, but like 90% of our configs is truly shared by all these sites, so it works for us and it was like a little bit of a face [unintelligible 00:35:19] but it totally works.
Speaker: Well, it worked after we realized that every [unintelligible 00:35:23] truly had the exact same data model. We didn’t know that at first.
Speaker: When we can verify that, then we could exhale. Again, I don't know if this would be a universal solution.
Speaker: Nothing is.
Speaker: Nothing is. Drupal doesn’t want you to do this. Drupal wants the site to own the config. We're trying to build a platform that supports multiple sites. I hope we're not violating Drupal 8 stuff, or maybe it can evolve more into that direction. Version numbers during development I think are always a challenge. Somehow the JavaScript community seems to have this figured out. They've been using a [unintelligible 00:35:57] package like JSON and they source packages in and they update their version numbers.
We were getting tangled with every change to a library incremented the semantic version number. We have to then update the JSON file on the web project and we can never keep them in sync. We ended up just going during the sprint, we would just call a developer and we just had developers, the name of the semantics we would use and then only when we were redeployed, we would actually update with individual version numbers. It was I characterized as a pitfall but it was really more of just a thing, a little additional cost we had to incur for all the benefits I think we got by embracing Composer.
I’m talking way too much about this because it's not strictly decoupled, but it was of interest to me. This is the lay of the land in terms of external and internal dependencies are now all mesh to Composer and deployed to all the brands. This is how we do multi-sites now and I think it's better. I’ll let you take this picture.
Speaker: All right. Thanks.
Speaker: This is just a micro illustration of what happened, the baseline library I just mentioned, which contains all the shared config among the sites on the platform, the shared settings.php. Anything, any configurations in setting.php that would be shared across the sites in this file, which gets included by the settings.php file in each web project.
Then shared dependencies. Basically, all of the Composer like JSON dependencies for all the sites live in that one based library. The repository definitions for those, unfortunately, have to be in the website project just because that's just how Composer does it.
It means the dependencies for all the sites are maintained in one place and gets pulled into each web site project, and that's how we're effectively sharing dependencies settings and config across all the sites on the platform. We're going to come back and talk about continuous migration in more detail, so I mentioned this earlier. How are we doing on time? We're doing all right?
Speaker: No.
Speaker: Is it on the last part, right? Oh, then I’m going to talk.
[laughter]
We talked about how do you support two platforms during transition, right? Historically, if you're doing every platform, you would build a new site off in a silo. Six months, eight months later, you would flip the switch and move everything over to the news site. By that time, the product team wants something different anyway. So, keeping true like agile methodology or taking a note from that, how do you support two platforms and iteratively move from one to the other while you're transitioning onto the new platform?
Speaker: It’s decoupled.
Speaker: In a decoupled architecture. The solution and this is something that we've done in the past with other brands, but this is sort of the decoupled twist. You need to maintain a manifest of URLs that are going to be served from the new platform. Drupal knows what it's capable of serving because we tell it. We know articles on this brand are ready to be surfing the new platform, but nothing else is.
We can tell Drupal to deliver all the URLs associated with that article content type to a manifest that can be read by the front end. As I mentioned before, right, express the front end is going to be doing the traffic coupling and it's going to say, is this from the new platform? Yes, surf a new platform, it's not, go to the legacy platform and just send back whatever it spits out. Yes, okay.
Anyway, I have a display that illustrates this. We ended up putting SQS and Redis in the mix and this was our ops team really encouraged us to like stay as decoupled as possible in this model. You could easily just have-- I guess you could have the front end. Just query Drupal directly and say, are you serving this URL now? Are we decoupled or are we not? Let's try to eat our own dog food through the decoupled architecture.
We have Drupal reporting its URLs into a queue, Amazon’s QSQ, and element. I’m sorry. That's a completely different topic. All right, [unintelligible 00:39:56] the multi- tenant front end is pulling stuff from that queue and writing into Redis cache. That Redis cache then becomes the system of truth for the front-end to tell what URLs going to be served from the front or the back end. Does that make sense? So, we've got this intermediary of SQS and Redis in their sort of keeping the two, the front the backend agnostic to each other when they're negotiating through this. Again, Express handles everything. All the requests for a domain are going through Express and it's doing all the negotiating.
Here's an illustration of the whole model. Drupal 8 reporting what it can handle or what the front-end should be handling from it, I should say, reporting it into AWS SQS we've got a simple shell script on a Linux Damon that's processing that queue and Express takes what's in queue writes it to Redis so for future requests for those URLs, they'll be trapped accordingly. What you see here, you're going to tell me if I get this wrong. This is a JSON object that represents what's delivered to express from the shell script?
Speaker: From Drupal on down the shell script.
Speaker: Okay, that's what Express sees and then what it stores the Redis when it gets this is this simple mapping one or zero, is in surf in the new stacker or it's not and its ability to process, they seem to be very fast. We really want to see any latency in this extra trip it has to make to determine where to serve a URL from. All right, one of the great things about being the boss is that you don't have to come up with all the great ideas but you get to talk about them at [unintelligible 00:41:45].
[laughter]
Content preview is a hard problem that decoupled that I think we solved really elegantly. The guy who came with the ideas in the room so if you have questions for him or more specific [unintelligible 00:41:56] can provide them. I'll direct you to him. He's right here. So previewing content in a decoupled architecture, I've got a newly piece of curated content in my Drupal database, I want to see what it looks like but I don't know the theme layer, I've got a front-end. I haven't saved it to the database yet. The data's not even database yet, it's just in my form. If I hit the preview button what am I going to see? Nothing because there's no theme layer, how are we going to get that form data to the front end so it can display it? It's not the database we can't use the conventional API.
Here's what we do. We take looks in the form and we post it to the front end. The front end is they're going to handle that request as if it was getting it via get from the API. It's the same data. It's the same data, so We construct that note object when we hit that preview button, we send it, we JSON code it, we send it to the front end by a post. The Express app is then trained to say post request, "Okay I'll take that post it to data payload and I'll use it to render the page." Guess what, it totally works. I was as surprised as anyone. except, you correct me if I got this wrong. One of the big got you's, there a few got you's, but the one that really stuck out was entity references, because when you save that and you hit save on that no data form we're going to invoke all the hooks that are needed to reconcile things like paragraphs or references to image entities or other entities that are in that node but with this we don't have such a reconciliation.
We have an ID because someone is using any reference field in the form. we've got an ID in the form but all we have is what's in the form. We have to do that reconciliation ourselves; we have to go take those IDs, do the look up, load those entities, JSON code that along with that.
Speaker: We change that a little bit.
Speaker: Okay. They don't tell me everything. [chuckling]
This was the basic problem solutions turned around that and there were others too and we'll be glad to enumerate them all for you but this was really pretty crafty and it kind of worked and I think if you have this problem, I think if you're doing decoupled Drupal and want to support preview functionality for editors, I think there's a pretty lightweight thing to at least experiment with them and see if it's going to work for your site so I highly recommend. We're five minutes short but I'll say at this moment thanks for listening hope you enjoyed it.
[applause]
Speaker: Thank you very much. I guess we'll take questions. Mark has a question. Yes, Mark.
Speaker: Alfonso you spoke about the manipulations that have happen, do happen at the JSON API. John you mention content preview taking that form data and posting it directly to the Express app. What about stuff that normally happens between the API from [unintelligible 00:44:56].
Speaker: We used a serialization content with JSON API so we do an internal build with the JSON API responsible.
Speaker: Do you want to come up? Do you want to come up for, I guess, recorded for posterity?
Speaker: For the recording, yes. Sure, please.
Speaker: This is Neal Bailey from Meredith. [applause]
Speaker: I can't take credit for this entirely because Adam, actually, when we first started this thought process that led to preview he said, “Well, if we're going to do that, look at the serialization feature because JSON API comes with a serialized format as well, which instead of having get request-
Speaker: Part of a symphonies serialization API.
Speaker: Right. What we do is we built some recursion in our custom API that goes through that entities, and built a single payload that includes all of the paragraphs, media entities, and everything in one data set. That goes through the JSON serializer which calls the normalizer, which does all of that extra work that was spoke about. At the end of the day, you get a one for one, you get an exact replica of what would be used in the get request.
Speaker: The front end doesn't have to do anything different?
Speaker: Well, it does.
[laughter]
I think part of your question was getting at is, does that get put through the data mapper? It does. The only thing that the front end needs to do differently is that when it realizes it's not a get request, it's a post request and there's a body associated with that request, then the only thing it skips is the call to the backend, to the API, but it treats the request. It puts the request through the rest of the middleware chain all the way down on controllers, and templates the same way would a get request except it's just that the data came from a different place, and the rest of the application does not know that the data came from a different place because, again, we haven't JSON API data mapper that just makes that transition transparent.
Speaker: Thank you.
Speaker: Yes.
Speaker: I’m just one confused. The title of this presentation was multi-tenancy. Do you have one single code base with one database together, or we have code base and multiple databases?
Speaker: Multiple bases. In that respect, it's similar to what you do with a multi-site. We were afraid, I think, at the time maybe rightfully so, to put all the enterprises content into one sequel database, but the code-sharing is accomplished. We have an individual. For every site there is a separate repository. Every site gets its own repository, but great vast portions of what is actually compiled are shared in libraries that those repos pulling by a composer. Does that make sense?
Speaker: Yes. That means this not 100% multi-tenancy. We have [unintelligible 00:48:11] product where everything’s desired is in one database.
Speaker: Right. As a platform its multi-tenant. It's a platform and hosts multiple tenants, but the way it does it under the hood is by separate databases. You wanted to-
Speaker: Yes. You pretty much said it now.
Speaker: Mark again.
Speaker: You mentioned that you basically got buy-in from all brands to share the same content model, is there never a request or requirement that applies to a single, or small subset of sites that’s not really necessary irrelevant to the others? How do you handle that for those other sites?
Speaker: Maybe if you have come by, but not many, and if they do, we just make sure we archityped it. Decoupled helps you do this. You don't say align left or align right, you say layout one or two. I think we try to just add the data to the data model in a way that it can be additive, and the other sites can choose to ignore it on their front end and just not do anything with it, but if they want to they can.
Speaker: I'm thinking more lime I’m the admin UI side.
Speaker: Like a field that other branches wouldn't want to see, period.
Speaker: [unintelligible 00:49:34] this might not be true, I don't know if a recipe content type is relative to every single one of your sites, but I’m sure it’s very relevant to some of them, and then I was like, “You could just hide that via permissions,” but your permissions are also shared [unintelligible 00:49:51]. Does that come up?
Speaker: No. Not all brands use all the content types, but it was just a decision made by business that they were okay with that because as John alluded to it, we originally tried to put a conflict for each of our content types in that module with it, but then we found while you can keep your code separate of dependencies, you can't keep your data model separate of dependencies because we have highly-- there's a lot of entity references between our data models and trying to keep them separate was just a nightmare to do a new site install.
We said, ''Our content model just needs to live all together, and that's just a sacrifice we have to make given our data model. Maybe, if your data models are not tied together, but ours were, and we had [unintelligible 00:50:36] product.
Speaker: I'm just tech on this. There's a non-technical aspect to this too, there's a cultural aspect. The leadership team is really unambiguous on this. If anything is good for one brand it should propagate to all the brands. All the brands should get all the goodness that we're putting it to a single brand. So, the mission of Multi-tenant, it seems unbelievable, but it really is to enforce that uniformity of features and content schema across the brands.
They really want that. We've poked at this lion and said. ''Really? Are you sure though? Aren't the brands going to start with that? They were like, ''Listen, if one brand wants something, it's awesome. We want that awesomeness everywhere.'' That really is a mission of this project.
Speaker: We were very skeptical at first that they were actually going to-- There's like, ''Oh, well, except for here.''
Speaker: The media world and I'm sure other industries are the same way. You have like, ''Let's consolidate and bring everything together. No, let's let everything go on, let's let the brands be the brands.'' It tends to be that ebb and flow and where this project is definitely, ''Let's consolidate the approach across.''
We're definitely at that part. The pendulum swung that way with this project. If you ask us, we got that answer for how we accomplished that, but if you were to ask our product team, they would say, ''Well, why would one brand get it and not all the brands get it?'' If it's good, let's give it to all the brands.
Speaker: One thing I would add is if you didn't get that answer from your product team, conflict filter is in conflict split. Especially, if you write custom conflict filter plug- ins you can do some incredible things with how you filter either the conflict coming in, pushing it in, and taking out from different spots per brand potentially.
That's what NIMBIS aims to do, but we just wrote a few custom plug-ins because NIMBIS wasn't ready. It was like a whole solution. I think Drupal 8, the configuration architecture scene is improving and I think there's going to be ways to do more of this flexibility.
Speaker: We are using in a limited form not so much of a content model but turning on and off extensions, and you mentioned the config does more than just content. We do use config split for things like migration utilities that are obviously, separate program. They're coming from different sources and migrations are completely different.
Brand to brand, there are some things that are turned on for that brand or turned off. Then, with settings at PHP, you can manipulate some of the config as well. For minor variances, we do that in the settings.php. We'll change up something programmed.
Speaker: We have built in the ability to be flexible anticipating that need, but right now, that need hasn't been articulated, but we're ready when it is. Any other questions? Thank you, everybody.
[applause]
[music]
