Getting the App Framework off the ground: a product and engineering log

There’s this airport in Berlin, you may have heard of it. In planning for 15 years and with 34 million projected annual passengers, the Berlin-Brandenburg airport will be one of the largest air traffic hubs in the country. Construction began in 2006, and the ribbon-cutting was set for October 2011… then pushed to 2012, then 2013 and 2014 due to complications in planning and project management. To cut a very long story short — it’s still not open.

This is what leadership is afraid of when you pitch them really big engineering projects. The longer the timeframes, the more unknown factors can arise that have the potential to put even more distance between you and your first customer. Even with agile methodologies, it’s still a risk.

How do you even get started developing a major, strategic project? How do you convince management that it’s worth every day of an entire development team’s time, maybe for a year? How do you make the right decisions and keep the project moving forward? These are the questions we had to answer while building the App Framework, and this is our chronicle of technical decisions and product strategies for getting a big project off the ground.

In the beginning there were webhooks

Interoperability with other technologies was part of Contentful’s architecture from day one. Webhooks were added to the platform in the very early days, and customers quickly started to rely on them to realize third-party integrations. We realized that while they were powerful, they often required coding on the receiving end — the data sent by Contentful needed to be transformed to match the format the third-party service expects. 

UI extensions were introduced in 2016 as a way for developers to customize the Contentful web app UI, but at the outset they were tied to individual fields in the content model. This limited the scope of what was customizable and where integrations could be located.

Customers were consistently asking how they could integrate more services with their Contentful implementation. The answer during that time was a combination of webhooks and field-based UI extensions, which worked. But the deeper their need for integration, the more complex the workarounds became. 

New team, new goals

We founded the extensibility team in early 2018 as an answer to this question: how can we make Contentful more flexible, while making it easier for customers to build awesome experiences using that flexibility? 

This is a big question for a brand-new team to tackle. Half of answering the “how” was technical decisions, and the other half was operational. On top of that, we needed to convince senior management that this whole approach was going to fly, and prove it to them.  

We had a good starting position: UI extensions and webhooks already saw great user adoption, which got us initial buy-in from management. Contentful’s strategic bet on interoperable tech stacks versus monolithic suites gave us more fuel for the fire. Improving extensibility was aligned with one of the most important strategic goals of the company, so it got us thinking more broadly about what our team could deliver. 

Because we are developers working for developers, our unsung strength was our honest excitement about making Contentful more extensible. Designing a product for a group you’re also a part of has risks too — namely, having pre-formed opinions about the technology — but that excitement also made it easy for us to be passionate about this project. We deeply understood the user’s pain points and wanted to give users more power to build cool things on top of the platform.

Hack, learn, ship, repeat

We decided on this basic approach to break the problem into two conceptual phases, with a few important practices happening throughout: 

Phase 1: Enable developers to do more with Contentful by adding new features and capabilities to our extensible platform.

Phase  2: Lower the entry barrier by reducing the complexity of the components and packaging them in an easy-to-use way.

Ongoing practices:

• Ship business value throughout by releasing incremental improvements as soon as we collect enough evidence for value.

• Gather continuous feedback from customers, internal stakeholders and analysts and iterate.

• Over-communicate results to management and the team and celebrate successes.

Building Contentful’s next generation of extensibility was the most ambitious thing we had done in a while. Our approach of small iterations, quick learning and constantly delivering incremental value was the reason we were able to ever make it to launching apps — it got us proof of value, it kept the team excited and it got us support from leadership.

Making webhooks more powerful with templates

The first component we tackled was webhooks. To make them more powerful, we wanted to give them the ability to call a third party without the need for middleware transforming the webhook payload. Three things needed to happen on the technical side to make this possible:  

1. Customizing webhook calls: We made it possible for users to specify the HTTP method to be used for requests and declare secret HTTP headers for authentication with third parties. We also allowed the use of non-proprietary content-type headers, like `application/json`

2. Transformations: To eliminate the need for coding on the receiving end of the webhook, we made it possible to transform the default payload to fit the schema that the third party expects. JSON pointer notation would resolve properties to be included in the final payload. 

3. AWS calls: Many of the webhooks our customers were using were for AWS services. Because AWS requires request signing, we added an option to provide the necessary keys and secrets so Contentful can sign the request for you. This made it possible to call over 100 AWS services like Lambda or DynamoDB directly with a webhook call. 

These improvements were great, but the goal wasn’t just better capabilities. We wanted to lower barriers to entry by making these components vastly more usable. We lowered the complexity for webhooks by packaging their configuration in a guided user interface: the webhook template.

What we learned from webhook templates

After introducing templates for services like Netlify, Algolia, AWS, Twilio, Mailgun and Jira, we noticed a significant spike in adoption of webhooks. And suddenly, a big chunk of all webhooks being deployed used a template. People found webhooks much more accessible with the help packaging, prebuilding and documentation. We found that by offering a guided experience, we could make the whole of integrations much better. 

When we reflected on this, we thought about how we can take more parts of Contentful and make customization much smoother, with little to no manual effort. The lightbulb went on when we realized webhook templates were essentially a “mini-app” already. If we could offer end-to-end experiences for all integrations, we’d basically be building apps. Inspired by the success of webhook templates, we set out to take this concept to the next level.

Opening up with more UI extensions

Before we could create apps themselves, we needed to open the Contentful platform to more points of integration. Based on the existing field-level UI extensions, we started creating more “locations” — custom widgets in different parts of the web app UI — where devs could safely hack on the platform and customize the UI experience.

1. Dialog window: With this addition developers could use the SDK’s `openDialog` method to create an overlay over the web app for focused work and rendering 3rd party integrations.

2. Entry editor sidebar: Previously, all entries used the same predefined sidebar that couldn’t be customized. To allow admins to rearrange and hide default widgets, and also embed their own custom widgets wherever they please, we extended the `EditorInterface` entity. This entity describes how the entry editor for a content type will look, so it made total sense to evolve this API. We added a new property, called `sidebar`, that allows admins to define order and include custom widgets.

3. Entry editorial area: We realized that sometimes for a successful integration, the entire editorial experience needs to be swapped. To do this, we added yet another `EditorInterface` property for defining a custom widget to be used as a replacement for all editorial widgets in the entry editor. This allowed for the creation of highly customized and well-crafted interfaces. We later used this mechanism to create an advanced experimentation UI for the Optimizely app.

4. Page: Just when we thought we were done, it dawned on us that some organizations may need to create a completely separate page within the Contentful web app, for example, for dashboards or reports. We added a new location, simply called `page`. Developers could now build pages with full routing and navigate to them through a fixed URL or using the SDK.

Extension definitions: It’s not an app unless it scales 

On top of making UI extensions more far-reaching, we knew we needed to make them truly scalable to earn the title of “apps.” UI extensions lived on the environment level — that made managing them at scale difficult. You needed to keep track of all spaces and environments where you used UI extensions and synch both code and config every time you did an update. This model was fine when UI extensions weren’t widely adopted. But when we saw organizations using multiple extensions in 100 different environments across many different spaces, we realized that sharing and managing them at scale would quickly become a deal-breaker.

The solution was to allow users to define “blueprints” for extensions at the organizational level, and just register an “installation” of that extension with environment-specific config in the individual environments. With this new `ExtensionDefinition` entity, code and deployment would occur once on the organization level, and an artifact created this way could be distributed across an entire organization. Updates to a definition were then automatically propagated to all its installations without overwriting environment-specific configuration. 

With one source of truth for the code living in the definition, extensions were now vastly more scalable, and sharing was as easy as sharing a definition ID. To make sure that access levels for managing app definitions were fine-grained enough, we introduced a new organizational role called “developer.” This user would be able to create an app on the organization level without gaining all of the rights of an admin or owner.

Taking it to the next level: Apps alpha 

With all of these improvements, we were ready to take our shiny new feature for a test run. When the apps alpha went live in 2018, we took the approach to ship new apps to our test customers as quickly as possible. The goal was to hack on the web app, get fast feedback, verify our assumptions about product needs, learn what works and what’s missing, iterate and move on. 

For this reason, apps released in alpha were only built by Contentful. We used internal APIs, hardcoded some things in the web app, and, of course, used all the other publicly available mechanisms for extensibility. Thanks to this rapid iteration, we identified features we still lacked: a way of implementing an installation screen, control over assigning apps to various locations in the web app, complex and nested data structures as configuration parameters, and many more.  

Based on customer feedback and an internal survey of our solution architects and sales engineers, we chose the most highly demanded use cases to test during the alpha. We tried to answer the following questions for each app: 

1. What integration needs do users request? What is most valuable to them?

2. What features do we need to add to our extensible platform to enable each use case in production?

For example, Netlify was the first alpha app we built and tested. Normally, to use the Netlify webhook, a user would have to copy and paste values by hand. At scale, this becomes really painful, so we fully integrated the configuration and installation process. In the end, all that was necessary was to authenticate with Netlify using OAuth, and everything else happened under the hood. We also created queues for status updates. When you build a static site, the build process takes time — the queue shows you when the build is done and you can directly navigate to it in the sidebar.

Netlify config in the web app 

Final tech and usability improvements for App Framework launch 

After running the alpha, we made some final adaptations to the technology, especially concerning usability, to make it truly worthy of the “App Framework” title. Our goal was to package everything as sleekly as possible so that all customers who wanted to build on top of the platform could — not just the ones who had the time to hack around. This is what we tackled.

UI extensions SDK 

This SDK controls what developers can do with the Contentful UI. Beyond just exposing new locations, we wanted to make the SDK more powerful for developers by exposing other useful functionality such as the `slideIn` editor, prompts, alerts and notifications for the creation of entries and assets.  

API improvements 

To make the APIs ready for prime time, we had to eliminate all the hardcoded logic and functionalities from our codebases and treat all apps as something external to Contentful’s main infrastructure so they could potentially be built by our customers too. We decided to keep APIs private for the time being as we do more discovery through our early-access program. 

Even though we achieved our goal and implemented all beta apps as extensions with no hacks in Contentful as a product, we weren't satisfied with the extensions APIs. There were more and more differences between apps and UI extensions, and we also anticipated that these two concepts would grow even further apart in the future. We made the decision to introduce new entity types: `AppInstallation` and `AppDefinition`, superseding `ExtensionDefinition`. The introduction of these new entities and endpoints allowed us to clean up and simplify the APIs and SDKs for building apps.

Packaging and usability

Once the API changes were done, we knew we were at the last mile of the race. There were a few final steps we needed to take to make our App Framework easy for everyone to use. 

We ported all apps to use only the new solutions and ensured their quality. We worked on 15 apps as if we were external developers; any of the apps we’ve now released could have been built in exactly the same way by someone outside of Contentful. We published the apps to our newly built marketplace along with extensive documentation written by the engineers themselves. These apps, which are also open-sourced, are also meant to serve as blueprints for similar use cases and inspiration for custom implementations. 

To streamline the design of new apps, we enabled the usage of our Forma 36 design system through the `create-contentful-extension` CLI, making it easy to build and deploy apps that have the Contentful look and feel. 

Button styles in Forma 36

The team vibe saved the day again and again 

In the end, this isn’t just a technical story, or even a product story. Keeping a team happy and motivated while working on a huge project that may literally take years is no small feat. The team vibe can be the difference between smooth sailing and a project slowly running into the ground. Here are a few things we did to keep the mood up in tough times. 

Major-minor roles: We set up our team for collaboration, with everyone doing a little bit of everything needed for the project. On top of their main role, each engineer had a “minor” role like writing documentation or doing screencasts for tutorial videos. They got deep into helping the user be successful beyond just building the apps themselves. 

Keeping it civil: Especially when we were building the individual apps, things were moving really fast. There was a lot of pressure, some moving priorities, and a release was always happening. People were also very passionate about the project and had very heated discussions — which is a great thing — but sometimes we stepped on each other’s toes. Our product designer came up with the concept of “wearing the red hat,” a friendly shorthand to signal when someone was a bit out of line. This gag worked surprisingly well, and we now have a ceremonial red hat (a “keep Austin weird” hat we got at the Atlassian hackathon) that we take along to meetings. 

Overshare: On the one side, we gathered as much information and customer feedback as we could and shared it within our team. Customer feedback kept the fire burning. On the other side, we took every chance we could get to be vocal about the results of our work with the rest of Contentful: Presenting at company town halls, evangelizing our successes to leadership, or celebrating with our cross-functional collaborators when apps went live. This gave us buy-in and purpose within the team, and got us a lot of trust from leadership. 

After the launch

It’s been just a month since the launch of the App Framework, and we’ve seen great initial adoption of apps. On the engineering side, we’re the most excited to see what kind of amazing things people will build now that we’ve given them even more room to play. We’re now also working on backend apps that will enable more complex workflows like updating search indexes or synching data between multiple systems. The past has shown us that the more we open up the platform, the more creative, unexpected and powerful implementations developers will create. 

Start building

Use your favorite tech stack, language, and framework of your choice.

Subscribe for updates

Build better digital experiences with Contentful updates direct to your inbox.