We’ve worked a lot with Spring Boot while developing the courses and upcoming book. The speed of development and ease of use is at a different scale altogether.

With that in mind we decided to do a live migration to a Spring Boot server that would work as a drop in replacement to app engine. To keep scalability in place we decided to use cloudflare (as we do on the main website). It makes scaling remarkably easy.

Because we already extracted the build UI from app engine in the past and now host it as part of our website, we have a clear API to the backend server. With that we could change all the calls that go into the app engine server and just point the exact same calls into the new cloud server. The cloud server decides whether it should handle a call locally or ask App Engine to perform the call for it. In this way we added a new layer for existing users, but it should be 100% compatible and shouldn’t fail.

The reason we had to take this approach is due to the plugins, since they are installed on the end user machines some might still point at the old App Engine. We don’t want all builds to break suddenly. We’d like users to move away gently from the old App Engine deployment.

Normally rewriting from scratch would have been easier but because we wanted as little disruption as possible we tried to setup the new Spring Boot server to be 100% compatible so we had to import code and try to convert the old mishmash of servlets with no tiers to a proper architecture that separates tiers properly. It still looks a bit messy as the original wire protocol is messy. But now that this is behind us we’ll be able to revisit this and improve a lot of the missing pieces.

A lot of things failed in unexpected ways.

The biggest problem we ran into in production was due to cloudflare. Cloudflare blocks DDoS attacks and one of the tricks it uses is blocking all traffic that doesn’t include the User-Agent header. A lot of our code uses the URL class and doesn’t add that header. So basic things worked great in the development environment but failed in production.

We migrated a lot of the old numeric keys as we moved from the datastore to SQL. So the new String based keys worked reasonably well but a couple of JavaScript client side functions used syntax such as: doThis(id).

This seemed to work but just failed silently when the key was a string. We had to add quotes to those methods to get it through. Since there are so many nuanced features in the platform we just missed testing this.

Some OTA and install features failed in production. This relates to relatively hidden/obscure code in the app engine implementation that we completely missed. In fact we used JSP for a couple of features in the old app engine. While Spring Boot supports JSP it has some deployment limitations so we just translated that code to a standard webservice call. Not an ideal solution but since there was so little code to port it was something we could address.

After working a bit Spring Boot would fail with RAM errors. Turns out we need to explicitly set the Xmx/Xms flags in Spring Boot using a conf file. We didn’t run into it in previous deployments as we didn’t do some of the heavy IO that we do here.

So many things “just worked”!

The migration to SQL was mostly smooth and included only a few pitfalls/schema changes. Being able to use proper SQL instead of datastore is a huge step forward. It’s so much faster we don’t need memcached and get better performance to boot!

The truly amazing thing is the queries and 3rd party tools. This has boosted our ability to address issues tremendously.

Despite the User-Agent issue cloudflare is a huge asset. It makes caching repeated queries trivial. Better yet, since we now proxy some downloads through the servers we can get faster/more reliable downloads thanks to cloudflare.

I can’t sing the praises of Spring Boot enough, it makes this trivial. It has it’s pain points (unreadable huge stack traces) but the ease of development is amazing. We manage our own infrastructure now through IaaS. It’s easier, faster, cheaper and scales better than the previous PaaS deployment. Four out of four criteria.

While we didn’t test scaling to the full extent so far CPU utilization is flat/low. This architecture would probably scale much better than app engine ever did. Google sells App Engine as a “Google Scale” solution but anyone who worked with it will know that this only applies if you can spend “Google Sums” to pay for that. App Engine tries to scale by adding computing resources instead of just slowing down.

That means that if you have 10k active users you’d pay for a lot of servers to handle them. Our current solution can handle 10k concurrent users easily. It would slow down but wouldn’t crash. It would still be cheap thanks to Linode.

We also took the opportunity to move most of our transactional emails to mailgun. So if you get an email from us you will notice it uses a different domain. One of the big problems developers had with signup in the past was due to corporate inboxes relegating us to spam. We made some bad technical choices assuming SendGrid can help us fix these issues. This probably isn’t SendGrid’s fault as much as it’s our lack of understanding in this field.

We decided to start a new leaf with a new domain for the emails. We didn’t move everything there and I’m not sure if we ever will as I’m concerned about deliverability. Regardless this is seems to be a good move as we have 100% deliverability so far.

I already discussed some of these here. But there is one additional entry in the pile: IPN.

We handle our subscriptions via PayPal. We’d love to add other options but there are logistical issues with global deployments for all of them. We don’t want any billing data on our servers as we don’t want to deal with that complexity.

PayPal billing can work via a solution called IPN which means paypal invokes us on every billing. That way we can update our user database based on payment received. So far so good.

Unfortunately you can’t change an IPN address after it was set. So existing subscribers still point to the old app engine URL. We have a workaround of polling app engine for subscription level for existing subscribers. This only impacts current paying users and might cause a situation where your subscription appears to have 2 days left even if it has more.

We plan to migrate the project in app engine so the IPN will be the only remaining piece and it will point back to our server. To do that we’ll need to bring app engine offline and set it up with a new project. That will take time. So for now this workaround is in place.

We plan to disable app engine builds by the end of the month. That means you would need to update your plugins to the latest in order to build. If you don’t you would get an error. We would still not remove app engine as a few other features are harder to migrate over.

Now that this is in place we can finally implement some cool features we’ve been craving…​ Higher build quotas for free users including features such as push notification etc.

All of these would be coming in the next few months…​

I don’t want to announce dates as I’m still working on the book and we need to push out Codename One 5.0 but it’s coming and hopefully before 5.0 which is now slated for September.