My biggest failure so far

It's time to share my biggest failure as a developer.

Some years ago, I signed for the development of an mobile app backend. It was a social network. Other freelancers and agencies were in charge of mobile frontends (iOS and Android) and back office user interface.

When I started, the client told me that the job was to finish an existing app in Python and Django and document it. The mission would take a couple of weeks.

However, the backlog was huge and I ended up working for 8 months on this project. We faced a lot of issues and misunderstandings with frontend developers. Frontend developers have changed two or three times during the development and both existing iOS and Android apps were remade from scratch.

After 6 months, the whole system was almost stable with few remaining bugs so I began load testing on the backend. It was a catastrophe, unacceptable: the app would not even support a thousand simultaneous users.

I worked to solve the performance issues on the backend, using database replication, rewriting queries. The process was slow and costly in terms of hosting. It improved the situation a bit. I guess the app would have need more radical changes though, and working more closely with frontend developers.

The client started being nervous and decided to follow the advice of one of the frontend freelance developer: rewrite the all app using Google Firebase! I have been considered guilty of the mess, hence the client demanded me to take care of the migration and to create a bridge so that the existing back office could still be used. I warned I'll do my best but it was a total rewrite, it would take several months, and I lacked the experience with this platform. It became complicated for me as I found another job, so I told I would quit by the end of the year. It was terrible, the rewrite was maybe 20% done. The client was furious, yelled at me on the phone, and did not pay the last bill. Failure for me, I felt bad, I tried later on to claim my payment but gave up thanks to boredom (…and threats). It was time to turn the page.

Of course, there is no such thing as failure, but learning occasions, right?

So, here are the lessons learned.

• Beware of time and material contracts. It seems comfortable at first, but when it lasts, when the project doesn't make money and if you don't meet stakeholders' expectations, it will look like a scam. Today I would prefer working on a delimited scope for a fixed price. Would I have taken such engagement in this specific context? Surely not.
• Building a social network where you want to retain people by showing them new things frequently is HARD. It involves lots of relations among the data, caching is complicated and it's easy to fall into important performance issues. Backend and frontend teams need a strong synergy to make a decent app. In this case, we weren't aligned.
• AWS can be very expansive. Elastic Beanstalk (what we used) did not respond to my needs. I could have tailored a Docker container to keep a better control of the whole stack. Otherwise find a more adapted provider.
• Follow my gut feeling. Retrospectively, I knew it was a mess but I kind of get boiled frog, and did not have the courage to talk about the problems early on.
• Performance is a feature. The applications we build are expected to scale.
• I may have been simply not competent enough for the job?

The app has been rewritten, it seems to be working well for the team. On my side, I had a strong lesson for my engineering career.