My biggest failure so far

It's time to share my biggest failure as a developer, so far.

Some years ago, I signed for the development of an mobile app backend. It was a social network, and other freelancers / agencies were in charge of mobile frontends (iOS and Android) and back office user interface.

When I started, the client told me that the job was to finish an existing app in Python and Django and document it. The mission would take a couple of weeks.

However, the backlog was huge and I end up working for 8 months on this project. We faced a lot of issues and misunderstanding with frontend developers. Frontend developers have changed two or three times during the development and both existing iOS and Android app were remade from scratch.

After 6 months, the whole system was almost stable with few remaining bugs so I began load testing on the backend. It was a catastrophe, unacceptable: the app would not even support thousand simultaneous users.

I worked to get rid of performance issues on the backend, using database replication, rewriting queries. The process was slow and costly in term of hosting. It improved the situation a bit. I guess the app would have need more radical changes though, and working in collaboration with frontend developers.

The client started being nervous and decided to follow the advice of one of the frontend freelance developer: rewrite the all app using Google Firebase! I have been considered guilty of the mess, hence the client demanded me to take care of the migration and to create a bridge so that the existing back office could still be used. I warned I'll do my best but it was a total rewrite, it would take several months, and I lacked the experience with this platform. It became complicated to me as I found another job, so I told I would quit by the end of the year. It was terrible, the rewrite was maybe 20% done. The client was absolutely furious, yelled at me on the phone and did not pay the last bill. Failure for me, I felt bad, I tried later on to claim for my payment but gave up thanks to boredom (…and threats). It was time to turn the page.

Of course, there is no such thing as failure, but learning occasions right?

So, here are the lessons learnt.

• Beware of time and material contracts. It seems comfortable at first, but when it last, when the project doesn't make money and if you don't meet stakeholders expectations, it will really look like a scam. Today I would prefer working on a delimited scope for a fixed price. Would I have taken such engagement in this specific context? Surely not.
• Building a social network where you want to retain people by showing them new things frequently is HARD. It involves lots of relations among the data, caching is complicated and it's easy to fall into important performance issues. Backend and frontend teams need a strong synergy to make a decent app. In this case, we weren't aligned.
• AWS can be very expansive. Elastic Beanstalk (what we used) did not respond to my needs. I could have tailored a Docker container to have a better control of the whole stack, or find a more adapted provider.
• Follow my gut feeling. Retrospectively, I knew it was a mess but I kind of get boiled frog, and did not had the courage to talk about the problems early on.
• Performance is a feature. The app we build are expected to scale.
• Perhaps I was simply not competent enough for the job?

The app has been rewritten, it seems to be working well for the team. On my side, I had a strong lesson for my engineering career. The story is not so sad after all.