AI maturity levels - understanding the software engineering disruption

The last 2 to 3 years have been a whirlwind of disruption in Software engineering due to the impact of AI. I was seeking about a way to represent an "ai maturity" scale to make sense of the changes and I think this video based on this article from Dan Shapiro is a good representation.

From these I created a graphic which represents visually key points as I understand it.

Remember about 2 or 3 years ago when you first used copilot in vscode? the predictive typeahead...pretty cool eh.

Then we all moved on to cursor and its much better typeahead. Btw what a great acquisition by cursor to buy Supermaven code completion in 2024. I believe it more than repaid their investment given cursors since skyrocketed valuation.

And then came ai chat for code gen. At the time codegen quality was low e.g. "write tests for function xyz". It used to get maybe 30% and then you finished the rest. Then we added rules to make the codegen better (we added 1000s of lines of rules for different tasks such as writing tests vs writing pure code) and models improved. Codegen quality improved to 40%, 60%, 80% code generated accurately.

Btw somewhere in this timeline Microsoft squandered their first mover "AI lead" and everyone moved to better tools such as cursor. 

Claude Sonnet was generally acknowledged "the best coding model". You're seeing Opus now, but remember dating Sonnet? or Haiku.

And codegen ai could handle more and more complex features.

Then Claude code was released. A terminal interface to ai! What!

Over this timeline devs are writing less and less and less code.

We have come a long way from "Level 0 Spicy Autocomplete". And this transition/disruption which software engineers are living through is not finished. Not by a long shot.

Most software engineers are operating somewhere between level 2 and level 3. 

Some engineers are adopting level 4 practices such as writing significantly more detailed and complete specifications. I am writing far more detailed specifications in user stories and tasks with ai which are then delegated to ai to implement.

An important point to recognize is that we have left the phase/era of Software engineers as the primary code generator (using modern languages over last ~30 years). The era of Software engineers writing all the code is over.

In this new phase we are in, Software engineers will spend far more time writing specifications, managing ais and managing quality and outcomes. The phase/era of Specification driven development is here. 

Responsibilities include: 

• creating ai skills to be reused across the org/community, and making use of useful 3rd party skills
• defining and managing agents and subagents which use skills and integrate with mcps (figma, atlassian, github etc.) 
• app architecture and tech stack decisions and specification
• user stories with detailed acceptance criteria
• subtasks with detailed specs to be used by ais to write the code
• defining rules for how you want code written (e.g. cursor rules and/or skills)
• long markdown files written by ais to be used by ais to write the code
• adding and configuring ai pr code reviewers which review code quality, find logic bugs and check for completeness against specs
• ensuring test coverage with e2e, component and unit tests
• ...and more

Additionally Software engineers will also have to develop other skills including user interface design skills, product management skills and project management skills which in many orgs have not been done by Software engineers. 

The disruption is not stopping. Just as the Level 0 and Level 2 Phases are in the past, this current Phase will also be exited. On to completely in Level 3 and on to Level 4.

When will organizations decide they can move to Level 4 and don't care about the way the code is written and don't need to review code at all? That depends on a number of factors. In some cases its already happening such as startups ideating and building a viable product; or devs creating scripts to manage in house workflows. And more and more, code as a black box is the future. 

We are technologists. We apply technology for purposes. Most of us work for for-profit businesses and apply technology to make the businesses more successful and yes, more profitable. Our work is in service of business goals. A new feature for customers. To reduce the costs of ownership etc.

You may have become a software engineer because you love coding. You may have graduated recently from college with a comp sci degree which has you unprepared for the way Software engineers work now. My advice is to succeed as a Software Engineer you have to embrace the technology, learn it and figure out how to use it to solve problems for the business, which btw also includes solving problems you deal with in your work so you and your colleagues are more efficient and your end product is better and sooner. 

There is real opportunity to make a impact as technologists. I've been on the "AI Champions" team at Opentable for about 2 years now. We're run pilots and evaluations and more. I'm getting deeper into skills these days. Now I see opportunities to write new skills in multiple places and am working on creating a new skill. We're all learning, there's also a need for education and training, even the basics. You can do that. Learn the technology and then teach others. Helping others is fulfilling, is appreciated and is a multiplier. And btw it's a great way to grow as an engineer.

So lets get after it.

Update: due to recent outages supposedly caused by quality of AI gen code released to production, Amazon reportedly recently called engineers to review and approve all AI assisted code changes. This doesn't mean Amazon and other orgs won't go more "black box" coding over time (Level 4 and 5). To me it indicates that appropriate and necessary quality controls were not in place, given the risks, to allow AI codegen to deploy to production human free. Over time the necessary and appropriate quality controls will be figured out and put in place.