My learnings from Addy Osmanis article on how good is AI React coding

These are my notes from the article How Good Is AI at Coding React (Really)? by Addy Osmani.

There's a lot of information packed into this presentation

Addy says that AI is a force multiplier. "It amplifies everything: good requirements, good architecture, good taste"

AI is most useful for scenarios such as building isolated components, scaffolding, implementing explicit requirements. Its less useful for scenarios such as: multi-step integration, design taste, complex state management. 

We can generalize this to: the higher the complexity the less useful (productive) is the LLM. I called this same point out in a presentation I made in October to our tech leaders. 

And in fact Addys says this explicity later in the article: "If you remember nothing else from this article, remember this: AI handles simple tasks well and then falls off a cliff as complexity rises."

I like that Addy calls out "Objective benchmarks". We've seen by now that LLM model evals can be biased to show the model in a better light. Wow! LLMs are phd level, so why does it struggle with complexity? 
That's because some benchmark tests are limited: oversimplified problems, limited task complexity, solutions seen before. These benchmarks/evals give a false impression of the models.

There are other evals and benchmarks which address these shortcomings and provide a more realistic picture of model performance:

  • Nextjs evals is one such benchmark which shows success rate of approx 42% on tests (https://nextjs.org/evals) 
  • SWE Bench Pro - models score <= 43% success rate on tasks (https://scale.com/leaderboard/swe_bench_pro_public)


React apps are not just code, they're products with user experiences which also include reliability, security, performance, accessibility. React devs should pay attention to quality, user experience quality, code quality, aesthetic and more. 

Addy references these eval sites:

  • Design Arena (https://www.designarena.ai/) where users rank ai generated content
  • Web Dev arena (https://lmarena.ai/leaderboard/webdev) how well models generate websites from prompts and follow ups


Comments

Post a Comment

Popular posts from this blog

angular js protractor e2e cheatsheet

Many from the protractor test suite category detailed task syntax example load page load a page browser.get('yoururl') browser btn actions browser back/forward browser.navigate().back() OR browser.navigate().forward() access elements on page See Protractor API list and examples from Protractors test suite access by angular ngmodel name (must match ngmodel in the html) this is the preferred approach element(by.model('person.name')); access by the css id on the page e.g, #user_name (but this can be brittle) element(by.id('user_name')) access angular binding to get generated value i.e. for {{}} element(by.binding('person.concatName')); access angular repeat list data (must match repeat stmt in the html) See Protractor test suit l...
Read more

angularjs ui-router query string parameter support

angularjs ui-router is pretty cool, we use it in our app now but have not yet used query string parameters (e.g. ?p1=age&p2=height). But now we need to. we're now adding a feature which is basically a report generator and its a good use case for query string parameters once on the page then you're basically in that ui-router "state" users can select filters etc. from dropdowns which will reload the data but should not destroy/recreate any controllers or ui widgets (we stay on the page and do not state transition) yet we do want to update the url for deep links and back button support we don't know the complete list of possible filter choices, but know it should be data driven and will change by report  urls to support this report feature are will be something like thus /reports  /reports/sales  /reports/sales?region=west&time=weeks so we need to use query string parameters Below are some of the things I discovered with ui-router query strin...
Read more

angular js - ngCookie

Warning: if you need to set cookie expiration, httpSecure, path or subdomain then ngCookies doesn't support as of v1.2.7. I plan to use jquery-cookie.js which angular issues also reference, and there is a number of issues open related to this gap in $cookies. Angular js provides support for cookie manipulation in the core library file angular-cookies.js (v1.2.7). That file defines a module ngCookies with 2 factory services: - $cookies - $cookieStore For setting of a simple property on a cookie then $cookies is the factory to use. For setting js objects (key-value pairs) in the cookie then $cookieStore is what to use. To use:  1. include the angular-cookies.js file in yout html 2. include the module ngCookies as a module dependency where you wish to use 3. request $cookies or $cookieStore (or both) to be injected by the dependency injector Using $cookie.  With $cookie you can treat a cookie name as a property on $cookie.   var cookieValue = $cookies.coo...
Read more