5.3 Assisted Development (Program Synthesis and the Future of Software Development)

Assisted Development ^0

Program synthesis and the future of software development

by Hobson Lane (mesa_python.gitlab.io)

shoutout to Maria Dyshel, Vishvesh Bhat

Agenda

  1. Assisted development
  2. Neurosymbolic reasoning
  3. Program synthesis example
  4. Shy fun

Assisted development

  1. Copilot+VSCode|JetBrains|Neovim
  2. Cursor.sh+VSCode|NeoVim|Jetbrains - $20/mo - just bought Supermaven CLI
  3. Windsurf+Codium (VSCode FOSS fork) - “agentic IDE”
  4. Zed - FOSS Rust by maker of Github’s Atom - focus on fast and eventually AI - 3 yr old almost feature parity with Sublime and VSCode
  5. Aider+terminal
  6. Cline+VSCode|Jetbrains
  7. shy-sh - FOSS Python CLI developer assisstant
  8. Gemini, Phind, Perplexity (Type 2 Assisted Development)

Program synthesis

  1. Deduction (or Compiling) - High level languages (Logic, Haskel, Python)
  2. DSL - including declarative languages (f-strings, Jinja2, Ansible/puppet/Terraform yaml)
  3. Induction (inference) - ML set of input output examples, verifier, or reverse engineering an existing system, like George hotz tiny corp jailbreaking an iphone

Examples of deduction

  1. Python 2to3
  2. regular expressions: r'(for|while|if|else|with|try|except|def)(.*):
  3. Juan’s HDMI screen capture OCR Python->Rust

Next token prediction

Assisted Development ^1 ^2

  1. Autocomplete (Tab-Nine, Sublime, Augment Code)

  2. Search (Phind.com, Perplexity.ai)

  3. Conversation (ChatGPT, shy-sh)

  4. Neurosymbolic reasoning - Shoulder-surfing AI (Anthropic “computer use”)

Execute a command in Bash?

cmd="ls -l"
eval " $cmd"

David Crenshaw

”Only 5% of developers use conversational AI to code” 1(#links) 2(#links)

(Type 3 Assisted Development)

Assisted Misalignment

Your favorite LLM was poisoned (fine-tuned) on misaligned and insecure code.3

LLM:

eval $(echo $command_from_some_guy_on_the_internet)

StackOverflow:

TBD proai.org shortURLs and mesa_python.gitlab.io blog post

neurosymbolic reasoning

examples from ARC challenge dsl graph search diagram (exponential complexity) if search on tokens with beam search it’s just depth-first stochastic search of the wrong graph neurosymbolic reasoning searches the graph of valid programs - like AlphaFold, AlphaGo, or AlphaCode

  • neurosymbolic reasoning

  • [-] optional slide comparing computer use, puppet and Ansible agents vs “operator” 200/mo https://git.lolcat.ca/lolcat/4get

  • slide listing 3 uses of LLMs and shout out to tailscale CEO blog post

  • 1. slide on code completion in sublime

  • 2. slide on phind and perplexity and compare to metager and https://git.lolcat.ca/lolcat/4get

  • 3. slide on cognitive load of translating concept to conversation

  • snake game reinforcement learning demo example with Claude

  • Gemini vs claude vs augment code

  • slide on zed

  • slide on Regex to match python keywords and a compiler optimization vs ml ngram approach

Backup (notes)

  1. Deduction or Compilers or transpilers Fortran Haskell python decorators & factories high level languages, Wikipedia max example
  2. Declarative & domain specific languages and fstrings Jinja2 Django templates and terraform Ansible puppet yaml declarative languages generating functional programs
  3. Induction or inference - ML set of input output examples, verifier, or reverse engineering an existing system, like George hotz tiny corp jailbreaking an iphone

4. Neurosymbolic reasoning agents - Shoulder-surfing AI ("computer use")

code synthesis slides

  1. Math function synthesis from examples or running of a verifier (each example is expensive to verify, using an optimizer or simulator).
  2. Simulation verifier would work for game play or puzzle/maze solving program generation. Generating a program that can solve any maze, input=maze, output=path
  3. Program synthesis, [automatic] code generation, compiler, templating engine, high level [templating] programming language, UML, low code platforms
  4. Deriving a regex from example matches
  5. Deriving a dialog engine or chatbot from example conversations
  6. Math function example max() from logical specification, build on it to create sort pair from max, then max of list, then merge 1 list with 1 val (binary search), then merger sort verifier for both. Functional programming
  7. Machine learning, generalization, regularization, ocams razor,
  8. A star, depth first search, breadth first search, beam search, but need continuous heuristic or metric to say how close you are to optimum program. LLMs really good at generating almost correct programs, so great for depth first search.
  9. If you can break problem into smaller function you can do intermediate depth search for exponential speedup
  10. Current research is focused on improving the accuracy rate on complete programs which will never succeed because Portugal space is far larger than dataset, not spanned by possible programs
  11. Neural network architectures and weights (dl training) are the ultimate in program synthesis, which could be very modular by focusing on layers that are similar
  12. Genetic algorithms for program synthesis
  13. Toy problem regex to recognize a valid pep8 compliant Python…
    • keyword
    • int
    • float
    • variable/function name
    • raw string literal
    • string literal
    • dict, tuple, list, argument, type hints, function signature
    • for loop, list comprehension, generator
  14. A regex is a grammar and a grammar can be generated/enumerated.
  15. A big refinement would be to constrain the llm token generator, with the Python grammar and vocabulary so that it can never generate a syntax error or non-pep8 code.
  16. Other heuristics could be encoded in the grammar like cyclomatic complexity variable name vocabulary and length limitations and abbreviation control and Hungarian notation, or NL grammar rules (active verbs for functions, is for bool retvals, to for transformation, no pattern not already available in standard Python library of builtins, no list comprehensive, just accumulators, all functions with type hints and args on one line.

Footnotes

  1. proai.org/3-types - How I Program with LLMs by David Crenshaw (CEO Tailscale) ↩

  2. changelog.com/podcast/629 - Jerod interviews David Crenshaw (Tailscale) ↩

  3. proai.org/assisted-misalignment - Mastodon Toot by @Techmeme ↩