Journal 2025-05-26

Bootstrapping, abstraction and context size

So Transmissions, my pipeliney thing is intended to simplify the creation of (relatively) complex data processing workflows. This is both for my benefit (as a mediocre coder) and that of AI. In both cases, the overarching issue is that when presented with a large codebase, it can be hard to see the wood for the trees, and vice versa.

At an abstract level, the steps may be relatively simple : things like "1. read a file; 2. send this prompt to an API...". At a code level, each of those is relatively simple. But the complete functioning system involves all levels of granularity. It gets overwhelming for both AI and mere mortals.

I'll expand my views on this elsewhere in the near future, but in short : the answer is task decomposition and lossy composition. Decomposition is pretty straightforward: break complex steps into simpler ones. The lossy composition is how I'll describe the abstraction today. The sequence (/DAG) of the simpler tasks, all lined as necessary, but without the detail.

We have limited working memory. We want to avoid filling it.

...to be continued

Here, have the #:transmission I'm currently working on (to package up a subset of content in a dir tree in a form that AI likes - cat with bells & whistles) :

@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix : <http://purl.org/stuff/transmissions/> .

#############################################################
# insert into pipe for debugging - one instance only like this
:DE a :DeadEnd . # ends the current pipe quietly
:H  a :Halt . # kills everything
:SC a :ShowConfig . # verbose report, continues pipe
:SM a :ShowMessage . # verbose report, continues pipe
:N  a :NOP . # no operation (except for showing stage in pipe)
:UF a :Unfork . # collapses all pipes but one
#############################################################

:terrapack a :Transmission ;
    rdfs:label "Repository terrapack" ;
    :pipe (:p10 :p20 :p30 :p40 :p50 :p60 :p65 :p70 :p80 :p90 :p100 :SM  :p110) .

:p10 a :FileReader ;
     :settings [
          :sourceFile "input/files.spec.json" ;
          :mediaType "application/json" ;
          :targetField "fileSpec"
     ] .

:p20 a :Restructure ;
     :settings :shiftTarget .

:p30 a :DirWalker .

:p40 a :FileFilter .

:p50 a :FileReader ;
     :settings[
          :processWhenDone "true"
     ] .

:p60 a :Restructure ;
     :settings :templateFields .

:p65 a :SetField ;
     :settings [
          :field "templateFilename" ;
          :value "input/file-template.njk" ;
     ] .

:p70 a :Templater .

:p80  a :Accumulate ;
    :settings :accumulator .

:p90 a :Restructure ;
     :settings :accToContent .

  # Accumulate does unfork!   
# :p100 a :Unfork .

:p100 a :SetField ;
     :settings [
          :field "destinationFile" ;
          :value "terrapack-output.txt" ;
     ] .

:p110 a :FileWriter  .