Rawer

Under Construction

there are many to-dos

#

Hoard

hyperdata-whiteboard

I got distracted yesterday while sorting out the #:packer #:transmission (for concat'ing repos for LLMs), then in the evening, when I got back to desk I was too tired to do anything, gave up. Returning to it now, I need a small repo to run it against. As it happens, this morning in bed, a passing thought led me to start another1 little whiteboard project (with Claude). I'll make that a new repo.

Online, I've created https://github.com/danja/hoard

Locally:

cd ~/github-danny/hyperdata/packages
git submodule add https://github.com/danja/hoard.git hoard
cd hoard
npm init -y

Ok, so now I can try :

cd ~/github-danny/transmissions
./trans packer ../hyperdata/packages/hoard

Ok, it fails as expected...

My head's a little fuzzy and Claudio's pestering. Dogwalk time.

Footnotes

  1. I've been around whiteboards a few times before, first time of note being a SVG Wiki Whiteboard in 2003. ↩

#

Light

Because it is.

Bear of Very Little Brain

Chatted with ClaudioB earlier. He's planning to set up a machine at home to serve online. I started having a go a few weeks back but got distracted. Setting up #:semem in #:tbox and making it live is my main target today, but as that'll involve poking around on my remote server I might as well have another look then.

Packer

Before any of that, I need to finish a little #:transmission I started last session. repomix has broken on me. As best I can determine, it's down to nodejs version, looks like repomix uses some experimental JSON bits that have been pulled. If that is the case, I could use that thing (?) to choose a version that does work. Faff. I've been meaning to make my own on #:transmissions for a while, this tipped me over. Claude gave me some starter code, I just need to plumb it in and tweak it to sanity, as per Claude code.

transmissions$ ./trans packer ./
...
[Error: ENOENT: no such file or directory, open '/home/danny/github-danny/transmissions/manifest.ttl']

Bum! I didn't get around to sorting out the command-line options.

Good-o! I did start sorting out the command-line options.

Yeah, it looks reasonably well separated. The CLI entry point ./trans calls src/api/cli/run.js which uses yargs - tee hee, they say it best :

Yargs be a node.js library fer hearties tryin' ter parse optstrings.

src/api/cli/run.js then calls src/api/common/CommandUtils.js. That does a little bit of path-splitting and simple logic, calling on src/core/ApplicationManager.js to get things going.

I remember now - I left a bit of refactoring around src/core/ApplicationManager.js in-progress. That's still not a high priority, I'll get to it when I next need to do things around there.

Hmm. The options do look very broken. But shouldn't take much fixing.

Well, you learn something every day. I wanted to check what was considered best practice for privates in JS. I asked Claude, confirmed with Perplexity :

Private Fields in JavaScript

Private fields, denoted by the # prefix, were introduced as part of the ECMAScript 2022 (ES13) specification. This feature provides true privacy encapsulation within classes, preventing access to these fields from outside the class

class Person {
  #secret;
  constructor(secret) {
    this.#secret = secret;
  }
  getSecret() {
    return this.#secret;
  }
}

const person = new Person("my secret");
console.log(person.getSecret()); // "my secret"
console.log(person.#secret); // SyntaxError: Private field '#secret' must be declared in an enclosing class

Supported in:

  • Node.js 12.0.0+
  • Chrome 74+
  • Firefox 90+
  • Safari 14.1+
  • Edge 79+

Closures for Encapsulation

Closures have been a traditional way to achieve encapsulation in JavaScript before the introduction of private class fields

function createCounter() {
  let count = 0;
  return {
    increment: function() {
      count++;
      console.log(count);
    }
  };
}

const counter = createCounter();
counter.increment(); // 1
console.log(counter.count); // undefined

Heh, not sure I've ever used that kind of closure in JS. I can easily imagine my confusion over that syntax. I wouldn't call the private field # syntax pretty, but it's a lot more suitable for this bear of very little brain.

#

Semem Fix & Deploy

I made a todo list here yesterday without any explanation. Didn't publish. Invisible todo list is absolutely not what I want on my blog front page! My top priority this week is doc organisation, so I'll let myself off this time. I've changed the date on the file, this is it I'll now fill it out...

A little job for this morning is a little explore of Semem.

It appears to (mostly) work, here's the console log of a run.

But, what is it doing?

What I want right now, as a bit of scaffolding for what I want to explore, is a nodejs version of memoripy that uses a (remote) SPARQL store as (part of) its knowledgebase.

I only tried running it after many hours implementing it in a Claude project. Then had a couple of hours manually tweaking to get it to (mostly) run. It's definitely interacting with Ollama for embeddings & chat, a SPARQL endpoint for persistence. I don't see the extracted concepts in the SPARQL store.

Last night I had a short session with Claude, aiming to document what we had so far. I've put the artifacts over here : https://github.com/danja/semem/tree/main/docs/description_2025-01-01.md It sounds marvellous, but how much is true?

Banebane

I had a chat with ClaudioB yesterday morning, pointed him to my notes so far on #:semem. He asked me if I had a live instance he could play with. A very good question! Not for the first time I'll quote -

Connolly's Bane

"The bane of my existence is doing things that I know the computer could do for me" \– Dan Connolly

Which reminded me I've not been in touch with Graham Klyne for many years. A blog post of his drew my attention to that quote. Good-o, he's on Mastodon : @gklyne@indieweb.social, I'll ping him there.

ClaudioB's question is the kind of message I want my #:hyperdata systems to be able to deal with automatically.

Here's how this human agent is handling it :

[prompt]

Is there a live instance of Semem?

[me]

  • immediate response, "not yet"
  • reformulate question in terms of concrete requirements ->
  1. (functioning, good-enough) Semem
  2. hosting
  3. deployment infrastructure

For 1 : all I really need to do right away is minor bugfixing and whatever's needed to use existing online LLMs For 2 : I've got a live Ubuntu server (which you are looking at now) and a domain name registered for a related plan, some nginx proxying already in place.
For 3 : I've got a Docker container in-progress for this stuff, tbox

  • meta-reasoning - evaluate this against existing dev paths, consider priorities, time needed etc

On this point there's a green flag to go ahead, it's near-enough in line with what I was planning to do already, slight rearrangement of immediate priorities.

I acted in several roles there. Humans are good at this. I want a community of independent autonomous intelligent agents (on #:kia).

Ops. Azzurra just reminded me that I've an appointment with her up at the comune at 15:00. Now 13:12. Clever Azzurra has me sussed. I'd completely forgotten. But she's given me just the right amount of time to pop this online, take Claudio out for a short walk, find some clean clothes & have a wash.

Soon...

Somewhere around my vocabs I need terms for this particular kind of input. I may already have something like Feature Request around the project management oriented bits, but it totally ties in with agent-oriented messaging.

  • #:todo fix #:semem failing on UPDATE
  • #:todo tease next steps from semem/docs/description_2024-12-30/

I sorted out the UPDATE.

  • #:todo deploy to #:tbox

  • #:todo #:spike for function/action calling a la ReAct

  • #:todo some FOAF

Installed 'Markdown Preview Mermaid Support' into VSCode

semem - query execution times, store metrics, total embeddings size

support Groq, Mistral, Claude, OpenAI

refactor

comments

install the willison-mentioned UI tool - has useful bits for APIs?

ditto LibreChat

LOGGER - see Perplexity

# 
node src/SPARQLExample.js

[INFO] Initialized FAISS index with dimension 1536
[INFO] Loaded 1 short-term and 0 long-term memories from store http://localhost:4030/test/query graph <http://example.org/mcp/memory>
[INFO] Adding interaction: 'What's the current state of the Semantic Web?'
[INFO] Memory initialized with 1 short-term and 0 long-term memories
[INFO] Extracting concepts...
[INFO] Extracted concepts: [ 'Semantic Web technologies', 'AI' ]
[INFO] Retrieving relevant interactions...
Response: Semantic Web technologies and Artificial Intelligence (AI) complement each other well, as they are both powerful tools for representing complex data in structured form, facilitating knowledge discovery and enhancing human-computer interaction. Here are some ways in which semantic Web technologies can be integrated with AI:

1. **Knowledge Extraction from Semantically-Structured Data**: Semantic Web technologies provide a standardized way of defining and storing the structured information necessary to build semantic web applications. The use of these tools allows for the extraction of knowledge about large amounts of data, making it easier for AI systems to process this information.

2. **Artificial Intelligence as a Knowledge Broker**: AI can be used to curate and organize vast amounts of data in semantic Web formats. This includes tasks such as categorizing documents, recognizing patterns and relationships within structured data sets, and integrating heterogeneous data sources into a single knowledge base.

3. **Interactive Experience Enhancement through AI-Driven Semantic Web Applications**: AI-driven applications can provide a more personalized user experience by using techniques like recommendation systems to suggest content based on the user's interests or previous behavior. These systems also use semantic Web technologies to ensure that the information presented is accurate and relevant to the user, enhancing their overall satisfaction with the application.

4. **Machine Learning for Enhanced AI Capabilities**: AI can be trained with data from large volumes of structured web documents, which can then be used to improve AI models’ understanding of context, language, and relationships within web content. This integration allows machines to learn complex patterns in unstructured or semi-structured data, further enhancing their ability to perform tasks that require human-like reasoning.

5. **Integration with Natural Language Processing (NLP) Systems**: Semantic Web technologies can be used with AI NLP systems to extract meaning from unstructured text and improve the understanding of natural language queries by machines. This integration allows for a more effective interaction between humans and computers, as the data is presented in a format that is easier for AI systems to interpret.

6. **Knowledge Management for Autonomous Agents**: Semantic Web technologies can be utilized with artificial intelligence-driven agents to store large volumes of structured information. These agents are capable of processing complex queries, making decisions, and performing tasks based on this knowledge. By integrating semantic web technologies into AI systems, we can create more intelligent autonomous agents that are better equipped to handle complex scenarios.

These integration strategies illustrate how the capabilities of both Semantic Web technologies and Artificial Intelligence can be combined to enhance our ability to process large volumes of information and solve complex problems efficiently.
[INFO] Extracting concepts...
[INFO] Extracted concepts: [
  'Semantic Web technologies',
  'Artificial Intelligence (AI)',
  'Knowledge Extraction from Semantically-Structured Data',
  'Artificial Intelligence as a Knowledge Broker',
  'Interactive Experience Enhancement through AI-Driven Semantic Web Applications',
  'Machine Learning for Enhanced AI Capabilities',
  'Integration with Natural Language Processing (NLP) Systems',
  'Knowledge Management for Autonomous Agents'
]
[INFO] Adding interaction: 'How can Semantic Web technologies be used with AI?'
[INFO] Saved memory to SPARQL store http://localhost:4030/test/update graph <http://example.org/mcp/memory>. Stats: 2 short-term, 0 long-term memories

This query gives one result, containing the prompt & response from the above :

PREFIX mcp: <http://purl.org/stuff/mcp/>
SELECT ?graph ?id ?prompt ?output ?timestamp
WHERE {
  GRAPH ?graph {
  ?s a mcp:Interaction ;
     mcp:id ?id ;
     mcp:prompt ?prompt ;
     mcp:output ?output ;
     mcp:timestamp ?timestamp .
    FILTER regex(str(?prompt), "Semantic")
}
}

This gives 2 results :

SELECT *
WHERE {
  GRAPH ?graph {
  ?s ?p ?o
    FILTER regex(str(?o), "Semantic")
}
}

One is the triple with the prompt, the other the triple with the LLM output.

#

Repo Boo

Very annoying. I've been using Claude projects heavily for recent dev. Made possible by repomix which essentially does an annotated concat of a repo's files. Tonight my thing decided to say:

danny@danny-desktop:~/github-danny/transmissions$ npm run rp

> transmissions@1.0.0 rp
> repomix -c repomix.config-small.json . && repomix -c repomix.config-large.json .

(node:61405) ExperimentalWarning: Importing JSON modules is an experimental feature and might change at any time
(Use `node --trace-warnings ...` to show where the warning was created)

📦 Repomix v0.2.12

Error: Config file not found at repomix.config-small.json
For more help, please visit: https://github.com/yamadashy/repomix/issues
(node:61421) ExperimentalWarning: Importing JSON modules is an experimental feature and might change at any time
(Use `node --trace-warnings ...` to show where the warning was created)

I've no idea what I've done that would impact there. Can't remember if this is running a global or local install. Ok, first thing, clear caches and try again. "Hey Claude, how do I..?"

rm -rf node_modules
rm package-lock.json
npm cache clean --force
npm install

While that's running, there is something else I want to do nearby. Squat the npm registry with a couple of things. #:transmissions & #:semem at least, there might be a couple of others. None will be out-of-the-box usable as they should be (as I would interpret proper etiquette), but I think I'm at a point where I can legitimately grab those names. I've put loads of work in, am committed to making them useful for other parties.

It might even encourage me to get the GitHub CI/CD bits in place (for the badges, innit).

Golly :

npm warn using --force Recommended protections disabled.
npm warn deprecated inflight@1.0.6: This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful.
npm warn deprecated glob@7.2.3: Glob versions prior to v9 are no longer supported
npm warn deprecated sourcemap-codec@1.4.8: Please use @jridgewell/sourcemap-codec instead
npm warn deprecated rimraf@3.0.2: Rimraf versions prior to v4 are no longer supported
npm warn deprecated glob@7.2.3: Glob versions prior to v9 are no longer supported
npm warn deprecated glob@7.2.3: Glob versions prior to v9 are no longer supported
npm warn deprecated glob@7.2.3: Glob versions prior to v9 are no longer supported

added 1141 packages in 9m

237 packages are looking for funding
  run `npm fund` for details

Aw, npm install squirt works (whatever that is), I'd have squatted that too. Ok, no biggy.

npm install squirty
npm error code E404
npm error 404 Not Found - GET https://registry.yarnpkg.com/squirty - Not found

For that I only need to do some minor renaming. #:transmissions would have been a proper pain. I did it once already - had the project working label "treadmill" initially, but decided something more accurately descriptive would be better. Horrible search & replace - not too bad with code, but docs got messy.