May 23, 2026

Flagging Ingested Duplicates @gbrain

Firstly, gbrain is a repository which I stumbled on X while scrolling.It happens to be like a second - brain for an AI agent.In general agents when accessed start with zero context.gbrain solves this problem by giving it information for eg from meetings, calls, research, notes etc.gbrain also exposes MCP tools to the client LLMs to read and write back.

Gbrain has a CLI endpoint where ingestion happens and it is a manual step.We could trigger it by this command:

gbrain ingest / path / to / vault /

Other than ingestion there are three major actions performed which are retrieval, writing - back and enrichment.

The system design of Gbrain.

The database design

This issue is regarding solving the problem of duplicates which arise due to ingestion of files due to the ambiguity of a directory and its subdirectory as slug value for both of them were same.Now the problem is gbrain solely uses slug value to distinguish between ingested files.Lets say the example of vault and vault / subdir, they will have the same file but will have different slug hence flagging the same file as duplicates.The major problem is the skewing of data inside eg the agent might summarize same content twice.

The solution

The logic of the code is in three steps.

// Check if a page with the same content hash already exists in this source under a different slug.const duplicateRows = await engine.executeRaw<{ slug: string }>(
    `SELECT slug FROM pages 
     WHERE source_id = $1 AND content_hash = $2 AND deleted_at IS NULL LIMIT 1`,
    [sourceId ?? 'default', hash]
);
const existingDuplicate = duplicateRows?.[0];

if (existingDuplicate && existingDuplicate.slug !== slug) {
    console.warn(
        `[warning] "${slug}" is a duplicate of existing page "${existingDuplicate.slug}" (same content hash). Skipping.`
    );
    return { slug, status: 'skipped', chunks: 0, parsedPage };
}