Move schema tables from _internal to _catalog

[simonw]

This came up in discussion over:

• Use $DATASETTE_INTERNAL in absence of --internal #2174

[simonw]

Here's the thinking: setting theDATASETTE_INTERNAL environment variable for your entire machine will cause any time you run datasette x.db to result in a process that shares the same internal.db database as other instances.

You might run more than one instance at once (I often have 4 or 5 going). This would currently break, because they would over-write each other's catalog tables:

datasette/datasette/utils/internal_db.py

Lines 5 to 62 in e4abae3

	async def init_internal_db(db):
	create_tables_sql = textwrap.dedent(
	"""
	CREATE TABLE IF NOT EXISTS catalog_databases (
	database_name TEXT PRIMARY KEY,
	path TEXT,
	is_memory INTEGER,
	schema_version INTEGER
	);
	CREATE TABLE IF NOT EXISTS catalog_tables (
	database_name TEXT,
	table_name TEXT,
	rootpage INTEGER,
	sql TEXT,
	PRIMARY KEY (database_name, table_name),
	FOREIGN KEY (database_name) REFERENCES databases(database_name)
	);
	CREATE TABLE IF NOT EXISTS catalog_columns (
	database_name TEXT,
	table_name TEXT,
	cid INTEGER,
	name TEXT,
	type TEXT,
	"notnull" INTEGER,
	default_value TEXT, -- renamed from dflt_value
	is_pk INTEGER, -- renamed from pk
	hidden INTEGER,
	PRIMARY KEY (database_name, table_name, name),
	FOREIGN KEY (database_name) REFERENCES databases(database_name),
	FOREIGN KEY (database_name, table_name) REFERENCES tables(database_name, table_name)
	);
	CREATE TABLE IF NOT EXISTS catalog_indexes (
	database_name TEXT,
	table_name TEXT,
	seq INTEGER,
	name TEXT,
	"unique" INTEGER,
	origin TEXT,
	partial INTEGER,
	PRIMARY KEY (database_name, table_name, name),
	FOREIGN KEY (database_name) REFERENCES databases(database_name),
	FOREIGN KEY (database_name, table_name) REFERENCES tables(database_name, table_name)
	);
	CREATE TABLE IF NOT EXISTS catalog_foreign_keys (
	database_name TEXT,
	table_name TEXT,
	id INTEGER,
	seq INTEGER,
	"table" TEXT,
	"from" TEXT,
	"to" TEXT,
	on_update TEXT,
	on_delete TEXT,
	match TEXT,
	PRIMARY KEY (database_name, table_name, id, seq),
	FOREIGN KEY (database_name) REFERENCES databases(database_name),
	FOREIGN KEY (database_name, table_name) REFERENCES tables(database_name, table_name)
	);

The breaking wouldn't be obvious because the catalog tables aren't used by any features yet, but it's still bad.

This convinced us that actually we should move those catalog_ tables OUT of internal.db. The _internal database will be reserved for plugins that want to use it for caching, storing progress, etc.

I think we move them to an in-memory _catalog database which is excluded from ds.databases (like _internal is ) but can be accessed using datasette.get_catalog_database() - similar to datasette.get_internal_database().

So each instance of Datasette gets its own truly private _catalog, which is in-memory and so gets cleared at the end of each process.

An interesting thing that came up about a shared _internal database is that it provides opportunities for things like a notes plugin which allows you to attach notes to any row in any table in a database... where those notes become available to multiple Datasette instances that you might launch on the same laptop.