What is the best way to get started writing a new backend?

[anthonyhoegberg]

I'm thinking about writing a new custom backend so support my needs, and i need some guidelines about what needs to be done and implemented, etc. Like what files i need to extends from and what i need to export etc to have a working backend.
The actual code for reading/writing data itself i will do myself, i just need some help getting started. A few pointers in the right direction would help me greatly. Thanks!

[james-pre]

@anthonyhoegberg,

Thanks for your interest in the project!

To create your own backend, you can create a new object which satisfies the Backend type, like so:

export const myBackend = {
	name: 'myBackend',
	isAvailable(): boolean {
		return true;
	},
	options: {
		name: {
			type: 'string',
			required: false,
			description: 'The name of the FS',
		},
	},
	create({ name }: TOptions) {
		return new TFilesystem(name); // example
	},
} as const satisfies Backend<TFilesystem, TOptions>;

In this example, TFilesystem just represents the type of whatever filesystem you use, and TOptions just represents the options passed to create.

You don't need to use ... as const satisfies Backend<...>, but I've found this to be the best in terms of typing since you get some nice autocompletion and constant checking.

You can look at the source for some backends for good examples, like InMemory, Fetch, and IndexedDB (IndexedDB being the most complicated of the 3).

A few things to note about the backend object:

• name can be any string
• isAvailable can return a boolean or a Promise<boolean>, so it supports async functions
• options is just an object with runtime metadata for the options passed to create. Importantly, whatever interface you use with the create options should map to the options specified.
  • For example, if the interface is { something?: number }, options should have something with required being false and type being number.
    • For complex types (i.e. objects, type is not as useful. You can write a validator method on the option (perhaps using instanceof or something) which further checks the value.
  • Also type is what the JS typeof operator should give for the option
• create is the function which returns an instance of the backend. It can return a TFilesystem or Promise<TFilesystem>
• Please refer to the docs for Backend.

Please let me know if you would like any more information, I'm happy to help!

[anthonyhoegberg]

So basically what i need is to implement the Backend type object and implement a FileSystem type?
I looked around and i saw there are multiple types of Filesystems i think? If i want a fully featured one i should just implement the FileSystem and everything will work?

[james-pre]

@anthonyhoegberg,

Yes. I recommend you reuse an existing FileSystem implementation (e.g. StoreFS) unless you really need a lot more customization. For example, using StoreFS greatly reduces the amount of work you need to do.

[anthonyhoegberg]

I have another question about zenfs files, when reading/writing files, it is possible to do it in parts?
like lets say a file is 20GB, i dont want to file/write the whole file, but instead use streams or offests and buffers with only relevant changes to the file? I looked at PreloadFile that one works in memory, but can i work with File in a partial way?

Im trying to implement a backend for remote server storage, but want it to "feel like its local"

[james-pre]

@anthonyhoegberg,

I have another question about zenfs files, when reading/writing files, it is possible to do it in parts?

Reading and writing data in parts is supported through the API, so it depends on the backend.

like lets say a file is 20GB, i dont want to file/write the whole file, but instead use streams or offests and buffers with only relevant changes to the file?

This is possible with File.read/File.readSync and File.write/File.writeSync.

I looked at PreloadFile that one works in memory, but can i work with File in a partial way?

File is an abstract class, and doesn't implement the actual reading and writing. It only implements some shared implementations and provides the interface. If you would like your backend to be able to read/write in parts and don't want to use PreloadFile, you may need to write your own File child class with a full implementation. If you're using StoreFS with a custom store, you can extend StoreFS and override createFile and openFile (and their sync variants).

Does this answer your questions?

[anthonyhoegberg]

It kinda did answer my questions somewhat
Im sorry for asking a lot of questions. Im currently trying to write a window manager for the web kinda like a Web based OS and i was thinking of using zen-fs to power the file stuff and use different backends for server files and local files etc

Im still not sure about how to be able to provide a whole file but only the parts the program asks for PreloadFile wont work for me bc i think it loads the whole file so i must load only the data the user requests over network aswell

[james-pre]

@anthonyhoegberg,

Im sorry for asking a lot of questions.

Please ask away, I'm happy to help!

Im currently trying to write a window manager for the web kinda like a Web based OS and i was thinking of using zen-fs to power the file stuff and use different backends for server files and local files etc

That's great, I hope you do well with it.

Im still not sure about how to be able to provide a whole file but only the parts the program asks for PreloadFile wont work for me bc i think it loads the whole file so ...

PreloadFile does load the entire file into memory, though you can read/write only specific parts. If you're looking to only load part of it, that may require a custom File implementation, which lazily loads parts of the file on reads. This would be feasible provided the backends support it.

i must load only the data the user requests over network aswell

If you are looking to load only part of a file over the network, #53 is tracking that.

In case you didn't know, SharedConfig.disableAsyncCache can be used to prevent Fetch from pre-caching of files. For example:

await configure({
	disableAsyncCache: true, // disable caching for all mounts
	mounts: {
		'/': { backend: Fetch, index, baseUrl }, // async only
		'/mnt': { backend: WebAccess, handle }, // async only
		'/home': { backend: IndexedDB, disableAsyncCache: false } // override the top level option
	}
});

However this will not prevent an entire file from loading when you read it.