deathau.weblog.lol/weblog/B4. Friendica & Beyond/B4.0014 Going on Safari (Chapter One).md

title	date	location	tags
B4.0014 Going on Safari (Chapter One)	2024-09-03	/B4.0014	Blog, Markdownload, Development, Swift, JavaScript

Going on Safari (Chapter One)

As mentioned in my previous blog post, I went down a bit of a rabbit hole regarding native messaging. I couldn't figure out how to communicate properly via stdin/stdout from a MacOS Swift app. If anyone has any pointers, that would greatly simplify the process; because oh boy is this a process.

In this post I'm discussing the prototype mentioned in the previous post. One that does allow for data to be passed back and forth, but it's convoluted. Part of that is my fault. I thought it would make sense to re-use the sidebar code in the native app, so I set up my shared ViewController with a WKWebView that loads the sidebar code.

I'm getting ahead of myself. I first built myself a prototype of the new MarkDownload extension utilising Manifest V3. Most of the work was handled in the sidebar code itself, calling a client script to get the html of the current page, then doing all the transformation logic in the sidebar JavaScript code. It uses ES6 modules and modern standards and works great, as a prototype. But of course, when I converted it to a Safari extension, following the instructions from Apple it obviously couldn't open a sidebar and therefore would do nothing.

The instructions linked above lead to the creation of an XCode project made up of multiple parts. There's the extension Info.plist and entitlements, for both macOS and iOS (yes, iOS. Hopefully, as a result of this, we can have an iOS version of MarkDownload 🥳). Then there's the AppDelegate and storyboards for the macOS and iOS apps. There's the shared code for the extension which includes a Resources folder referencing my existing extension code, and a SafariWebExtensionHandler (which we will come back to later). Finally there's the shared app code, which includes a ViewController with a WKWebView and a html page with associated JavaScript to communicate with the ViewController to get and display the current state of the Safari extension. It also has the button to close the app and open the Safari preferences.

And so this is where we begin. I already have a template built in for communicating from the web view to the view controller and back. Also a handler for messages from the extension, which returns replies. This is going to be a piece of cake, right? Well...

Let's go through the main parts of what Apple already provided.

Sending a message from web view → app

This is pretty simple and already set up in the template. From the javascript side of the web view, there is this line of code:

webkit.messageHandlers.controller.postMessage("message-string");

To receive this in the ViewController, inside the viewDidLoad() function, this line sets up the web view to be able to receive messages and load the html file:

self.webView.configuration.userContentController.add(self, name: "controller")
self.webView.loadFileURL(Bundle.main.url(forResource: "Main", withExtension: "html")!, allowingReadAccessTo: Bundle.main.resourceURL!)

And for this to work we also have to implement the WKScriptMessageHandler protocol, like so:

class ViewController: PlatformViewController, ..., WKScriptMessageHandler {
...
    func userContentController(_ userContentController: WKUserContentController, didReceive scriptMessage: WKScriptMessage) {
        let body = scriptMessage.body as! String
        // process and act on message
    }
}

So far, so simple. It's basically trivial to substitute "Main" with "sidepanel", add everything in the shared extension "Resources" directory to the macOS build target and have my sidebar magically appear inside an app. This is going great!

My sidebar currently sends messages to the background page, etc. by using browser.runtime.sendMessage, as well as using browser.runtime.onMessage.addListener to receive messages back, where applicable. To cope with the change necessary in the app, I made myself a messenger.js which contains a class like this:

globalThis.browser ??= chrome

export default class Messenger {
  static addListener = browser.runtime.onMessage.addListener
  static removeListener = browser.runtime.onMessage.removeListener
  static sendMessage = async (data) => await browser.runtime.sendMessage(data)
}

However, I specifically do not include this file in the macOS target, and instead create a new version that does get included which contains code similar to the following:

class Messenger {
  static addListener = //??
  static removeListener = //??
  static sendMessage = (data) => webkit.messageHandlers.controller.postMessage(JSON.stringify(data))
}

Sending message from app → web view

There are some question marks in the above. How do I send messages from app to the web view? In the template provided by Apple, they... don't. They just do webView.evaluateJavaScript("//javascript code").

So, I used that to my advantage and came up with the following solution. Modifying the app-based messenger.js above:

class Messenger {
  static listeners = []
  static addListener = (x) => Messenger.listeners.push(x)
  static removeListener = (x) => Messenger.listeners = Messenger.listeners.filter(y => y !== x)
  static sendMessage = async (data) => await webkit.messageHandlers.controller.postMessage(JSON.stringify(data))
  static recieveMessage = (message) => Messenger.listeners.forEach(x => x(message))
}

I just keep a static list of listeners, and add a new recieveMessage function that calls all the functions in the list of listeners. This allows me to do the following in my ViewController:

self.webView.evaluateJavaScript("Messenger.recieveMessage(\(String(decoding: try JSONEncoder().encode(message), as: UTF8.self)))")

Neat!

Although, so far, none of that communication is going to my extension.

Sending message from app → extension

There's an example of this built into the template, too. Easy peasy. The ViewController contains this function:

    func sendMessageToExtension(name: String!, userInfo: [String:String]) {
#if os(macOS)
        SFSafariApplication.dispatchMessage(withName: name, toExtensionWithIdentifier: extensionBundleIdentifier, userInfo: userInfo) { error in
            debugPrint("Message attempted. Error info: \(String.init(describing: error))")
        }
#endif
    }

(I'm ignoring the #if os(macOS) for now. I'm going to cross the iOS bridge when I come to it)

To receive this message in the javascript I need to implement this in my background.js

let port = browser.runtime.connectNative("au.death.MarkDownload") // the identifier doesn't matter. It will only connect to the related app
port.onMessage.addListener((message, sender) => {
	// process the message
	// if I want to send a message back, I can use
	// sender.postMessage(response)
	// ... right?
});

Sending message from extension → app?

This is where things start to get a little shaky. As you can see in the comments above, I'm questioning sender.postMessage. In theory that sends a message back through the port to the macOS app. In practice, I have no idea how to listen for this message on the app side. If anyone has any knowledge of this, I would really appreciate it. It might render the rest of this post moot.

Surely Apple provides an example for this? Well, they do, but there's a catch. Let's get into the example code. Rather then sending a message through the open port, they do this:

browser.runtime.sendNativeMessage("application.id", {message: "Hello from background page"}, function(response) {
  console.log("Received sendNativeMessage response:");
  console.log(response); 
});

Again, the application ID doesn't matter - Safari will only communicate with the linked app. To receive the message, there is this code...

func beginRequest(with context: NSExtensionContext) {
	let request = context.inputItems.first as? NSExtensionItem

	let profile: UUID?
	if #available(iOS 17.0, macOS 14.0, *) {
		profile = request?.userInfo?[SFExtensionProfileKey] as? UUID
	} else {
		profile = request?.userInfo?["profile"] as? UUID
	}

	let message: Any?
	if #available(iOS 15.0, macOS 11.0, *) {
		message = request?.userInfo?[SFExtensionMessageKey]
	} else {
		message = request?.userInfo?["message"]
	}

	os_log(.default, "Received message from browser.runtime.sendNativeMessage: %@ (profile: %@)", String(describing: message), profile?.uuidString ?? "none")

	let response = NSExtensionItem()
	if #available(iOS 15.0, macOS 11.0, *) {
		response.userInfo = [ SFExtensionMessageKey: [ "echo": message ] ]
	} else {
		response.userInfo = [ "message": [ "echo": message ] ]
	}

	context.completeRequest(returningItems: [ response ], completionHandler: nil)
}

... in the SafariWebExtensionHandler. And the thing about that class is that it has a target membership of (that is to say, it is included in) the Extension, but not the App. So wait, there's another layer here? The javascript is sending a message to a piece of native swift code that is not part of my app. So, the title for this section should have been Sending message from extension → extension? Or perhaps ... javascript extension → native extension

Okay then, how do I actually send data to my app then?

Sending message from (native) extension → app

There is no example for this. Check out the documentation. It covers app → (javascript) extension and (javascript) extension → (native) extension, but nowhere does that data actually make it back to the app. In fact, the documentation does actually contain a paragraph hinting at this:

Because the macOS or iOS app and the native app extension each run in their own sandboxed environments, they cannot share data in their respective containers. You can store data in a shared space that both the macOS or iOS app and the native app extension can access and update, by enabling app groups. For more information, see Sharing Data with Your Containing App.

I didn't see this at first, and did a whole bunch of googling, the result of which was often replies to the effect of "use UserDefaults". The link in that quote does point to a page describing how the app and the app extension run in different sandboxed contexts, and therefore both need to be added to a shared "app group". This way there is a shared container which both the app and extension can read and write to.

Figure 4-1 An app extension’s container is distinct from its containing app’s container

Okay, so... First thing, I have to add in the App Groups capability to each of the targets in my XCode project  Then add in an identifier for the group that will be identical across targets:  Side note: I found that for the iOS targets, I had to use `group.` as the first part of my identifier.

Once that is in place, I can modify the native extension code to store the data in the shared User Defaults

func beginRequest(with context: NSExtensionContext) {
	let request = context.inputItems.first as? NSExtensionItem
	...
	let message: Any?
	...
	var dict = message as? [String:Any] ?? [:] // assuming I'm sending an object from javascript and not just a plain string
	let defaults = UserDefaults(suiteName: "my.group.identifier")
	do {
            let jsonData = try JSONSerialization.data(withJSONObject: dict, options: .prettyPrinted)
            defaults?.set(jsonData, forKey: "message")
        }
        catch {
			// do something with the error?
        }
	...
	context.completeRequest(returningItems: [ response ], completionHandler: nil)
}

I can also set up an observer in my ViewController like so:

class ViewController: PlatformViewController, WKNavigationDelegate, WKScriptMessageHandler {
	override func viewDidLoad() {
        super.viewDidLoad()   
        UserDefaults(suiteName: "my.group.identifier")?.addObserver(self, forKeyPath: "message", options: .new, context: nil)
        ...
	}

	override func observeValue(forKeyPath keyPath: String?, of object: Any?, change: [NSKeyValueChangeKey : Any]?, context: UnsafeMutableRawPointer?) {
        if object is UserDefaults {
            // Here you can grab the values or just respond to it with an action.
        }
    }

	deinit {
        UserDefaults(suiteName: "my.group.identifier")?.removeObserver(self, forKeyPath: "message")
    }
}

However, take the above with a grain of salt, because on my machine, it doesn't work. I'm not sure why. The observer never gets triggered, and accessing the User Defaults by a button press or something just returns nil for the "message" key, every time. This is despite being able to retrieve it in the SafariWebExtensionHandler (where it was written in the first place). It even persists between runs!

I can't figure this out. So, on to something else...

Sending message from (native) extension → app? Please?\

At this point, I could think of only one way to get a message from my extension to my app: A custom url protocol handler. This is far from ideal, as I don't think I can send an entire webpage converted to markdown via url. There's a cap on the length. But, if I can get it working, perhaps there's something else I can do? Maybe write the data to a temp file then notify the app of said file's existence via pinging a url?

Well, either way, to wrap this post up, here's how I managed to get url handling working.

The first step is to update the Info.plist with a new url type. In my case, I chose markdownload as my url scheme

To handle the url, I need to add a function to my AppDelegate. From there I need to get the ViewController, which is theoretically attached to the main window, so I can pass the data there (as is the whole point of this exercise).

@main
class AppDelegate: NSObject, NSApplicationDelegate {
	...
    func application(_ application: NSApplication, open urls: [URL]) {
        let url = urls.first
        if(url != nil) {
            let mainVC = application.mainWindow?.contentViewController
            ( mainVC as! ViewController).handleUrl(url: url!)
        }
    }
}

In the ViewController, I've created that handleUrl function to get data if it receives a url of the form markdownload://message-recieved and fetch the data that was saved in the User Defaults. (As stated above, this isn't working for me, so I'm not actually doing anything with the data apart from forwarding it to the web view, but I thought I'd document it anyway)

    func handleUrl(url: URL) {
        print(url)
        var host:String?
        if #available(macOS 13.0, iOS 16.0, *) {
            host = url.host()
        } else {
            // Fallback on earlier versions
            host = url.host
        }
        
        if(host == "message-recieved") {
            let defaults = UserDefaults(suiteName: "my.group.identifier")
            let message = defaults?.string(forKey: "message")
            if(message != nil) {
				defaults?.removeObject(forKey: "message") // don't need it anymore
				self.webView.evaluateJavaScript("Messenger.recieveMessage(\(String(decoding: message!, as: UTF8.self)))") // send it to the webview javascript for processing
            }
        }
    }

And now that is all in place I can modify the web extension handler to ping the url after adding the data into the defaults

func beginRequest(with context: NSExtensionContext) {
	...
	defaults?.set(jsonData, forKey: "message")
	NSWorkspace.shared.open(URL(string:"markdownload://message-recieved")!)
    ...
}

That part of it actually works! I can hit breakpoints in the handleUrl function I made (something I found I can't do in the web extension handler, what with it being a separate process/sandbox)

First leg complete

Even though I never really got anything working to the extent I wanted to, I learned a lot in this first leg of my Safari journey. About how Apple handles extensions, inter-process communication... Would you believe I didn't even know Swift before starting this? Thankfully, I did know Objective-C, C# and JavaScript, which Swift feels like a mashup of, so it was easy to follow and understand the code samples provided by Apple.

My code so far is an absolute shambles, and I think I'm going to have to start my "from scratch" branch all over again from scratch. But I've learned, I've grown, and in my googling, I may have stumbled upon an approach that could work for browsers other than Safari, which opens the possibility of bringing whatever app functionality I end up creating to other browsers. We shall see.

For now, I'm tired from cutting through all this jungle, and need to set up camp, have a rest and consult my map and compass before setting out on the next leg of my Safari safari.