Streams vs. documents
From Software libre para los países en desarrollo
Oh, this is a wiki, after registration (antispam, sorry), GO WILD editing it, it's yours. Please add references and ideas that you might find around (dear lazyweb!). Oh, I know most of the ideas aren't new -- the innovation lies at the intersection.
Contents |
Streams
Bryce Harrington has a very good idea here:
And then, as I looked around my desktop at the other things running on it - internet radio, IRC chat, mutt - that I noticed that I hardly ever use traditional "document-centric" applications. Everything running had something to do with dealing with _streams_ of information.
Others augment and second that:
People spend more time learning than creating.
Terminology
A stream is a flow of data from one or more data sources through several optional filters and yokes into an endpoint suited to display said data flow, most likely an activity center-type application or plasma widget. The data sources may generate data based on periodic pull from apps/daemons/widgets (think KMail periodic mail check), or (more conveniently) generate data and push that data based on asynchronous events (think tail -f or IMAP NOTIFY push-type email).
Streams are more frequently used / consulted than documents today
And we have a really suboptimal interface for dealing with streams. Ever wonder why KSystemLog and system-logviewer feel so wrong? Why your KMail is so cluttered? This is the reason: we're trying to shoehorn an app-, desktop- and document-centric paradigm around what essentially constitutes a live stream of data feeding into your brain. Apps are fine for editing documents, don't get me wrong, but apps and the desktop metaphor just get in the way of efficient information processing.
The problem with streams in modern desktop environments
I observe this very phenomenon in my desktop, right now and every single day. What do I have?
- A chat client, sitting on my notification area. This chat client is an entry point for interaction (creation) of new streams with other people. Contrast several open chat windows, each showing a separate conversation. Inconsistent UI. Presence history simply gets lost.
- A mail client, in a single window. Dealing with separate streams (folders where mails fall into via rules) is very cumbersome. Lots of clutter in the UI.
- An RSS newsreader. Formats my news in a river-of-news style, which isn't so bad but could use a series of enhancements. Again, unneeded clutter in the UI.
- Several terminal windows open and monitoring various processes and files. I can't start filtering the monitoring processes without interrupting them and reformulating the command line.
- A Web browser. With a stream constituted by the different stops on my navigation history. If I want to know which Web pages I visited two hours ago, I have to go into the application instead of consulting a list of things I've been doing.
- A music player. Playing a stream of music. What tracks did I play in the last hour?
- A system monitor like top. Showing me streams of activity and processes.
I posit that these applications are cumbersome to use today (especially in activities that require recall and time/spatial ordering) because they all isolate valuable data in their little silos. Projects like desktop search alleviate some of these issues somewhat, but they fail at providing extended/rich interaction with the search results they display, or they have high coupling. They also mostly don't provide real-time updates based on events - it would be extremely impractical to keep N desktop search windows as a means to effectively interact with the constant flow of data into my brain. It's not a matter of excising data from silos and indexing it -- it's a matter of adaptive, real-time display of that data in a persistent, unified, dynamic fashion.
What do I categorically almost never use? Icons. For the amount of information I (and most everyone) file around, they're useless. Icons are good if you have fifty of them. If you have ten thousand, good luck!
What's missing in the user experience
What do I sorely and constantly miss? This:
- It is generally impossible to filter streams in an ad-hoc fashion. You know, connect one endpoint of a stream (like an strace output) into another (like a grep). To do this in said example, you need to have a terminal open, and interrupt the current stream, then rewrite the command line. Another example: why can't I grep my Akregator news or KMail mail in real-time? The user interfaces added full-text-search as an afterthought, not as an integral way of browsing through data, and their bolt-on legacy shows. Live search folders in Evolution kind of sort of want to be this, but there's the ginormous UI load associated with the app, when what I want to focus on is only the data itself.
- It's obviously impossible to make stream (much less persistent ones) that connect sources, filters, and data display endpoints. This is what I mean about grepping my Akregator stuff -- e.g. I want a separate window showing a river of news about 'Dawkins' and 'religion'. Yahoo Pipes can do this, why do I have to subscribe to a Web service to do something that should be so easy on the desktop?
- It is impossible to group streams into themes or activities. I want my Dawkins news delivered as a stream and a separate stream with PZ Myers email. But I want them spatially nearby, because I'm likely to resort to one when I'm focused on the other.
- Timeline view in my file manager? Do I need to wait another ten years for this?
- Why can't I see my files as streams of themes, like music, pictures, or constrained by random tags?
- Having to manually intervene / script / automate whatever I want to publish based on a stream. Why is it 2008 and I still can't draw a nice pipeline to push my favorite songs of this week into my Web site? Maybe send a packaged stream to someone else via email?
- Having a short list of copy/paste history. Why can't I consult unlimited copy/paste?
- Web conversations. Why can't I persist something like a CoComment stream display that lets me reply to conversations as they are developing in real-time?
You see, in some cases, we have valuable data that we're losing. In other cases, the data display mechanism is very cumbersome because it's been shoehorned into the application metaphor instead of low-profile displays with emphasis on the data rather than the application itself.
What do we need to jump forward
This is a mess, so if you feel inclined to sort these ideas out in a way that each independent idea stands out and complements each other (reduce coupling between ideas, I guess!) I'd be infinitely thankful:
- Zoomable user interface. Remember the user interface (video) that let you arrange icons in piles? In bird's view, you see real representations of both documents and streams. I should be able to group streams by theme. Imagine if you could arrange those "icons" (some documents, some live streams) in "Work", "Fun", and other sorts of piles when on bird's view. Imagine being able to pile close together your "Work" email folder, a couple of documents and emails, some KNotes and the stream of conversation (email/IM) with your boss in a single pile. Alt+Tab would cycle you among the topologically closest activities in the "current" or "nearest" pile. When zooming closer, documents would just become larger, and streams would be dynamically replaced by the "window" of the following concept:
- Contemporary apps that work with / show streams should be reengineered as "stream managers" (kind of file managers but for streams) which should be activity centers or linear data/list/thumbnail/summary displays instead of around the authoring model of main window plus toolbar. Not that there's anything wrong with that, but why can't I detach the "Science" category in Akregator from the main "RSS list", or my Biology mail folder from Kmail's folder list, and pluck it onto my desktop? (note that Kmail filters could now be expressed using stream filters instead of directly in the app) Of course, the "views" (technically, windows) that I have defined via GUI manipulation should be remembered by the application and activated whenever I view them. This would give you real "live documents" (if uneditable except for their stream definition, at first). Think of this as the "file manager of the future", where you don't use the file manager to view just document icons, but stream elements (as lists, summaries or timelines) and their contents (using KPart-like technology) and interact with them.
- Apps that consume and produce streams, in principle, should be able to merge and combine streams in various fashions and output to different formats or to an application with the ability of letting me inspect the stream in a cursory fashion. I should be able to insert filters into the streams pipeline at any point, just like GStreamer elements can do for media.
- Merging streams into a separate icon should in principle be as simple as selecting the appropriate stream icon/thumbnail representations in the Zoomable user interface, right clicking them and choosing "Make new live document" or "Merge streams".
- Apps that display streams should also have generic controls that let me manipulate the stream and create a derived stream with a filter tacked to the end. Think about Akregator river of news, now you hit a button named "Display new window only with Science posts" or something like that, and BAM you get an icon representation. Or a log viewer application that lets you separately monitor (grep) only the kernel ringbuffer with a few UI actions. Or a timeline view of the latest 50 dance tracks you've been playing -- or another with a timeline of the 100 latest added tracks.
- Apps that work with documents (which are not streams, obviously, but could appear as stream elements in various types of streams) should strive to gain a unified interface. Branding is a concern and a hard sell here, because app authors don't want to lose their apps' uniqueness because that would mean losing some of the distinct identity / branding. But really, this is not a major point at the initial stages of implementation, since the thing that would unify everything is the generic streams viewer with persistency.
Things to solve / give more thought to
- how do we implement all of the required complexity that modern Akregator/Kmail type apps have? We'd basically need to port that UI into the appropriate module/component/part that will power the streams manager, without overcrowding it too much. Menus and toolbars are prime candidates to be whacked top/left (Fitt's law) as they should have been from the start, and they should only appear when a particular stream element is focused.
- "I'm out of RAM". Separate data source from app, keep data source light and running permanently (maybe some optimizations can be made in the case of user-initiated "refreshes" on data sources), load UI only when zoomable user interface deems it sensical, kill -TERM UI if not busy or in plain view, and memory pressure is high, maybe using compositing to save the latest window pixmap so we can quickly display it when the user pans around or taskflips and there's no need for UI interaction with that particular stream manager.
- posit, for example, a music player playing a Net radio station. The music player is now conceivably a source of song ID3 data, that could be reused as a stream somewhere else. in this case, it is the audio player that generates the stream. There needs to be a mechanism to collect these events from applications (we have one that we could reuse, it's called D-Bus). Same goes with document-centric apps and the Recent Files listing.
What about non-stream-oriented activities?
One of the problems that need to be figured out is entry points for non-stream objects, such as starting a convo with someone on your addressbook/IM list, or starting a document, or initiating a desktop search. Tasks which are really infrequent compared to stream consumption in modern desktops, but that represent the bread and butter of contemporary desktops, with concepts well-understood (the IM application, the Start menu, notification area icons -- which are really misused).
But we already have infrastructure for those: they can be nicely put into a task-oriented (NOT application-oriented) menu-type thing like Kmenu, Kickoff or Krunner (textual), or Khalki (KDE3) and, once launched, then persisted for future reference into an 'unsorted stream' pile perhaps (and be made an item in a "desktop events" type stream, for future chronological reference).
We should however consider the needs and preferred defaults of people coming from the (warning: sarcasm alert) evil proprietary camps, and default to their preferred mode of operation. Entry point UIs will probably still be needed (application menu, IM contact list, general incoming mail list, etcetera) but shouldn't be overcrowded with menus and buttons like the typical contemporary app. We don't want culture shock, but we do want to encourage the users to change their habits if they experiment and find it positive.
In KDE 4
In KDE 4, infrastructurally, we are going to have NEPOMUK-Strigi which will help a lot to provide almost-ready-made data sources for these ideas. And basically it's the same concept of decoupling data sources from display components that is embodied in Plasma -- only oriented to more keyboardable activity centers instead of to widgets. But if the data sources were there, there would be nothing stopping Plasma from consuming them. Heck, in fact, Plasma did hit the nail and the tech should be reused.
There's also no reason to destroy the document-centric paradigm. This could very well be compatible and run along it. Document-centric was supposed to kill the idea of the "app", and they almost did, but streams just wouldn't fit the "no-app" like documents did. Now we might finally be able to do so.
Here's a list of technologies which let us enable this sort of computing paradigm:
- Plasma
- D-Bus
- Telepathy
- Strigi/NEPOMUK
- RSS/Atom
- The KDEPIM data source framework (the name escapes me, damnit)
The hard trick is to turn contemporary data stores into streamable ones, I guess. And perhaps building the appropriate generic application to display them... although we have KParts.
Not all activities are worthwhile of shoehorning into the stream paradigm, though. The last tracks I played can very well appear in a stream, be susceptible of querying and filtering, but the playlist "window" still needs to exist and the Amarok search interface is more than adequate to find tracks to add into the playlist. KSCD is another good example. The notification area could persist to access frequently-used applications and data streams (clock is a great example).
In summary
It is not an option to not discuss these issues. Windows Vista and OSX already have facilities to power such a desktop metaphor. We might have a chance to leap in front of them here.
Very good ideas are here: Dataflow programming. The killer app, however, is the generalized stream viewer ("file manager of the future") and its persistency capabilities.
