JS9: image display beyond the desktop

Eric Mandel
Alexey Vikhlinin
Center for Astrophysics | Harvard & Smithsonian

AAS 238: June 7, 2021

js9
In the beginning ...
  • SAOimage rose from the ashes of the Challenger disaster

  • SAOtng added a connection to external analysis via XPA

  • DS9 has long been the gold standard on the desktop

Steve
Steve Murray (aka the boss): Einstein/ROSAT HRI, Chandra HRC, ADS, 1944-2015
The world has moved beyond the desktop, as have our data analysis needs:
  • Web/cloud technology makes data available everywhere

  • Just in time for the upcoming avalanche of optical/IR data

  • [hears about Smart-X] ... OMG in X-ray astronomy too??
Dropbox/Cloud
How much can we put into the browser? All of DS9? More?
  • Responsive colormaps, scaling, contrast/bias, zoom

  • Create, edit, load, store regions of interest

  • Load and run external analysis tasks

  • Support a stable API for scripting

  • Utilize browser capabilities, such as blending

  • Multiple displays, synchronization, plugins, callbacks ...

  • For more demos, visit https://js9.si.edu

shout

That we exceeded our own expectations is largely thanks to the amazing software coming from the web community:
  • Emscripten: C to wasm translation to reuse trusted code

  • Socket.io: websocket-based messages to connect with our existing analysis environments

  • Electron: circle back to unify web apps and the desktop

Along with many, many others: HTML5 (canvas widget), fabric.js (regions graphics), jquery (programming ease), dynamic drive (light windows), flot (plotting), swis (context menus), arrive.js (mutation observer), spin.js (spinner), tinycolor.js (colormap generation), sprintf.js (C programmers always need sprintf!), bgrins(nice colorpicker), appcropolis (double-tap support), uuuidjs (uuid generation), lukemorton (caret positioning), sindresorhus(open programs, process list), image filter algorithms from lots of places ...

https://emscripten.org: run C/C++ code in the browser
  • Open-source compiler toolchain compiles to WebAssembly or JavaScript

  • Drop-in replacements for configure, make, gcc ...

  • Runtime environment with memory-based file system

  • Often requires only a thin layer to call from JavaScript

  • Wasm binary format runs at near-native speed
emscripten

https://socket.io: communicate with Python or the shell

  • The browser connects to the JS9 helper via socket.io

  • Then upgrades to a persistent websocket connection

  • The browser sends analysis requests to the helper and gets back results for display

  • pyjs9 and js9msg send API requests to the helper, which are relayed to the browser ...

  • ... browser responds back to the helper, which in turn sends results to the requester
websockets
https://www.electronjs.org/: Chrome-based desktop apps
  • Today, we still need desktop apps that foster "unconscious ease of use"

  • Electron combines Chrome's UI libraries with Node.js

  • App generation even handles Mac notarization, signing

  • This talk utilizes a web page loaded into Electron

  • We are working on a desktop app called Voyager, the next in the SAOimage line
whitehead
Of course, there are challenges ...
  • Most Browsers limit the amount of available memory
    • Firefox seems to be semi-unlimited
    • Chrome's limit seems to be 4Gb, iPad much less
    • Sounds reasonable, but we're used to unlimited
    • Even worse, FITS files get stored in memory ...

  • Sandboxing and other browser security restrictions
    • Electron: no default access to underlying file system
    • To access file system, turn off some security ... hmmm

  • The ubiquitous problem of downloading large data files has not gone away ...

  • Yeah, what about Python?
    • Mozilla working on it: Pyodide is Python in a browser
    • https://github.com/pyodide/pyodide
    • Not surprisingly, they use Emscripten
    • Can we actually use Pyodide, or is it just a toy?

  • And of course, there's always CSS to contend with ...
We need much deeper experience with web techniques

  • Especially if we want to increase analysis capabilities in browsers

  • E.g., I'm exploring an Emscripten-based cfitsio driver to access a file without downloading it completely

  • Can we make a serious run at Python-based analysis in the browser?

  • A web-based future requires more heads and hands: please join me!
Conclusion: the web community is doing amazing things, thanks to efforts by groups large and small. We should join them.
40+ years of image display at CfA:

hip
HEAO Image Processor: late 1970s
saoimage
SAOimage: late 1980s
saotng
SAOtng: early 1990s
ds9
DS9: late 1990s ... today
Disclaimer: my career as a tech prophet has been a bit uneven:
hip
Sun-2 workstation, circa 1983
"Why would you want to display two terminal windows at the same time?"
"No one is going to write a socket-based server just to display status reports!"
saotng
Mosaic browser, circa 1993
css
js9msg
https://js9.si.edu/js9/talks/aas-20210607/aastalk.html