I noticed OCR was particularly buggy when trying to use this service listening to podcasts. I also noticed that you can actually get the text content of the live transcribe window using the accessibility service, so I just did that and took the last 10 or so words. That seemed to be enough/not too much in some limited testing to keep up with scrolling but there's probably some more tuning that could be done.

This will partially fix #8 too (at least as far as it can be fixed from this service's side - live caption still has to support the languages) since we're not depending on OCR anymore.

HUGE thanks to alexschneider's https://github.com/KyleFin/live-scroll-transcript/pull/9

Get caption text directly from AccessibilityEvent source. This greatly simplifies the code and should help performance (CPU / battery usage, not require downloading or supporting OCR libraries).

This provides access to a large amount of historical caption text, but we'll only process the most recent caption text.

Added notes about numCaptionCharsToLookAt and captionViewScrollsThreshold.

This prevents an IllegalState exception which crashes the accessibility service if root is null (for example if captions scroll and window is requested while a user is switching between apps).

This is a working version that uses OCR to read recent Live Captions and search for them in the current screen's accessibility tree.

Demo video: https://photos.app.goo.gl/erc5cugnLUC4tfp8A

Adjusting frequency of how often we search for text and how much text we search (from captions and in accessibility tree) can improve the user experience.

For example:

• More frequent/granular searching for articles/podcasts.
• Less frequent but more accurate scrolling in a full novel (https://github.com/KyleFin/live-scroll-transcript/issues/5).

Users may have more context to decide what is appropriate.

My hesitation in making this configurable is that I don't want users to have to deeply understand the inner workings of how this feature works, and I'm not sure how best to make the user experience simple.

When I was using OCR to read Live Captions, I was thinking to use numCaptionLines to determine scroll frequency. My thought was that it may be intuitive for users that "we scroll every time the caption box has been entirely replaced" so if you want to scroll less frequently and consider more caption text (like in an ebook), you can double tap the caption box to expand it.

Another solution might be to have configurable options in Settings > Accessibility > Live Scroll Transcript.

This is dependent on Live Caption ~and ML Kit~ supporting non-English languages. Once that is available it shouldn't be hard to add.

Consider looking into StringSearch which may be helpful for matching text in different languages.

We could also translate captions and/or transcripts into different languages. (E.g. listen in English and read along in a Spanish transcript) This could be done now for English audio ~and text in languages supported by ML Kit~.

The initial implementation works well for text that is all in one page (Chrome or apps with webviews like Gospel Library), broken down by paragraph into small AccessibilityNodeInfos that can be accessed from rootInActiveWindow. (https://github.com/KyleFin/live-scroll-transcript/issues/6 is about communicating this to users)

There are many use-cases I would like to support. Here are some thoughts:

Use cases

(See sample code for an example of how to print a11y tree)

• Scrolling apps where text is in one giant node or not exposed in a11y tree (Google Docs, Drive, Gmail)
  • If we can't access a11y tree, we might use OCR and swipe gestures to scroll. One difficulty is knowing if we should continue scrolling when images filling the whole screen or if we miss scrolling and audio goes beyond the current screen.
  • Create a generic "ctrl + f" macro functionality to find text in the current app?
• PDFs
  • Could be very useful. OCR and swipe gestures. In addition to images, another complication is knowing which way to swipe (how to handle multiple columns on the same page)
• Page-turning apps (Hoopla, Libby, Google Play Books)
  • Some may provide good a11y info, but only for the current page. We can probably turn page quite reliably with just a tap or swipe gesture. Also may support NEXT_PAGE action.
  • How to know which apps support page turning (can AccessibilityService query support for NEXT_PAGE?)
  • How to decide when to turn the page? (Once we've matched text on the page, turn page immediately to stay ahead? Wait until audio goes to next page and try to keep up? What if there are images?)
• Kindle (page-turn or continuous)
  • This may be lower priority because some Kindle books have WhisperSync with Audible to automatically sync text/audio. Live Scroll would extend support for using different versions of the media and work for virtually any title.
  • Page-turn is same as other page-turning apps and provides a11y info for current page.
  • Continuous scroll provides no text a11y info (confirm if implementing), but we could scroll with OCR and gestures.
    • We can know we're in page-turn or continuous mode from package name (Kindle) and content description in KRFView (has text for page, empty for continuous). Confirm if implementing.
  • Same concerns about images and how to recover if we fall behind.
• Photos (panning instead of scrolling)
  • Very low priority, but it could be neat.
  • Requires OCR and gestures.

Solutions

• OCR

  • How to decide where to search for text.
    • Everywhere except in Live Caption box? Exclude system headers?
  • May be helpful to use window changed a11y events to know when current screen has changed (maybe user switched apps so we should start or stop trying to use OCR to scroll).

• Gestures

  • We may prefer sending more direct commands if possible (show_on_screen, next paragraph, scroll, etc) because they're more targeted and probably efficient.
  • See dispatchGesture documentation and sample GestureDescription code below.
  • Pros: work with any app
  • Cons: could interfere with other apps. If we fall behind audio, it's hard to recover.

• How to determine how to swipe/tap?

  • Curated package/view list? (Specific apps to allow or block)
    • ML model to determine given a screenshot and/or a11y tree how to interact with a given app?
  • Fallback after attempting to use a11y tree.
    • Need to notify user if this is happening so we're not needlessly trying to scroll when user doesn't want scrolling.
  • Can a11y service determine if current screen is scrollable or supports page-turn?

• How to determine where to swipe?

  • Start from where word is matched and swipe to top of screen?

Sample code

  // Logging current a11y tree (very similar to getNodesContainingWord)
  private fun printAccessibilityTree(root: AccessibilityNodeInfo, level: Int) {
        if (root == null) return
        Log.d(tag, "Node at level %s with childCount %s: %s".format(level, root.getChildCount(), root))
        for (i in 1..root.childCount) {
            root.getChild(i - 1)?.let { printAccessibilityTree(it, level + 1) }
        }
    }

  // From AccessibilityService:
  printAccessibilityTree(this.rootInActiveWindow, 0)

  private GestureDescription advanceTextGestureDescription() {
    if (currGestureRegion.equals(paginatedAppGestureRegion)) {
      return tapRightSideOfScreen(); // swipeLeftGestureDescription();
    } else if (currGestureRegion.equals(scrollableAppGestureRegion)) {
      return swipeUpGestureDescription(currGestureRegion.bottom);
    }
    return null;
  }

  private GestureDescription swipeUpGestureDescription(int initialY) {
    // ** Swipe up (e.g. to scroll down). */
    Path path = new Path();
    path.moveTo(currGestureRegion.left, initialY);
    path.lineTo(currGestureRegion.left, currGestureRegion.top);
    StrokeDescription strokeDescription =
        new StrokeDescription(path, /*startTime=*/ 0L, /*duration (in ms)=*/ 500L);
    return new GestureDescription.Builder().addStroke(strokeDescription).build();
  }

  private GestureDescription swipeLeftGestureDescription() {
    // ** Swipe left (e.g. to turn to next page). */
    Path longSlowPath = new Path();
    longSlowPath.moveTo(900, 1000);
    longSlowPath.lineTo(200, 1000);

    Path flickPath = new Path();
    flickPath.moveTo(200, 1000);
    flickPath.lineTo(100, 1000);

    StrokeDescription strokeDescription =
        new StrokeDescription(
            longSlowPath, /*startTime=*/ 0L, /*duration (in ms)=*/ 400L, /* willContinue= */ true);
    strokeDescription.continueStroke(
        flickPath, /*startTime=*/ 0L, /*duration (in ms)=*/ 100L, /* willContinue= */ false);

    return new GestureDescription.Builder().addStroke(strokeDescription).build();
  }

  private GestureDescription tapRightSideOfScreen() {
    // ** Tap right side of screen (e.g. to turn to next page). */
    Path path = new Path();
    path.moveTo(screenHeight / 2, 5 * (screenWidth / 6));
    StrokeDescription strokeDescription =
        new StrokeDescription(path, /*startTime=*/ 0L, /*duration (in ms)=*/ 10L);
    return new GestureDescription.Builder().addStroke(strokeDescription).build();
  }

In app description:

• Works with any audio that works with Live Caption (bluetooth, not casting)
• Any text that TalkBack can read? (I don't think this is true and this issue is to find a better description) Explain that some apps block Live Caption or screenshots

The initial implementation works well for text that is all in one page (Chrome or apps with webviews like Gospel Library), broken down by paragraph into small AccessibilityNodeInfos that can be accessed from rootInActiveWindow.

I want to determine how to clearly identify and communicate to users (in documentation or pop-ups) which apps are supported and why.

Perhaps there is a way to identify apps which follow a11y guidelines and work well with TalkBack (IIUC that set of apps should heavily overlap with apps we can scroll).

The initial implementation works well for articles, podcasts, etc but not as well for longer text (i.e. the full text of Dracula on Project Gutenberg)

A few of my thoughts about how to improve performance:

• We may not want to search as frequently. (Scrolling would be more accurate but more delayed.)

  • Frequency time or captionViewScrollsThreshold could be user-configurable, but I'd strongly prefer having it "just work" since tuning these parameters may be finicky, especially without the logs to really understand what's going on.
  • DO THIS? Increase captionViewScrollsThreshold? If user expands Live Caption box this would also allow having more caption text to build up, possibly giving a better keyword (longest word).
    • The more I think about this, the more I love this solution. We could adjust threshold dynamically based on how many lines are in the caption box. We can document/suggest that users should expand the box for longer pages to allow better keyword selection and prevent too-frequent scrolling. I don't love that the caption box will cover up more of the screen, but it seems pretty intuitive to me that instead of reasoning about frequency you just need to understand "Live Scroll will attempt to scroll every time the Live Caption text has been completely replaced. More lines == less frequent scrolling and higher chance of matching"
  • Add a timer to enforce how frequently we scroll?
  • Reducing frequency should fix screenshot-too-soon errors.

• We want to avoid searching the full page more than necessary

  • Once we've successfully scrolled a node, we could start searching from there (or its parent) first instead of from root every time.
    • Maybe we can use the "next" actions used by TalkBack to search for or display the next paragraph, etc.
    • This may resolve the "stack too big" error I've seen a few times.

Toasts suggesting page refresh and screenshot too soon in initialImpl are helpful.

Consider adding more messaging for things like:

• Attempted to search for text in current screen (every time so user knows we attempted and they can understand frequency of attempts? Every N times to remind user the service is running and they may want to turn it off?)
• After failed to find match N times? (Remind user the service is on when they aren't actively using it. I want to investigate performance of leaving the service running constantly. Maybe it's fine.)
• Scrolled
• Highlight text that was matched?
• Other failures?