Thread Subject: Re: Cognitive - compatibility

Return to this mailing list's archives

From: Andi Snow-Weaver
Date: Mon, Mar 05 2007 2:15 PM
Subject: Re: Cognitive - compatibility

My action item from last week's meeting was to update the cognitive
proposal dealing with word lookup, spelling correction, and text to address
concerns raised on the mailing list and in the meeting.

* Software for rendering or authoring text shall provide the following
features or support for assistive technology that implements the features:
** the ability to determine the definition of a word in the rendered text
** the ability to read the rendered text outloud

* Software for authoring text shall provide the ability to check for
spelling errors and correct them or support for assistive technology that
implements the feature.

Comments?

Andi

From: Slatin, John M
Date: Mon, Mar 05 2007 3:45 PM
Subject: Re: Cognitive - compatibility

Andi wrote:

<q>* Software for rendering or authoring text shall provide the
following features or support for ...</q>

Is there software for authoring text that doesn't also render text?

Is there some way we want to constrain what "render" means here?

John

"Good design is accessible design."

Dr. John M. Slatin, Director
Accessibility Institute
University of Texas at Austin
FAC 248C
1 University Station G9600
Austin, TX 78712
ph 512-495-4288, fax 512-495-4524
email = EMAIL ADDRESS REMOVED =
Web http://www.utexas.edu/research/accessibility



-----Original Message-----
From: = EMAIL ADDRESS REMOVED =
[mailto: = EMAIL ADDRESS REMOVED = ] On Behalf Of Andi
Snow-Weaver
Sent: Monday, March 05, 2007 3:09 PM
To: = EMAIL ADDRESS REMOVED =
Subject: Re: [teitac-websoftware] Cognitive - compatibility



My action item from last week's meeting was to update the cognitive
proposal dealing with word lookup, spelling correction, and text to
address concerns raised on the mailing list and in the meeting.

* Software for rendering or authoring text shall provide the following
features or support for assistive technology that implements the
features:
** the ability to determine the definition of a word in the rendered
text
** the ability to read the rendered text outloud

* Software for authoring text shall provide the ability to check for
spelling errors and correct them or support for assistive technology
that implements the feature.

Comments?

Andi

From: Andi Snow-Weaver
Date: Mon, Mar 05 2007 4:20 PM
Subject: Re: Cognitive - compatibility

John wrote:

"Is there software for authoring text that doesn't also render text?"

John, there is software for rendering text that doesn't also allow
authoring, such as the Adobe reader, MS Office viewers, or even Web
browsers. If such software uses accessibility APIs sufficiently, then ATs
for people with cognitive disabilities could provide the function.

Andi

From: Slatin, John M
Date: Mon, Mar 05 2007 4:55 PM
Subject: Re: Cognitive - compatibility

Andi replied to my comment as follows:

<andi>
John, there is software for rendering text that doesn't also allow
authoring, such as the Adobe reader, MS Office viewers, or even Web
browsers. If such software uses accessibility APIs sufficiently, then
ATs for people with cognitive disabilities could provide the function.
</andi>

I agree, thanks. But I thought such "rendering-only" software was
covered by your second provision (I thought there was one that applied
to software that "authors or renders text" and a second one that applies
to software that only "renders" text. If these were meant as alternative
possibilities, never mind...

John

"Good design is accessible design."

Dr. John M. Slatin, Director
Accessibility Institute
University of Texas at Austin
FAC 248C
1 University Station G9600
Austin, TX 78712
ph 512-495-4288, fax 512-495-4524
email = EMAIL ADDRESS REMOVED =
Web http://www.utexas.edu/research/accessibility



-----Original Message-----
From: = EMAIL ADDRESS REMOVED =
[mailto: = EMAIL ADDRESS REMOVED = ] On Behalf Of Andi
Snow-Weaver
Sent: Monday, March 05, 2007 5:13 PM
To: TEITAC Web/Software Subcommittee
Subject: Re: [teitac-websoftware] Cognitive - compatibility


John wrote:

"Is there software for authoring text that doesn't also render text?"

John, there is software for rendering text that doesn't also allow
authoring, such as the Adobe reader, MS Office viewers, or even Web
browsers. If such software uses accessibility APIs sufficiently, then
ATs for people with cognitive disabilities could provide the function.

Andi

From: Gregg Vanderheiden
Date: Mon, Mar 05 2007 11:30 PM
Subject: Re: Cognitive - compatibility

Nice to see some concrete cognitive proposals.


Gregg
-- ------------------------------
Gregg C Vanderheiden Ph.D.



> -----Original Message-----
> From: = EMAIL ADDRESS REMOVED =
> [mailto: = EMAIL ADDRESS REMOVED = ] On Behalf
> Of Andi Snow-Weaver
> Sent: Monday, March 05, 2007 3:09 PM
> To: = EMAIL ADDRESS REMOVED =
> Subject: Re: [teitac-websoftware] Cognitive - compatibility
>
>
> My action item from last week's meeting was to update the
> cognitive proposal dealing with word lookup, spelling
> correction, and text to address concerns raised on the
> mailing list and in the meeting.
>
> * Software for rendering or authoring text shall provide the
> following features or support for assistive technology that
> implements the features:
> ** the ability to determine the definition of a word in the
> rendered text
> ** the ability to read the rendered text outloud
>
> * Software for authoring text shall provide the ability to
> check for spelling errors and correct them or support for
> assistive technology that implements the feature.
>
> Comments?
>
> Andi
>
>

From: Hoffman, Allen
Date: Tue, Mar 06 2007 6:00 AM
Subject: Re: Cognitive - compatibility

I think this could be combined with the authoring and user-agent items.
It might make the items too long if not worded very well.

I'd also be interested in the following general analysis:
Is there AT now that can allow the user to highlight a word in anything
on screen and offer spelling and/or dictionary function?
Ditto for reading out loud--but I'm sure this part is available.

Second, what are the underlying obstacles to such software beyond such
things as scanned images of text?

In other words if a software application meets the other existing
software requirements now, will AT of this type generally work? If so,
is this requirement redundant,, or do we want to require this be a
built-in feature in preference to add-on?




Allen Hoffman -- 202-447-0303

-----Original Message-----
From: = EMAIL ADDRESS REMOVED =
[mailto: = EMAIL ADDRESS REMOVED = ] On Behalf Of Gregg
Vanderheiden
Sent: Tuesday, March 06, 2007 1:28 AM
To: 'TEITAC Web/Software Subcommittee'
Subject: Re: [teitac-websoftware] Cognitive - compatibility



Nice to see some concrete cognitive proposals.


Gregg
-- ------------------------------
Gregg C Vanderheiden Ph.D.



> -----Original Message-----
> From: = EMAIL ADDRESS REMOVED =
> [mailto: = EMAIL ADDRESS REMOVED = ] On Behalf Of Andi
> Snow-Weaver
> Sent: Monday, March 05, 2007 3:09 PM
> To: = EMAIL ADDRESS REMOVED =
> Subject: Re: [teitac-websoftware] Cognitive - compatibility
>
>
> My action item from last week's meeting was to update the cognitive
> proposal dealing with word lookup, spelling correction, and text to
> address concerns raised on the mailing list and in the meeting.
>
> * Software for rendering or authoring text shall provide the following

> features or support for assistive technology that implements the
> features:
> ** the ability to determine the definition of a word in the rendered
> text
> ** the ability to read the rendered text outloud
>
> * Software for authoring text shall provide the ability to check for
> spelling errors and correct them or support for assistive technology
> that implements the feature.
>
> Comments?
>
> Andi
>
>

From: Lybarger, Barbara (MOD)
Date: Tue, Mar 06 2007 7:50 AM
Subject: Re: Cognitive - compatibility

How about adding a grammar checker? For a lot of folks with dyslexia,
the grammar checker is as important as the spell checker. It surfaces a
variety of perception errors, such as repeated words (e.g. "the the"),
omitted words (e.g. "the" is missing all together), and noun/verb
agreement (e.g. plural noun with a singular verb and vis versa). Many
people with dyslexia do not perceive those sorts of errors.

Barbara Lybarger
Massachusetts Office on Disability


-----Original Message-----
From: = EMAIL ADDRESS REMOVED =
[mailto: = EMAIL ADDRESS REMOVED = ] On Behalf Of Andi
Snow-Weaver
Sent: Monday, March 05, 2007 4:09 PM
To: = EMAIL ADDRESS REMOVED =
Subject: Re: [teitac-websoftware] Cognitive - compatibility


My action item from last week's meeting was to update the cognitive
proposal dealing with word lookup, spelling correction, and text to
address concerns raised on the mailing list and in the meeting.

* Software for rendering or authoring text shall provide the following
features or support for assistive technology that implements the
features:
** the ability to determine the definition of a word in the rendered
text
** the ability to read the rendered text outloud

* Software for authoring text shall provide the ability to check for
spelling errors and correct them or support for assistive technology
that implements the feature.

Comments?

Andi

From: Lazzaro, Joe (ITD)
Date: Tue, Mar 06 2007 2:55 PM
Subject: Re: Cognitive - compatibility

I like Barbara's suggestion. It remediates the problem using
off-the-shelf solutions, and this will for obvious reasons be more cost
effective.

Joe




Joe Lazzaro
Manager: Assistive Technology Group
Information Technology Division
Commonwealth of Massachusetts
One Ashburton Place
Room 1601
Boston, MA 02108
Voice: 617-626-4410
Email: = EMAIL ADDRESS REMOVED =
Web: www.Mass.gov/ITD


-----Original Message-----
From: = EMAIL ADDRESS REMOVED =
[mailto: = EMAIL ADDRESS REMOVED = ] On Behalf Of
Lybarger, Barbara (MOD)
Sent: Tuesday, March 06, 2007 9:47 AM
To: TEITAC Web/Software Subcommittee
Subject: Re: [teitac-websoftware] Cognitive - compatibility

How about adding a grammar checker? For a lot of folks with dyslexia,
the grammar checker is as important as the spell checker. It surfaces a
variety of perception errors, such as repeated words (e.g. "the the"),
omitted words (e.g. "the" is missing all together), and noun/verb
agreement (e.g. plural noun with a singular verb and vis versa). Many
people with dyslexia do not perceive those sorts of errors.

Barbara Lybarger
Massachusetts Office on Disability


-----Original Message-----
From: = EMAIL ADDRESS REMOVED =
[mailto: = EMAIL ADDRESS REMOVED = ] On Behalf Of Andi
Snow-Weaver
Sent: Monday, March 05, 2007 4:09 PM
To: = EMAIL ADDRESS REMOVED =
Subject: Re: [teitac-websoftware] Cognitive - compatibility


My action item from last week's meeting was to update the cognitive
proposal dealing with word lookup, spelling correction, and text to
address concerns raised on the mailing list and in the meeting.

* Software for rendering or authoring text shall provide the following
features or support for assistive technology that implements the
features:
** the ability to determine the definition of a word in the rendered
text
** the ability to read the rendered text outloud

* Software for authoring text shall provide the ability to check for
spelling errors and correct them or support for assistive technology
that implements the feature.

Comments?

Andi

From: Peter Korn
Date: Tue, Mar 06 2007 5:55 PM
Subject: Re: Cognitive - compatibility

Hi Allen,

> Is there AT now that can allow the user to highlight a word in anything
> on screen and offer spelling and/or dictionary function?
> Ditto for reading out loud--but I'm sure this part is available.
>

Software like this used to exist for the Macintosh, a long time ago. It
was really quite slick.

To do this today, you would need to either have an off-screen model (and
all of the video hooks and reverse engineering implied by such) like
those in our screen readers, or be running on a system which provides
all of the text that is on the screen via a standard API. We're
starting to get there on the Macintosh. We're already there in GNOME
UNIX systems. But in neither place do we have such a utility.

Hmmm.... That'd be a good Google Summer of Code project for a summer
intern...

> Second, what are the underlying obstacles to such software beyond such
> things as scanned images of text?
>

Having a well-respected API for getting all of the text & text
boundaries from the apps running on the system.

> In other words if a software application meets the other existing
> software requirements now, will AT of this type generally work? If so,
> is this requirement redundant,, or do we want to require this be a
> built-in feature in preference to add-on?
>

I personally think this is the kind of feature that works best as a
separate utility - one user interface that works the same everywhere on
the desktop.


Regards,

Peter Korn
Accessibility Architect,
Sun Microsystems, Inc.

From: Jonathan Avila
Date: Wed, Mar 07 2007 6:20 AM
Subject: Re: Cognitive - compatibility

<quote>
To do this today, you would need to either have an off-screen model (and
all of the video hooks and reverse engineering implied by such) like
those in our screen readers, or be running on a system which provides
</quote>

I also have concerns about how this would work for non-content materials.
For example, are you asking for the ability to select static text in a
dialog box with the keyboard in order to get a definition of the word(s).
Providing keyboard access to such an item is likely to be confusing,
currently difficult to implement, and may cause issues for keyboard only
users. Providing a mouse based hit test when the mouse is placed over the
word such as accessibleObjectFromPoint would work in cases were MSAA or
another API was available. However, even in new Windows form libraries
static text isn't always written as separate objects that can be accessed by
point. Often static text is not it's own object. If the text is exposed
through OS settings then technology such as this should be 3rd party
software like a screen reader that would build an off-screen model of the
text and then provide this type of access.

Jonathan

-----Original Message-----
From: = EMAIL ADDRESS REMOVED =
[mailto: = EMAIL ADDRESS REMOVED = ] On Behalf Of Peter Korn
Sent: Tuesday, March 06, 2007 7:51 PM
To: TEITAC Web/Software Subcommittee
Subject: Re: [teitac-websoftware] Cognitive - compatibility

Hi Allen,

> Is there AT now that can allow the user to highlight a word in anything
> on screen and offer spelling and/or dictionary function?
> Ditto for reading out loud--but I'm sure this part is available.
>

Software like this used to exist for the Macintosh, a long time ago. It
was really quite slick.

To do this today, you would need to either have an off-screen model (and
all of the video hooks and reverse engineering implied by such) like
those in our screen readers, or be running on a system which provides
all of the text that is on the screen via a standard API. We're
starting to get there on the Macintosh. We're already there in GNOME
UNIX systems. But in neither place do we have such a utility.

Hmmm.... That'd be a good Google Summer of Code project for a summer
intern...

> Second, what are the underlying obstacles to such software beyond such
> things as scanned images of text?
>

Having a well-respected API for getting all of the text & text
boundaries from the apps running on the system.

> In other words if a software application meets the other existing
> software requirements now, will AT of this type generally work? If so,
> is this requirement redundant,, or do we want to require this be a
> built-in feature in preference to add-on?
>

I personally think this is the kind of feature that works best as a
separate utility - one user interface that works the same everywhere on
the desktop.


Regards,

Peter Korn
Accessibility Architect,
Sun Microsystems, Inc.

From: Peter Korn
Date: Wed, Mar 07 2007 7:10 AM
Subject: Re: Cognitive - compatibility

Hi Jonathan,

Hmmm... I think we are close to saying the same thing here. I come at
this from the view of technology implementation. If we decide that it
is desirable to support the use case of a user (potentially with a
cognitive disability) getting a definition of every non-graphic piece of
text on the screen, then we must either:

a. build a dictionary into every application that renders text, with
some guidelines for how to invoke the dictionary lookup for every piece
of text it renders (dialog boxes and menu items as well as content), or
b. define one or more "rendered text retrieval" APIs which all
applications support for every piece of text they render (which is then
used by a 3rd party dictionary/definition lookup program which defines
its own mechanism for invoking the dictionary lookup), or
c. hope that one or more independent software developers sees enough of
a market in developing such software to duplicate the
hacks/reverse-engineering/off-screen-model work of our existing Windows
screen readers in order to create such software, or
d. some combination of the above

Please don't base your assumptions of what is and isn't possible on what
MSAA does and does not provide. We have done a pretty remarkable job of
providing exactly this support in the UNIX/GNOME desktop. With the
collection of applications that Sun ships with Solaris 10 and later, and
that Ubuntu ships with Ubuntu 6.10 and later (to name two OSes I'm
fairly familiar with), exactly this information is provided via a
supported API, and compliance with/implementation of this API is quite
good. No off-screen model would be needed to implement such a feature,
greatly lowering the cost to a 3rd party (e.g. AT) developer to
developing such software.

I say this with confidence because the flat-review feature of our Orca
screen reader is able to review word by word (and letter by letter)
every piece of non-graphic text in the applications we ship in Solaris.
For example, starting with the top-left piece of text in the StarOffice
Writer application, Orca will speak and highlight every word of text,
from "File" in the menu bar, through all of the text content, and ending
with the static text in the statusbar at the bottom of the page. In
fact, it will also speak and highlight the icons in the toolbars (as
well as the words of the font style and font name in the combo-boxes in
the toolbars). No off-screen model is ever used to do this. No
hacks/reverse-engineering is ever used to do this.


Regards,

Peter Korn
Accessibility Architect,
Sun Microsystems, Inc.

> <quote>
> To do this today, you would need to either have an off-screen model (and
> all of the video hooks and reverse engineering implied by such) like
> those in our screen readers, or be running on a system which provides
> </quote>
>
> I also have concerns about how this would work for non-content materials.
> For example, are you asking for the ability to select static text in a
> dialog box with the keyboard in order to get a definition of the word(s).
> Providing keyboard access to such an item is likely to be confusing,
> currently difficult to implement, and may cause issues for keyboard only
> users. Providing a mouse based hit test when the mouse is placed over the
> word such as accessibleObjectFromPoint would work in cases were MSAA or
> another API was available. However, even in new Windows form libraries
> static text isn't always written as separate objects that can be accessed by
> point. Often static text is not it's own object. If the text is exposed
> through OS settings then technology such as this should be 3rd party
> software like a screen reader that would build an off-screen model of the
> text and then provide this type of access.
>
> Jonathan
>
> -----Original Message-----
> From: = EMAIL ADDRESS REMOVED =
> [mailto: = EMAIL ADDRESS REMOVED = ] On Behalf Of Peter Korn
> Sent: Tuesday, March 06, 2007 7:51 PM
> To: TEITAC Web/Software Subcommittee
> Subject: Re: [teitac-websoftware] Cognitive - compatibility
>
> Hi Allen,
>
>
>> Is there AT now that can allow the user to highlight a word in anything
>> on screen and offer spelling and/or dictionary function?
>> Ditto for reading out loud--but I'm sure this part is available.
>>
>>
>
> Software like this used to exist for the Macintosh, a long time ago. It
> was really quite slick.
>
> To do this today, you would need to either have an off-screen model (and
> all of the video hooks and reverse engineering implied by such) like
> those in our screen readers, or be running on a system which provides
> all of the text that is on the screen via a standard API. We're
> starting to get there on the Macintosh. We're already there in GNOME
> UNIX systems. But in neither place do we have such a utility.
>
> Hmmm.... That'd be a good Google Summer of Code project for a summer
> intern...
>
>
>> Second, what are the underlying obstacles to such software beyond such
>> things as scanned images of text?
>>
>>
>
> Having a well-respected API for getting all of the text & text
> boundaries from the apps running on the system.
>
>
>> In other words if a software application meets the other existing
>> software requirements now, will AT of this type generally work? If so,
>> is this requirement redundant,, or do we want to require this be a
>> built-in feature in preference to add-on?
>>
>>
>
> I personally think this is the kind of feature that works best as a
> separate utility - one user interface that works the same everywhere on
> the desktop.
>
>
> Regards,
>
> Peter Korn
> Accessibility Architect,
> Sun Microsystems, Inc.
>

From: Stephen Baum
Date: Wed, Mar 07 2007 8:30 AM
Subject: Re: Cognitive - compatibility

Hi Allen.

You asked if there was AT that would allow the user to highlight a
word in anything on screen, and offer spelling and/or dictionary function.

A number of programs come close, but the strict answer would be no,
there isn't. Nor, for that matter, is there any AT that will read out
loud any word highlighted on the screen.

I guess the first qualifier would be to ask what software is being
used to generate the word on the screen, and what operating system.
I'll limit my response to Windows, since its what I know best.

All sorts of programs, including some freeware, will read, define, or
spell aloud some text, displayed by some programs, on a screen. These
programs can be quite useful, but none of them will read everything.

First, as you indicated in your message, the text that is being
displayed might be a picture of text, produced by scanning hard copy.
For that, you need OCR. Since OCR is designed to read scanned images
- not images posted onto a screen - the obvious solution of using
capturing the display on the screen and sending it through OCR will
be (besides slow) fairly inaccurate.

Secondly, there is text that is displayed in such a manner that it
cannot be pointed to or highlighted, and that cannot be extracted in
standard ways from the active application. Usually this is done as a
part of digital rights management - the application is locking down
the content to prevent other applications from stealing it. This also
prevents adaptive technology from providing alternative access to the
text. Some, but by no means all, of those applications provide access
to trusted programs (including, say, a screen reader or two) through
specialized APIs.

Stephen Baum
Kurzweil Educational Systems
781-276-0618

At 08:39 AM 3/6/2007, you wrote:
>Date: Tue, 6 Mar 2007 07:54:20 -0500
>From: "Hoffman, Allen" < = EMAIL ADDRESS REMOVED = >
>Subject: Re: [teitac-websoftware] Cognitive - compatibility
>To: "TEITAC Web/Software Subcommittee"
> < = EMAIL ADDRESS REMOVED = >
>Message-ID:
> < = EMAIL ADDRESS REMOVED = >
>Content-Type: text/plain; charset="us-ascii"
>
>I think this could be combined with the authoring and user-agent items.
>It might make the items too long if not worded very well.
>
>I'd also be interested in the following general analysis:
>Is there AT now that can allow the user to highlight a word in anything
>on screen and offer spelling and/or dictionary function?
>Ditto for reading out loud--but I'm sure this part is available.
>
>Second, what are the underlying obstacles to such software beyond such
>things as scanned images of text?
>
>In other words if a software application meets the other existing
>software requirements now, will AT of this type generally work? If so,
>is this requirement redundant,, or do we want to require this be a
>built-in feature in preference to add-on?
>
>
>
>
>Allen Hoffman -- 202-447-0303

From: Hoffman, Allen
Date: Wed, Mar 07 2007 9:25 AM
Subject: Re: Cognitive - compatibility

Thanks to all who are responding, this is a great discussion.

So do any of you have suggestions to refine the language proposed by
Andi?

I think I'd consider Peter Korn's writing carefully and think both
outside the scope of any vendor space, and think about near future when
things will have changed due to continuous refinements in all sets of
E&IT.

Personally I'm not convinced that requiring functionality to define
identified words, spell/grammar check text, and other items is
impossible, but will be easier and harder dependent upon the
accessibility layers of various sets of E&IT. This doesn't mean it
isn't something needed for access to information and data by some people
with disabilities, and so should be considered carefully as a
recommended addition to our requirements.



Allen hoffman -- 202-447-0303

-----Original Message-----
From: = EMAIL ADDRESS REMOVED =
[mailto: = EMAIL ADDRESS REMOVED = ] On Behalf Of Stephen
Baum
Sent: Wednesday, March 07, 2007 10:31 AM
To: = EMAIL ADDRESS REMOVED =
Subject: Re: [teitac-websoftware] Cognitive - compatibility

Hi Allen.

You asked if there was AT that would allow the user to highlight a word
in anything on screen, and offer spelling and/or dictionary function.

A number of programs come close, but the strict answer would be no,
there isn't. Nor, for that matter, is there any AT that will read out
loud any word highlighted on the screen.

I guess the first qualifier would be to ask what software is being used
to generate the word on the screen, and what operating system.
I'll limit my response to Windows, since its what I know best.

All sorts of programs, including some freeware, will read, define, or
spell aloud some text, displayed by some programs, on a screen. These
programs can be quite useful, but none of them will read everything.

First, as you indicated in your message, the text that is being
displayed might be a picture of text, produced by scanning hard copy.
For that, you need OCR. Since OCR is designed to read scanned images
- not images posted onto a screen - the obvious solution of using
capturing the display on the screen and sending it through OCR will be
(besides slow) fairly inaccurate.

Secondly, there is text that is displayed in such a manner that it
cannot be pointed to or highlighted, and that cannot be extracted in
standard ways from the active application. Usually this is done as a
part of digital rights management - the application is locking down the
content to prevent other applications from stealing it. This also
prevents adaptive technology from providing alternative access to the
text. Some, but by no means all, of those applications provide access to
trusted programs (including, say, a screen reader or two) through
specialized APIs.

Stephen Baum
Kurzweil Educational Systems
781-276-0618

At 08:39 AM 3/6/2007, you wrote:
>Date: Tue, 6 Mar 2007 07:54:20 -0500
>From: "Hoffman, Allen" < = EMAIL ADDRESS REMOVED = >
>Subject: Re: [teitac-websoftware] Cognitive - compatibility
>To: "TEITAC Web/Software Subcommittee"
> < = EMAIL ADDRESS REMOVED = >
>Message-ID:
>
< = EMAIL ADDRESS REMOVED = >
>Content-Type: text/plain; charset="us-ascii"
>
>I think this could be combined with the authoring and user-agent items.
>It might make the items too long if not worded very well.
>
>I'd also be interested in the following general analysis:
>Is there AT now that can allow the user to highlight a word in anything

>on screen and offer spelling and/or dictionary function?
>Ditto for reading out loud--but I'm sure this part is available.
>
>Second, what are the underlying obstacles to such software beyond such
>things as scanned images of text?
>
>In other words if a software application meets the other existing
>software requirements now, will AT of this type generally work? If so,

>is this requirement redundant,, or do we want to require this be a
>built-in feature in preference to add-on?
>
>
>
>
>Allen Hoffman -- 202-447-0303

From: Jonathan Avila
Date: Thu, Mar 08 2007 6:50 AM
Subject: Re: Cognitive - compatibility

<quote>
Please don't base your assumptions of what is and isn't possible on what
MSAA does and does not provide.
</quote>

Peter, I agree that MSAA is lacking and I am very familiar with JAVA
Accessibility programming. However, I don't think we can deny the current
state of most Windows applications. 90 out of 100 software applications
I've seen at the Federal level are not using a robust API like this. What
percent of Federal agencies are using UNIX/GNOME desktop software for end
users? My experience with current I&T at the Federal level would lead me t
o guess a very very small percentage. Requiring API access to all static
text sounds good but I have concerns about push back from the industry on
this.

For example, under the current 508 guidelines for forms in software
applications it was deemed accessible by the access board if textual labels
where in close proximity to form controls and the text was written to the
screen using OS functions. This was seen as acceptable at the time because
of the lack of accessibility APIs with development at that time.

Jonathan

-----Original Message-----
From: = EMAIL ADDRESS REMOVED = [mailto: = EMAIL ADDRESS REMOVED = ]
Sent: Wednesday, March 07, 2007 8:58 AM
To: = EMAIL ADDRESS REMOVED = ; TEITAC Web/Software Subcommittee
Subject: Re: [teitac-websoftware] Cognitive - compatibility

Hi Jonathan,

Hmmm... I think we are close to saying the same thing here. I come at
this from the view of technology implementation. If we decide that it
is desirable to support the use case of a user (potentially with a
cognitive disability) getting a definition of every non-graphic piece of
text on the screen, then we must either:

a. build a dictionary into every application that renders text, with
some guidelines for how to invoke the dictionary lookup for every piece
of text it renders (dialog boxes and menu items as well as content), or
b. define one or more "rendered text retrieval" APIs which all
applications support for every piece of text they render (which is then
used by a 3rd party dictionary/definition lookup program which defines
its own mechanism for invoking the dictionary lookup), or
c. hope that one or more independent software developers sees enough of
a market in developing such software to duplicate the
hacks/reverse-engineering/off-screen-model work of our existing Windows
screen readers in order to create such software, or
d. some combination of the above

Please don't base your assumptions of what is and isn't possible on what
MSAA does and does not provide. We have done a pretty remarkable job of
providing exactly this support in the UNIX/GNOME desktop. With the
collection of applications that Sun ships with Solaris 10 and later, and
that Ubuntu ships with Ubuntu 6.10 and later (to name two OSes I'm
fairly familiar with), exactly this information is provided via a
supported API, and compliance with/implementation of this API is quite
good. No off-screen model would be needed to implement such a feature,
greatly lowering the cost to a 3rd party (e.g. AT) developer to
developing such software.

I say this with confidence because the flat-review feature of our Orca
screen reader is able to review word by word (and letter by letter)
every piece of non-graphic text in the applications we ship in Solaris.
For example, starting with the top-left piece of text in the StarOffice
Writer application, Orca will speak and highlight every word of text,
from "File" in the menu bar, through all of the text content, and ending
with the static text in the statusbar at the bottom of the page. In
fact, it will also speak and highlight the icons in the toolbars (as
well as the words of the font style and font name in the combo-boxes in
the toolbars). No off-screen model is ever used to do this. No
hacks/reverse-engineering is ever used to do this.


Regards,

Peter Korn
Accessibility Architect,
Sun Microsystems, Inc.

> <quote>
> To do this today, you would need to either have an off-screen model (and
> all of the video hooks and reverse engineering implied by such) like
> those in our screen readers, or be running on a system which provides
> </quote>
>
> I also have concerns about how this would work for non-content materials.
> For example, are you asking for the ability to select static text in a
> dialog box with the keyboard in order to get a definition of the word(s).
> Providing keyboard access to such an item is likely to be confusing,
> currently difficult to implement, and may cause issues for keyboard only
> users. Providing a mouse based hit test when the mouse is placed over the
> word such as accessibleObjectFromPoint would work in cases were MSAA or
> another API was available. However, even in new Windows form libraries
> static text isn't always written as separate objects that can be accessed
by
> point. Often static text is not it's own object. If the text is exposed
> through OS settings then technology such as this should be 3rd party
> software like a screen reader that would build an off-screen model of the
> text and then provide this type of access.
>
> Jonathan
>
> -----Original Message-----
> From: = EMAIL ADDRESS REMOVED =
> [mailto: = EMAIL ADDRESS REMOVED = ] On Behalf Of Peter
Korn
> Sent: Tuesday, March 06, 2007 7:51 PM
> To: TEITAC Web/Software Subcommittee
> Subject: Re: [teitac-websoftware] Cognitive - compatibility
>
> Hi Allen,
>
>
>> Is there AT now that can allow the user to highlight a word in anything
>> on screen and offer spelling and/or dictionary function?
>> Ditto for reading out loud--but I'm sure this part is available.
>>
>>
>
> Software like this used to exist for the Macintosh, a long time ago. It
> was really quite slick.
>
> To do this today, you would need to either have an off-screen model (and
> all of the video hooks and reverse engineering implied by such) like
> those in our screen readers, or be running on a system which provides
> all of the text that is on the screen via a standard API. We're
> starting to get there on the Macintosh. We're already there in GNOME
> UNIX systems. But in neither place do we have such a utility.
>
> Hmmm.... That'd be a good Google Summer of Code project for a summer
> intern...
>
>
>> Second, what are the underlying obstacles to such software beyond such
>> things as scanned images of text?
>>
>>
>
> Having a well-respected API for getting all of the text & text
> boundaries from the apps running on the system.
>
>
>> In other words if a software application meets the other existing
>> software requirements now, will AT of this type generally work? If so,
>> is this requirement redundant,, or do we want to require this be a
>> built-in feature in preference to add-on?
>>
>>
>
> I personally think this is the kind of feature that works best as a
> separate utility - one user interface that works the same everywhere on
> the desktop.
>
>
> Regards,
>
> Peter Korn
> Accessibility Architect,
> Sun Microsystems, Inc.
>

From: Peter Korn
Date: Thu, Mar 08 2007 2:55 PM
Subject: Re: Cognitive - compatibility

Hi Jonathan,

> <quote>
> Please don't base your assumptions of what is and isn't possible on what
> MSAA does and does not provide.
> </quote>
>
> Peter, I agree that MSAA is lacking and I am very familiar with JAVA
> Accessibility programming. However, I don't think we can deny the current
> state of most Windows applications. 90 out of 100 software applications
> I've seen at the Federal level are not using a robust API like this. What
> percent of Federal agencies are using UNIX/GNOME desktop software for end
> users? My experience with current I&T at the Federal level would lead me t
> o guess a very very small percentage. Requiring API access to all static
> text sounds good but I have concerns about push back from the industry on
> this.
>

You raise valid concerns. Personally, I believe the way to deal with
them is not to limit what we ask for, but rather increase the amount of
time that we give industry to provide it. If we want to address
cognitive impairments in any significant way, and we believe we
understand how to do so technically (e.g. with a robust API, but perhaps
via some other mechanism), then I suggest we push forward with that,
allowing sufficient time to realize the results. We will get there much
faster than if we say "it is too hard to do now, so say nothing about
it", and then have this conversation again 10 years hence.

> For example, under the current 508 guidelines for forms in software
> applications it was deemed accessible by the access board if textual labels
> where in close proximity to form controls and the text was written to the
> screen using OS functions. This was seen as acceptable at the time because
> of the lack of accessibility APIs with development at that time.
>

This example to me is a good argument for the "sufficient techniques"
approach. State what is desired (e.g. that form labels be clearly
associated with what they are labeling), and then enumerate techniques
for achieving that. Such an enumeration can be in a prioritized order,
or noted with caveats (e.g. "on a platform where an API exists to
associate labels with editable text fields, use that; on a platform
where there isn't such an API, use close proximity").

When we have a shared, strong belief in the best technical solutions, we
should spell them out and encourage their use; not water down what we
ask for because those solutions don't exist on some platforms today. So
long as what we are asking for is readily achievable within a small
number of years and also represents a shared understanding of the best
technique to use, that is what we should state. If it fails those
tests, then it is another matter altogether...


Regards,

Peter Korn
Accessibility Architect,
Sun Microsystems, Inc.

> Jonathan
>
> -----Original Message-----
> From: = EMAIL ADDRESS REMOVED = [mailto: = EMAIL ADDRESS REMOVED = ]
> Sent: Wednesday, March 07, 2007 8:58 AM
> To: = EMAIL ADDRESS REMOVED = ; TEITAC Web/Software Subcommittee
> Subject: Re: [teitac-websoftware] Cognitive - compatibility
>
> Hi Jonathan,
>
> Hmmm... I think we are close to saying the same thing here. I come at
> this from the view of technology implementation. If we decide that it
> is desirable to support the use case of a user (potentially with a
> cognitive disability) getting a definition of every non-graphic piece of
> text on the screen, then we must either:
>
> a. build a dictionary into every application that renders text, with
> some guidelines for how to invoke the dictionary lookup for every piece
> of text it renders (dialog boxes and menu items as well as content), or
> b. define one or more "rendered text retrieval" APIs which all
> applications support for every piece of text they render (which is then
> used by a 3rd party dictionary/definition lookup program which defines
> its own mechanism for invoking the dictionary lookup), or
> c. hope that one or more independent software developers sees enough of
> a market in developing such software to duplicate the
> hacks/reverse-engineering/off-screen-model work of our existing Windows
> screen readers in order to create such software, or
> d. some combination of the above
>
> Please don't base your assumptions of what is and isn't possible on what
> MSAA does and does not provide. We have done a pretty remarkable job of
> providing exactly this support in the UNIX/GNOME desktop. With the
> collection of applications that Sun ships with Solaris 10 and later, and
> that Ubuntu ships with Ubuntu 6.10 and later (to name two OSes I'm
> fairly familiar with), exactly this information is provided via a
> supported API, and compliance with/implementation of this API is quite
> good. No off-screen model would be needed to implement such a feature,
> greatly lowering the cost to a 3rd party (e.g. AT) developer to
> developing such software.
>
> I say this with confidence because the flat-review feature of our Orca
> screen reader is able to review word by word (and letter by letter)
> every piece of non-graphic text in the applications we ship in Solaris.
> For example, starting with the top-left piece of text in the StarOffice
> Writer application, Orca will speak and highlight every word of text,
> from "File" in the menu bar, through all of the text content, and ending
> with the static text in the statusbar at the bottom of the page. In
> fact, it will also speak and highlight the icons in the toolbars (as
> well as the words of the font style and font name in the combo-boxes in
> the toolbars). No off-screen model is ever used to do this. No
> hacks/reverse-engineering is ever used to do this.
>
>
> Regards,
>
> Peter Korn
> Accessibility Architect,
> Sun Microsystems, Inc.
>
>
>> <quote>
>> To do this today, you would need to either have an off-screen model (and
>> all of the video hooks and reverse engineering implied by such) like
>> those in our screen readers, or be running on a system which provides
>> </quote>
>>
>> I also have concerns about how this would work for non-content materials.
>> For example, are you asking for the ability to select static text in a
>> dialog box with the keyboard in order to get a definition of the word(s).
>> Providing keyboard access to such an item is likely to be confusing,
>> currently difficult to implement, and may cause issues for keyboard only
>> users. Providing a mouse based hit test when the mouse is placed over the
>> word such as accessibleObjectFromPoint would work in cases were MSAA or
>> another API was available. However, even in new Windows form libraries
>> static text isn't always written as separate objects that can be accessed
>>
> by
>
>> point. Often static text is not it's own object. If the text is exposed
>> through OS settings then technology such as this should be 3rd party
>> software like a screen reader that would build an off-screen model of the
>> text and then provide this type of access.
>>
>> Jonathan
>>
>> -----Original Message-----
>> From: = EMAIL ADDRESS REMOVED =
>> [mailto: = EMAIL ADDRESS REMOVED = ] On Behalf Of Peter
>>
> Korn
>
>> Sent: Tuesday, March 06, 2007 7:51 PM
>> To: TEITAC Web/Software Subcommittee
>> Subject: Re: [teitac-websoftware] Cognitive - compatibility
>>
>> Hi Allen,
>>
>>
>>
>>> Is there AT now that can allow the user to highlight a word in anything
>>> on screen and offer spelling and/or dictionary function?
>>> Ditto for reading out loud--but I'm sure this part is available.
>>>
>>>
>>>
>> Software like this used to exist for the Macintosh, a long time ago. It
>> was really quite slick.
>>
>> To do this today, you would need to either have an off-screen model (and
>> all of the video hooks and reverse engineering implied by such) like
>> those in our screen readers, or be running on a system which provides
>> all of the text that is on the screen via a standard API. We're
>> starting to get there on the Macintosh. We're already there in GNOME
>> UNIX systems. But in neither place do we have such a utility.
>>
>> Hmmm.... That'd be a good Google Summer of Code project for a summer
>> intern...
>>
>>
>>
>>> Second, what are the underlying obstacles to such software beyond such
>>> things as scanned images of text?
>>>
>>>
>>>
>> Having a well-respected API for getting all of the text & text
>> boundaries from the apps running on the system.
>>
>>
>>
>>> In other words if a software application meets the other existing
>>> software requirements now, will AT of this type generally work? If so,
>>> is this requirement redundant,, or do we want to require this be a
>>> built-in feature in preference to add-on?
>>>
>>>
>>>
>> I personally think this is the kind of feature that works best as a
>> separate utility - one user interface that works the same everywhere on
>> the desktop.
>>
>>
>> Regards,
>>
>> Peter Korn
>> Accessibility Architect,
>> Sun Microsystems, Inc.
>>