GDIHooking library for NVDA

[aleksey_s]

i am implementing a library, with which nvda can to have access to the text, which is written by GDI functions directly. this is the delphi sources, as well as precompiled binaries. GDIHook.dll is an in-process dll which making hooks to api, GDIHookHandler.dll is manager for it. GDIHookTest.exe is an test application which works with GDIHookHandler exported functions.
note, that all stuff can have bugs and might crash your system!

[pvagner]

To test the library the following code is implemented in the patch attached to this ticked
1) ability to read owner-drawn menuItems (such as those found in the open with submenu in the windows explorer context menu)
2) ability to read owner-drawn ListBoxItems? (tested with miranda-im chatrooms)
3) test script bound to NVDA+f4 which can be used to speak all the text drawn to the client rect of the current navigator object.
4) some more class mappings in the iaccessible which I am using; so I did not want to split my testing copy to two branches.

Some notes
This is really a smal preview for testing purposes or for those wishing to help in the development. We do need to come up with better logic when and where to hook into. Current implementation might crash any application. Most likely it will crash applications which user interface is runing in various threads. Also memory usage shal be monitored while testing because of pending bugs.

[aleksey_s]

i decided to post here my changelog, to any who want to see how development goes on.
1 alex 2008-03-21

basic commit

2 alex 2008-03-21

*injecting with using windows hooks, not CreateRemoteThread?
*AttachToProces? -> AttachToWindow?
+DetachFromWindow?
+synhronization

3 alex 2008-03-22

*improved synchronization
*Now log file will create in user temporary dir

4 alex 2008-03-22

*code was rewritten to store and use text rectangles, not only coords. so now when new text appears with coords on one pixel different from other (it is really possible, lol) it will be rewritten instead of saving it. also it means GetTextAtCoords? will return no text at that really coords but text with which rectangle given coords intersects.
-removed hooking of functions which are really wrappers of other (such as DrawText?, DrawTextEx?)
*implemented bit handling of multiline text, it is trivial but it works and now we have not (i hope) repeated text
*possibly bugs fixed, added a lot of new

[aleksey_s]

5 alex 2008-03-23

*Started removing abuses of using shared memory, now Attaching and unattaching read info from remote process directly.

6 alex 2008-03-24

+implemented class, which incarnate dynamic arrays in shared memory. so now memory usage is more better.
+Added PTInRect function, becouse that, which windows api provides works not such i thought :-)
*fixes to ClearStorageInRect?
*other improvements

Now there precompiled debug versions of both libs, named GDIHook_debug.dll and GDIHookHandler_debug.dll. you can put their in nvda folder and delete suffix "_debug", then libraries will provide a log with all happened stuf and you can see what is incorrect. log can be found at your_temp_dir/gdihook.log

[aleksey_s]

7 alex 2008-03-25

*fixed crash when detaching

8 alex 2008-03-25

*a lot of try..except added, so now error handling will be more smart. when not debug compilation, may be error messages can be not so informative, but with debug compilation i hope it will be easier to find where an error occurs.
*again fix to synchronization code, i hope last :-)
*more new debug messages to SharedDynamicArrays?.pas and Storage.pas

9 alex 2008-03-25

*a litle fix to prevent crashes when twice attaching to one application

[aleksey_s]

10 alex 2008-03-26

+Now library will know exactly where hooked multiline text drawn on the screen. I believe, it is big step forward.
Problem was becouse when program want draw multiline text, it cann pass it whole to one of drawing functions with start coordinates, and rectangle where text will be clipped. so program can call drawing function many times with passing each time one line less, and drawing function will calculate what text will be really drawn by given rectangle.
+OpenStorage? function which try to open existing Shared Array instead of creating new
*GDIHookHandler exported functions will try to OpenStorage? instead of doing it in DLL_PROCESS_ATTACH, so when GDIHookHandler will hook to process in which need to inject GDIHook it will no greater try to init storage (possibly crash fixed)

currently there are following problems in the library:
*i do not know how to handle (determine real coordinates from logical etc) text from windows with handle 0 (especially dialogs)
*i do not know how to handle menu closing, dialog closing, window closing events to clear their rectangles, but GDIHookHandler exports function ClearStorageInRect? which might be called from NVDA to do these things

[pvagner]

wrapper for GDIHook.dll. In this version all the code which tryes to avoid duplicates is removed as it's perfectly done within the library it-self. anyway my attempt was not correct.

[aleksey_s]

11 Ђ¤¬Ё­Ёбва в®а <Ђ¤¬Ё­Ёбва в®а@NOTEBOOK> 2008-04-02

*Completely rewritten hooking mechanizm, now it must be stable in multithreaded applications. Thanks to my friend Sergey Starovoy for routines to calculate opcode size and other tips.

[aleksey_s]

r12

[aleksey_s]

12 Ђ¤¬Ё­Ёбва в®а <Ђ¤¬Ё­Ёбва в®а@NOTEBOOK> 2008-04-03

*fixed some small bugs in InterceptAPI.pas, HookedFunctions?.pas

[jteh]

I've updated Peter's patch for current trunk and committed it to a bzr branch for further development. The branch is at:
http://bzr.nvaccess.org/nvda/gdi/

[jteh]

Aleksey, any further progress on this?

• Have there been any updates to the dlls since they were last posted here?
• What are the current limitations/problems of which you are aware?
• You noted that you needed some assistance at some point in order to make further progress. Can you elaborate about the information you need or at least the problems you are unable to solve?

Are you using bzr for version control? If so, we should get you an account on our hosting server at some point so you can publicly post the bzr branch.

I am moving this out of 0.6. This does not mean that it will not go into 0.6, but it means that it will not block 0.6 if it's not ready when 0.6 is to be released.

[jteh]

• Ideally, we want to attach in gainFocus of Window NVDAObjects and detach in loseFocus. This allows any higher level object to use the GDI support as needed.
• While testing, I experienced crashes in Notepad++ if attaching for all Window NVDAObjects. Any ideas on why this might be or can anyone else reproduce this?
• It seems WM_PAINT is not really the correct way to redraw the window. This seems to work quite nicely and can be done in one call:

  user32.RedrawWindow(windowHandle, None, None, RDW_INVALIDATE)
  

  where RDW_INVALIDATE = 1.

[aleksey_s]

Replying to jteh:

Aleksey, any further progress on this?
* Have there been any updates to the dlls since they were last posted here?

yes, i have worked around some major mechanizms of intercepting and attaching/unattaching, as well as storing data. i decided to store each hooked data in the in-process structures, and write it in shared memory when needed. so now each gdihook library when injecting creates some hided window to receive commands, and uses shared memory as some type of stack to receiving and sending data to handler. however, all problems i noted earlier still exist (e.g. with clearing unnecessary data, controls in dialogs etc).

* What are the current limitations/problems of which you are aware?
* You noted that you needed some assistance at some point in order to make further progress. Can you elaborate about the information you need or at least the problems you are unable to solve?

look at my earlier comments.

Are you using bzr for version control? If so, we should get you an account on our hosting server at some point so you can publicly post the bzr branch.

yes, i am.

I am moving this out of 0.6. This does not mean that it will not go into 0.6, but it means that it will not block 0.6 if it's not ready when 0.6 is to be released.

i am completely agree.

[jteh]

(In #151) This issue occurs because some menu items which display an icon don't seem to expose their name to MSAA. As far as we are aware, the only way to work around this is to obtain the name using display information. (We believe this is how other commercial screen readers solve this problem.) This requires display hooks. Although we have a very early implementation of display hooks, it is not yet ready for inclusion in NVDA. See #40 for details.

Note that the "Open With" menu in Vista does not seem to suffer from this issue.

[Bernd]

Aleksey, any further progress on this?
Have you done more work on this ticket

since your last post here? I'm asking because I'm interrested on your work.

[aleksey_s]

Replying to Bernd:
no. i am waiting for implementation of unified way for nvda to receive text info from another process, as i know Mick is working on this.

[aleksey_s]

i am thinking about continuing my work. for this i have to consider few necessary moments:

• is it possible to continue development using delphi (or other pascal compiler)? i know object pascal best of all, but i want to progress and practice in C++. for now i haven't nothing completed in this language, just learning programs.
• form in which gdi hooks will be distributed and api must be considered. in form of distribution i mean will it be a separate files or may be included to nvdaHelper? it depends on how do you see the last. if it will be library, which contains all low level (read - c++) code for NVDA, then including gdi hooks in it is quite reasonable. as i know it allready loads in all processes when nvda starts. then it must be rewritten in c++, with some restructurization / more abstraction. Also, we must to decide carefully which api gdi hooks will provide. i really feel that those current exist are not way to go. i think that gdi hooks and winevent processing depends on each other. you can see it with menu stuff. i consider that responding on events twice (inside nvda code and inside nvdaHelper) is not good idea, however for now it is so. it must be decided at some point how to proceed with it.
• what injection mechanizm to use? win hooks is nice for system wide hooking. but it has its disadvantages. i found some of that on http://www.codeproject.com/KB/system/hooksys.aspx
  • Windows Hooks can degrade significantly the entire performance of the system, because they increase the amount of processing the system must perform

for each message.

• It requires lot of efforts to debug system-wide Windows Hooks. However if you use more than one instance of VC++ running in the same time, it would simplify the debugging process for more complex scenarios.
• Last but not least, this kind of hooks affect the processing of the whole system and under certain circumstances (say a bug) you must reboot your machine in order to recover it.

injection also possible through CreateRemoteThread, but in this case system-wide injection is difficult to implement: we cannot receive notifications about new processes creation without nt kernel driver. so we have to decide whether or not we want gdi hooks injected in all processes or just specific ones. i myself prefer first way.

• what interception mechanizm to use? I do not like hacking import table of each module for changing pointer to real function to the own fake. advantage of this method is that it is multithread-safe, but bigger disadvantage is that when you receive pointer to function with dynamic linkage (by calling GetProcAddress), you get pointer to the real function. however, i am not sure in it. Richter recommends this method in his "windows internals".

second method is to replace a few first bytes of function with jmp instruction to our code, which will execute needed manipulations and after put real code back, call true function and after again put jmp instruction to the beginning of true function. big disadvantage of this method is that it isn't multithread-safe: consider situation when one thread calls api function, jmp points processor to the our code, which make special processing of parameters and after replaces beginning of api function with it real code (to call true function). and in this moment, other thread calls api function too. what we will have? thread two is calling not hooked function. worst, it might call function when instructions are not completely replaced - with unpredictable result...
third method is most complicated. However, i will try to describe it here. note also, that in current implementation it is allready partially used. for installing hooks, injected code freezes al running threads (for garantee that no one will cal uncompletely hooked function). then for each function, it copies a few of it first instructions (how many needed to match 5+ bytes size). most difficult is to calculate length of instruction. it works like disassembler. then it must replace all relative jumps/calls to nonrelative (absolute). so, it creates buffer in memory, where it writes these (possible changed) instructions, and on the end of the buffer, it puts jmp instruction to address after these cutted instructions in hooked function. instead of cutted instructions in real function, it writes jmp to the fake function which will process parameters etc. after doing it work, fake function will jmp to the known buffer, in which real instructions are copied and jmp on they continuing. this method is multithread-safe and has a good performance, you cannot somehow jump through fake function (as you can with first method i described), but it has its difficulties in implementation: for now i do not change relative addresses in cutted instructions to nonrelative, so if there are allready one api hook installed, it will be broken.

for me, even major is your opinions on the first question (about architecture), if you are unfamiliar with all this lowlevel technical details i will try to investigate they myself.