Lazarus Development

While working on Lazarus 0.9.23, which took more than 6 months, Giuliano Colla proposed to create a stable branch. Lazarus development goes very rapid with about 100 commits each week and with each daily update, there is a reasonable chance that something got broken. This is not desirable for people that need to do production work with Lazarus.

The idea was welcomed, although I had some doubts about the feasibility, especially the man power needed. Lazarus development goes rapid and there is still a lot to do, so the current Lazarus developers can better spend their time on new development than on maintaining a stable branch. So, we offered Giuliano to maintain it him self and we would support him.

After the Lazarus 0.9.24 release in November, I created the fixes_0_9_24 branch and set the version number to 0.9.24.1. Windows snapshots using fpc 2.2.0 and the fixes branch are created on a daily basis from ftp. Giuliano would send me a list of revision numbers to be merged and I would apply thoses merges to the fixes branch. After a couple of merges, the branch became 'super'-stable, because there were no changes. A lot of changes are related and Giuliano wanted to test them before applying. This takes a lot of time. Also, it is sometimes not trivial to see the dependencies between the different revisions, espescially if a revisions contains new functionality, that you don't want to merge and a simple fix that is needed when you want to merge a later bug fix revision.

So we are still struggling to find a good strategy for maintaining the fixes branch. In the last week, I took a different approach. I looked at the list of revisions and merged the new component bar images, splash screen and most the bug fixes I made in Lazarus 0.9.25. Almost 200 revisions were merged from trunk to the fixes branch, so things might have been broken, but I have good hopes that 0.9.24.1 is at least as good as 0.9.24.

There are still about 500 revisions in trunk not (yet) merged:

• Documentation hints
• Handle rewrite
• New project options and environment options dialog
  
• A number of bug fixes for other widget sets than win32 (my only expertise).
• Improvements to images in the toolbar of the IDE
• Several improvements to the Lazarus IDE dialogs
  

Because it is hard to guess what people want in the fixes branch (except stability), we want to ask users of the fixes branch to indicate what they want to be merged from trunk. If we think the feature is stable already, we will try to merge it.

I encourage everybody to test the fixes branch and give feedback on the Lazarus mailing lists. If the fixes branch proves succesfull (stable updates without occassional breakage), I am considering to put snapshots of it in the Lazarus-Testing Ubuntu repository.

This afternoon I tagged Lazarus 0.9.24. The coming days we will build all the windows installers, rpms, debs and dmgs for the suported OS-es and CPU's.

As part of the release testing, I created an ubuntu lazarus-testing repository at http://www.hu.freepascal.org/lazarus/dists/lazarus-testing/. Currently it still contains Lazarus 0.9.22 and FPC 2.0.4, but as soon as the Lazarus 0.9.24 and debs are ready they will be put there, until the Lazarus 0.9.24 release can be announced. After the announcement, they will be available in the lazarus-stable repository too. For more information about how to get debs from the Lazarus testing repostitory, see the wiki.

Please, let us know if you have problems using the testing repository and during the upgrade from 0.9.22 to 0.9.24.

Lately I have been trying to fix recompiling the Lazarus (needed to install components) from the IDE. On windows 2000 and higher, using .rc files is working OK if windres is not on the path, but on windows 98 there are still some problems.

Marc has done a large graphic rewrite in Lazarus 0.9.23 and he also noticed that there are noticeable difference between Windows 98 and Windows XP.

I wondered if there still many people using Windows 98 and Windows ME for Lazarus development or as target for LCL application. Therefore I created a new poll on the Lazarus website, asking people about the oldest Windows version is, that they use for Lazarus development.

Currently when we program on the win32 interface we try to use only functions available in windows 95, or provide a fall back if we want to use a function not available on windows 95. Some features like graphics, Unicode support, the shell provided by cmd32.exe and the availability of console (used for the debugger) are better and easier in Windows 2000 and later. So it would be nice if we could restrict ourselves to these newer Windows versions.

Personally, I don't think we can drop support for windows 98 yet, but I doubt anybody is using Windows 95 to program with Lazarus. Let us know where you stand and cast your votes in the poll. This poll is about the OS you use for Lazarus, a future poll may be held about the Windows versions you want your developed applications to run on.

Note: people who don't use Lazarus on Windows, don't need to vote.

I think most users who used TPageControl in windows ide suffered from the inability to stitch its Tabs by clicking them in the designer. Indeed, the gtk ide had no such bug while the windows one had. I got annoyed by the fact too and as result I started my research of the problem.

My first assumption was that somewhere a csDesigning (you know that usual "if csDesigning in BlaBla.ComponentState" ;) ) condition was written and I spent much time (searching code) to prove myself that I am wrong.

Ok, if I am wrong (I thought) then it is by design or by the designer :) So I started my research of how the win32 designer works. Usually my research consists of Alt+F7 in Far manager for some search key (in this case for 'designer') and if Alt+F7 doesn't help, then in debugging the code. I suppose on that occasion I had been satisfied by search in files. First of all I found 'TWin32WidgetSet.GetDesignerDC' method and later I've checked that this method is indeed what I searched for. Look at TDesigner.DrawDesignerItems and you'll find that yourself.

So, I've known the enemy by sight :) If you notice GetDesignerDC creates a visually transparent window over WindowHandle and returns the Device Context for that overlay window. The Device Context is not a interesting thing for us, but the overlay window is. That overlay window is placed on top of a form (and all child controls). And the most interesting thing in that overlay window is the window procedure OverlayWindowProc.

Inside OverlayWindowProc we can find that overlay window easts all messages except WM_KEYFIRST..WM_KEYLAST, WM_MOUSEFIRST..WM_MOUSELAST and that bunch of messages it redirects to the parent (look at Windows.GetParent(Window)). If you look at TDesigner.DrawDesignerItems then you'll know that the Parent of OverlayWindow is the Form. So nothing wondering now that the PageControl Tabs cannot be switched in designer - PageControl gets no clicks (they are caught by our overlay window).

Hmm ... I thought about a solution. The first thing that flashed through my mind was sending those mouse clicks to the underlying control. But it is so easy to forget about some messages or make other errors if you want to emulate windows behaviour. So I rejected that way as defective. Another way is to make some parts of our overlay window transparent for mouse. Windows has a special message WM_NCHITTEST to check what a given coordinate means. And a window can return HTTRANSPARENT as result of a message to indicate that message must be sent to the underlying window (so this point is transparent for the window).

As result the whole solution consisted of handling WM_NCHITTEST in OverlayWindowProc and return HTTRANSPARENT if underlying control wanted mouse messages for that point at design time. I've added a new method GetDesignInteractive to TWSWinControl.

class function GetDesignInteractive(const AWinControl: TWinControl; AClientPos: TPoint): Boolean;

Now I had a method to check, but of course I needed the implementation of that method for TPageControl (which checked that mouse is over control Tabs). So I've added TWin32WSCustomNotebook.GetDesignInteractive, rebuilt the win32 ide and hurray - I switched PageControl tabs in designer w/o problems.

You can look at implementation in revisions: 12245, 12620 (method has been renamed).

One or two weeks ago our qt-ide was almost fine (if form designer was not taken into account) except one important detail - the speed of the unit editor. Yes, you were able to edit source, but the speed .... it was deadly slow. You could type a key and wait second or two before that key was printed in editor. So we started the fight for speed.

The core of the unit editor is the Synedit component. Lazarus uses an early version of Synedit but it is heavily changed for our needs (adopted, extended). Synedit is a TCustomControl, so it does all its painting and event handling itself. When I looked at the source for the first time, I found nothing criminal. A well done component which repaints only changed areas.

I tried to find out what piece of code makes the component so slow under qt. And I found that it is the qt QPainter.DrawText method. My first assumption was that in the OnPaint event we were repainting the whole Synedit, not only the changed area.

I checked how invalidateRect was done and there was no error - QWidget.Update(Rect) was called. My next assumption was that we were ignoring that Rect in the OnPaint event, but no - we did clip the paint area by that rect. What was it - a mystery?

I checked how clipping worked and found, that in DrawText the clipping was gone. The reason was in Synedit - it used double buffered painting and as result DrawText performed on a non clipped bitmap instead of a clipped window. Revision 12175 changed things. There was no more bitmap, but painting was still as slow as before.

My next assumption was that qt doesn't check the clipping region before DrawText. And bingo - manual check of clipping rect before DrawText gave us no bad speed (revision 12183). But cpu usage was very big in spite of the achieved speed. So something was still wrong, but what?

Zeljan noticed that caret eats too much cpu time. It was because of a not optimal use of timer and widget repainting. After fixing those things in revisions 12198-12200 there was not so high cpu rate as before.

I was already happy but Zeljan was at his best and found one more speed killer. That was our ExtTextOut implementation. ExtTextOut have 2 arguments related to text: Str - the text and Count - count of chars. We made the assumption that Pascal components pass Str arguments that are already trimmed to count chars, but this was not true (at least for Synedit). There were cases where count was equal to 1, but Length of Str equal to 300 (or 400 chars). After an appropriate patch (revisions 12201,12205)the speed of Synedit has become really fast.

Now qt-ide as fast as it should be.

One of the best held secrets in the Lazarus community is the release date of Lazarus 0.9.24. Sometimes people ask when the next Lazarus release will be. The only correct answer is: "When it is finished". And even the Lazarus developers don't know when that is.

Fortunately, we have a criterium for finished. We want our next release as good as the previous one, so things that work in 0.9.22, must work in 0.9.24 too. If soemthing stops working and it is found out by people using the snapshots, we ask create a bug report for it in the Lazarus bug tracker. Such regressions are marked with "LazTarget 0.9.24". On the view issues page, you can filter on those issues. The list of 0.9.24 issues is currently 23 long, most them are already assigned to some developer. The list also contains patches contributed by Lazarus users, we want to include in the next release. Currently the list is too long to give a reliable estimate of the next Lazarus release date.

In the final stages before a release and during the release preparation we use the Detailed todo page on the wiki as a kind of check list.

You might wonder if nothing is happening on the Lazarus front. The truth is, that very much has been done since 0.9.22. Almost 200 bugs have been fixed (see Changelog in the bug tracker) in Lazarus 0.9.23 and more than 1300 svn commits have been made. You can track the svn commits to the Lazarus svn repository though rss or at the Lazarus page at CIA.vc.

In the last couple of days I have made changes to the build servers and the mirrors. All snapshots that were using fpc 2.0.4 are now using fpc 2.2.0.

The next Lazarus release (version 0.9.24) will be based on fpc 2.2.0 too, so these snapshots give a good preview of the next release. At his moment there are still about 20 bugs open that need to be fixed before the release, so there is still plenty time for testing Lazarus 0.9.23. Download it from one of the snapshot mirrors:

• Hungary
• Germany
  
• USA
  

The chart component in Lazarus was written two years ago by Philippe Martinole for his TeleAuto project. He submitted the component to the Lazarus Code and Component Repository. Later Luis Rodrigues extended it and made it more Delphi compatible. The component was considered stable and useful enough to add it to the default installation of Lazarus. Luis Rodrigues now is the main maintainer of the component.

A couple of weeks ago Luis submitted a patch for the chart component. Because I don't know the inner workings of the chart components and don't have a test program, I usually do two things:

• Do a visual inspection of the patch to see if it contains sensible things.
  
• Check if the patched version compiles on under windows, my main development OS.

People submitting these patch are major users of the component, so I assume they are better testers than I possibly can be.

This particular patch contained the following code to fix clipping:

 //set cliping region so we don't draw outsite
- Rgn := CreateRectRgn(XImageMin, YImageMax, XImageMax, YImageMin);
+ p[0].x := XImageMin;
+ p[0].y := YImageMax;
+ p[1].x := XImageMax;
+ p[1].y := YImageMin;
+
+ {$IFDEF windows}
+ LPtoDP(Canvas.Handle, p, 2);
+ {$ENDIF}
+ Rgn := CreateRectRgn(p[0].x, p[0].y, p[1].x, p[1].y);
 SelectClipRgn (Canvas.Handle, Rgn);

I don't know how clipping works and what LPtoDP exactly does, but having a {$IFDEF} here in the middle of a procedure, doesn't look like good portable code. There seemed to be some inconstencies between the CreateRectRgn functions for win32 widgetset and the gtk widgetset: one expects logical points and the other device points.

Even if we wanted to use a {$IFDEF} the current one is probably not correct. The difference is not the OS, but the used widget set, so using {$IFDEF LCLwin32} would have been better. Remember, that the qt and the gtk2 widget can be compiled on windows too.

I asked Luis to find a better solution and maybe some people on the mailing list could help (unfortunately I had no clue how to fix this properly). Luis found another way without IFDEFS:

 //set cliping region so we don't draw outsite
- Rgn := CreateRectRgn(XImageMin, YImageMax, XImageMax, YImageMin);
- SelectClipRgn (Canvas.Handle, Rgn);
+ IntersectClipRect(Canvas.Handle, XImageMin, YImageMax, XImageMax, YImageMin);

This patch I happily applied after I tested the compilation.

From time to time, the Lazarus snapshots at Scenergy disapear. It is probably happens when two build servers upload a new version at the same time. The update process has roughly the following steps:

1. Delete the files currently in the temp location
   
2. Copy current files from the ftp-directory to a temp location.
3. Remove old snapshot
4. Add new snapshot
5. Rsync the temp location to the ftp-directory

The temp location is the same directory for each upload. Now if one build server is at step 4 and a second build server starts at step 1, the rsync job will finish quickly with no files to copy ...

For normal snapshots this is not a big problem, they will be regenerated within 24 hours. But occasionally I upload a wince cross-installer too and that doesn't get updated automatically yet. So until this issue has been fixed (probably by adding some locking and using different temp locations), I have put the latest wince snapshot at sourceforge, in the Lazarus Testing package.

Two weeks ago I setup an ubuntu deb repository for Lazarus releases. I wondered how much they were used and asked the server admin to install webalizer on the server (it's name is Scenergy). In these two weeks, the repository generated 15 GB trafic and the i386 archicture is still favorite.

Scenergy is also one of the Lazarus snapshot servers. So I added the ftp stats too. In August this server served 50 GB of snapshots. The same ftp server has fpc releases too and the fpc 2.0.4 windows installer causes 43% of trafic: 128 GB. Interesting figures ...

Not long (about 2-3 months) ago my attention was drawn to the lazarus Qt widgetset (child of Den - author of the bindings and Felipe - author of the lazarus widgetset). It is easy for me to develop the Qt widgetset, since I can run the Qt enviroment on my main platform - windows xp. Another developer - Zeljan (with qt on linux) worked hard on qt widgetset at that moment. My goal was to achieve a working Qt ide. Zeljan had another goal (as I know). He had a real application and tested the qt widgetset on that.

Step by step and day by day we fixed bugs and implemented missing features (of course, what else can we do?) :) There were problems with mouse, keyboard, painting, widget sizes and positions and so on (you can look at the changelog for more details). As far as I can see my goal is achieved now. Yes, ide is mostly functional: you can edit units, you can use form designer, you can play with lazarus dialogs (really I don't know what you can't do).

But please dont expect too much - the ide still has bugs. Some of them are known and fixing them is in progress (form designer has no rubber band, some controls have wrong size until parent resize, tool windows have no close button), but some are not known. So we need testers.

We'll have lazarus-qt snapshots soon. Stay tuned.

Last night FPC 2.2.0 was tagged. This means that the source code for the release is frozen, no more changes can be made. Now the release builders can start their job and create Windows installers, Linux rpms and debs and Mac OS X dmgs with installer packages. When they have finished their jobs, the FPC team will announce their release, probably in two weeks or so.

I am glad we have reached this point; the last critical bugs for Lazarus were fixed almost a month ago, all the Lazarus snapshots using fpc 2.1.5 were compiling fine and there had not been any reports about failures with fpc 2.1.5. So I was very surprised to get an email this morning from the debian builder, that Lazarus trunk could not be compiled with fpc 2.2.0.

Compiling memcheck.pasmemcheck.pas(211,17)
Error: Identifier not found "pptruint"memcheck.pas(211,17)
Error: Error in type definitionmemcheck.pas(222,33)
Error: Illegal qualifier memcheck.pas(1110,22)

The memcheck unit is a copy of the heaptrc unit from the RTL with some extras. Because there were no problems with fpc 2.1.5, I suspected it contained something like {$IFDEF VER2_1} which should be extended to version 2.2 too. But the source was clean of such version dependent ifdefs. I started to doubt if the snapshot were built with fpc 2.1.5 after all, the only difference between fpc 2.1.5 from yesterday and fpc 2.2.0 is the version number.

Then I got a idea, and looked in the build unit for the codetools allcodetoolunits.pp. It contained the following lines:

uses
 {$IF defined(VER2_2) or defined(VER2_3)}
 MemCheck,
 {$ENDIF}

We changed this to:

uses
 {$IFDEF VER2_3}
 MemCheck,
 {$ENDIF}

and now Lazarus is fpc 2.2.0 ready.

The memcheak unit depends on the heap manager and besides the use of unsigned integers the fpc 2.3 heap manager has some improvements for multithreaded programs, which make it incompatible with the fpc 2.2 heap manager.

Theo suggested to start a blog to give people some insight in the development process of Lazarus. A bit of tips and tricks on improving Lazarus.

So I opened an account at blogger and created this blog. Let's see if I have something to tell.

I am still a bit sceptical about its use, but having a blogspot makes the threshold to write something down easier. The future will tell, if this was a good idea or not.