PATCH: Improve Java build times

Mark Mitchell mark@codesourcery.com
Tue Jun 4 09:49:00 GMT 2002

• Previous message (by thread): PATCH: Improve Java build times
• Next message (by thread): gcc-3.1 build problem in the java area(?)
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

---

> Right, these are JNI failures which cannot have anything to do with
> Mark's patch. Mark, please go ahead and check in the patch!

I've checked in the attached combination of my patches and Bryce's
patches to the 3.1 branch; these together significantly speed up the
Java compiler.
Thanks to all for reviewing, testing, and being patient while we
built up enough confidence to get this checked in.
-- 
Mark Mitchell mark@codesourcery.com
CodeSourcery, LLC http://www.codesourcery.com
2002年06月03日 Mark Mitchell <mark@codesourcery.com>
2002年06月03日 Mark Mitchell <mark@codesourcery.com>
	2002年05月18日 Mark Mitchell <mark@codesourcery.com>
	* java-tree.h (CLASS_BEING_LAIDOUT): Remove duplicate definition.
	* jcf-io.c (dirent.h): Include it.
	(fnmatch.h): Likewise.
	(compare_path): New function.
	(java_or_class_file): Likewise.
	(memoized_dirlist_entry): New type.
	(memoized_dirlist_lookup_eq): New function.
	(memoized_dirlists): New variable.
	(caching_stat): New function.
	(memoized_class_lookup_eq): New function.
	(memoized_class_lookups): Likewise.
	(find_class): Use memoized_class_lookups and caching_stat.
	* jcf.h (JCF_USE_SCANDIR): Define.
	* parse.y (java_expand_classes): Write the class files in reverse
	order.
	2002年05月13日 Mark Mitchell <mark@codesourcery.com>
	* jcf-write.c (write_classfile): Unlink the temporary file if it
	cannot be renamed. Use concat to build up the name of the
	temporary file.
	2002年05月13日 Mark Mitchell <mark@codesourcery.com>
	* jcf-write.c (write_classfile): Unlink the temporary file if it
	cannot be renamed. Use concat to build up the name of the
	temporary file.
2002年06月03日 Mark Mitchell <mark@codesourcery.com>
	
	2002年05月23日 Bryce McKinlay <bryce@waitaki.otago.ac.nz>
	* Makefile.am (all-recursive): Depend on $all_java_class_files so that
	they build first.
	* Makefile.in: Rebuilt.
	2002年05月08日 Mark Mitchell <mark@codesourcery.com>
	* Makefile.am (all_java_source_files): New variable.
	(all_java_class_files): Likewise.
	.java.class: New rule.
	(CLEANFILES): Remove tmp-list.
	* Makefile.in: Regenerated.
Index: gcc/java/java-tree.h
===================================================================
RCS file: /cvs/gcc/gcc/gcc/java/java-tree.h,v
retrieving revision 1.136.2.2
diff -c -p -r1.136.2.2 java-tree.h
*** gcc/java/java-tree.h	24 Apr 2002 22:16:08 -0000	1.136.2.2
--- gcc/java/java-tree.h	4 Jun 2002 16:41:19 -0000
*************** extern tree *type_map;
*** 1424,1434 ****
 layout of a class. */
 #define CLASS_BEING_LAIDOUT(TYPE) TYPE_LANG_FLAG_6 (TYPE)
- /* True if class TYPE is currently being laid out. Helps in detection
- of inheritance cycle occurring as a side effect of performing the
- layout of a class. */
- #define CLASS_BEING_LAIDOUT(TYPE) TYPE_LANG_FLAG_6 (TYPE)
-
 /* True if class TYPE has a field initializer finit$ function */
 #define CLASS_HAS_FINIT_P(TYPE) TYPE_FINIT_STMT_LIST (TYPE)
--- 1424,1429 ----
Index: gcc/java/jcf-io.c
===================================================================
RCS file: /cvs/gcc/gcc/gcc/java/jcf-io.c,v
retrieving revision 1.33
diff -c -p -r1.33 jcf-io.c
*** gcc/java/jcf-io.c	3 Dec 2001 19:13:40 -0000	1.33
--- gcc/java/jcf-io.c	4 Jun 2002 16:41:19 -0000
***************
*** 1,5 ****
 /* Utility routines for finding and reading Java(TM) .class files.
! Copyright (C) 1996, 1997, 1998, 1999, 2000 Free Software Foundation, 
Inc.
 This program is free software; you can redistribute it and/or modify
 it under the terms of the GNU General Public License as published by
--- 1,5 ----
 /* Utility routines for finding and reading Java(TM) .class files.
! Copyright (C) 1996, 1997, 1998, 1999, 2000, 2002 Free Software 
Foundation, Inc.
 This program is free software; you can redistribute it and/or modify
 it under the terms of the GNU General Public License as published by
*************** The Free Software Foundation is independ
*** 29,34 ****
--- 29,39 ----
 #include "tree.h"
 #include "toplev.h"
 #include "java-tree.h"
+ #include "hashtab.h"
+ #if JCF_USE_SCANDIR
+ #include <dirent.h>
+ #include <fnmatch.h>
+ #endif
 #include "zlib.h"
*************** DEFUN(find_classfile, (filename, jcf, de
*** 304,313 ****
 #endif
 }
 /* Returns a freshly malloc'd string with the fully qualified pathname
! of the .class file for the class CLASSNAME. Returns NULL on
! failure. If JCF != NULL, it is suitably initialized.
! SOURCE_OK is true if we should also look for .java file. */
 const char *
 DEFUN(find_class, (classname, classname_length, jcf, source_ok),
--- 309,460 ----
 #endif
 }
+ #if JCF_USE_SCANDIR
+
+ /* A comparison function (as for qsort) that compares KEY (a char *
+ giving the basename of a file) with the name stored in ENTRY (a
+ dirent **). */
+
+ static int
+ DEFUN(compare_path, (key, entry),
+ const void *key AND const void *entry)
+ {
+ return strcmp ((const char *) key,
+ 		 (*((const struct dirent **) entry))->d_name);
+ }
+
+ /* Returns nonzero if ENTRY names a .java or .class file. */
+
+ static int
+ DEFUN(java_or_class_file, (entry),
+ const struct dirent *entry)
+ {
+ const char *base = basename (entry->d_name);
+ return (fnmatch ("*.java", base, 0) == 0 ||
+ 	 fnmatch ("*.class", base, 0) == 0);
+ }
+
+ /* Information about the files present in a particular directory. */
+ typedef struct memoized_dirlist_entry
+ {
+ /* The name of the directory. */
+ const char *dir;
+ /* The number of .java and .class files present, or -1 if we could
+ not, for some reason, obtain the list. */
+ int num_files;
+ /* The .java and .class files in the directory, in alphabetical
+ order. */
+ struct dirent **files;
+ } memoized_dirlist_entry;
+
+ /* Returns true if ENTRY (a memoized_dirlist_entry *) correponds to
+ the directory given by KEY (a char *) giving the directory
+ name. */
+
+ static int
+ DEFUN(memoized_dirlist_lookup_eq, (entry, key),
+ const void *entry AND const void *key)
+ {
+ return strcmp ((const char *) key,
+ 		 ((const memoized_dirlist_entry *) entry)->dir) == 0;
+ }
+
+ /* A hash table mapping directory names to the lists of .java and
+ .class files in that directory. */
+
+ static htab_t memoized_dirlists;
+
+ #endif
+
+ /* Like stat, but avoids actually making the stat system call if we
+ know that it cannot succeed. FILENAME and BUF are as for stat. */
+
+ static int
+ DEFUN(caching_stat, (filename, buf),
+ char *filename AND struct stat *buf)
+ {
+ #if JCF_USE_SCANDIR
+ char *sep;
+ char *base;
+ memoized_dirlist_entry *dent;
+ void **slot;
+
+ /* If the hashtable has not already been created, create it now. */
+ if (!memoized_dirlists)
+ memoized_dirlists = htab_create (37,
+ 				 htab_hash_string,
+ 				 memoized_dirlist_lookup_eq,
+ 				 NULL);
+
+ /* Get the name of the directory. */
+ sep = strrchr (filename, DIR_SEPARATOR);
+ if (sep)
+ {
+ *sep = '0円';
+ base = sep + 1;
+ }
+ else
+ base = filename;
+
+ /* Obtain the entry for this directory form the hash table. */
+ slot = htab_find_slot (memoized_dirlists, filename, INSERT);
+ if (!*slot)
+ {
+ /* We have not already scanned this directory; scan it now. */
+ dent = ((memoized_dirlist_entry *)
+ 	 ALLOC (sizeof (memoized_dirlist_entry)));
+ dent->dir = xstrdup (filename);
+ /* Unfortunately, scandir is not fully standardized. In
+ 	 particular, the type of the function pointer passed as the
+ 	 third argument sometimes takes a "const struct dirent *"
+ 	 parameter, and sometimes just a "struct dirent *". We rely
+ 	 on the ability to interchange these two types of function
+ 	 pointers. */
+ dent->num_files = scandir (filename, &dent->files,
+ 				 java_or_class_file,
+ 				 alphasort);
+ *slot = dent;
+ }
+ else
+ dent = *((memoized_dirlist_entry **) slot);
+
+ /* Put the spearator back. */
+ if (sep)
+ *sep = DIR_SEPARATOR;
+
+ /* If the file is not in the list, there is no need to stat it; it
+ does not exist. */
+ if (dent->num_files != -1
+ && !bsearch (base, dent->files, dent->num_files,
+ 		 sizeof (struct dirent *), compare_path))
+ return -1;
+ #endif
+
+ return stat (filename, buf);
+ }
+
+ /* Returns 1 if the CLASSNAME (really a char *) matches the name
+ stored in TABLE_ENTRY (also a char *). */
+
+ static int
+ DEFUN(memoized_class_lookup_eq, (table_entry, classname),
+ const void *table_entry AND const void *classname)
+ {
+ return strcmp ((const char *)classname, (const char *)table_entry) == 0;
+ }
+
+ /* A hash table keeping track of class names that were not found
+ during class lookup. (There is no need to cache the values
+ associated with names that were found; they are saved in
+ IDENTIFIER_CLASS_VALUE.) */
+ static htab_t memoized_class_lookups;
+
 /* Returns a freshly malloc'd string with the fully qualified pathname
! of the .class file for the class CLASSNAME. CLASSNAME must be
! allocated in permanent storage; this function may retain a pointer
! to it. Returns NULL on failure. If JCF != NULL, it is suitably
! initialized. SOURCE_OK is true if we should also look for .java
! file. */
 const char *
 DEFUN(find_class, (classname, classname_length, jcf, source_ok),
*************** DEFUN(find_class, (classname, classname_
*** 324,334 ****
 char *dep_file;
 void *entry;
 char *java_buffer;
 /* Allocate and zero out the buffer, since we don't explicitly put a
 null pointer when we're copying it below. */
! int buflen = jcf_path_max_len () + classname_length + 10;
! char *buffer = (char *) ALLOC (buflen);
 memset (buffer, 0, buflen);
 java_buffer = (char *) alloca (buflen);
--- 471,497 ----
 char *dep_file;
 void *entry;
 char *java_buffer;
+ int buflen;
+ char *buffer;
+ hashval_t hash;
+
+ /* Create the hash table, if it does not already exist. */
+ if (!memoized_class_lookups)
+ memoized_class_lookups = htab_create (37,
+ 					 htab_hash_string,
+ 					 memoized_class_lookup_eq,
+ 					 NULL);
+
+ /* Loop for this class in the hashtable. If it is present, we've
+ already looked for this class and failed to find it. */
+ hash = htab_hash_string (classname);
+ if (htab_find_with_hash (memoized_class_lookups, classname, hash))
+ return NULL;
 /* Allocate and zero out the buffer, since we don't explicitly put a
 null pointer when we're copying it below. */
! buflen = jcf_path_max_len () + classname_length + 10;
! buffer = (char *) ALLOC (buflen);
 memset (buffer, 0, buflen);
 java_buffer = (char *) alloca (buflen);
*************** DEFUN(find_class, (classname, classname_
*** 381,387 ****
 	 else
 		continue;
 	 }
! 	 class = stat (buffer, &class_buf);
 	}
 if (source_ok)
--- 544,550 ----
 	 else
 		continue;
 	 }
! 	 class = caching_stat(buffer, &class_buf);
 	}
 if (source_ok)
*************** DEFUN(find_class, (classname, classname_
*** 393,399 ****
 	 for (m = 0; m < classname_length; ++m)
 	 java_buffer[m + l] = (classname[m] == '.' ? '/' : classname[m]);
 	 strcpy (java_buffer + m + l, ".java");
! 	 java = stat (java_buffer, &java_buf);
 	 if (java == 0)
 	 break;
 	}
--- 556,562 ----
 	 for (m = 0; m < classname_length; ++m)
 	 java_buffer[m + l] = (classname[m] == '.' ? '/' : classname[m]);
 	 strcpy (java_buffer + m + l, ".java");
! 	 java = caching_stat (java_buffer, &java_buf);
 	 if (java == 0)
 	 break;
 	}
*************** DEFUN(find_class, (classname, classname_
*** 464,469 ****
--- 627,638 ----
 #endif
 free (buffer);
+
+ /* Remember that this class could not be found so that we do not
+ have to look again. */
+ *htab_find_slot_with_hash (memoized_class_lookups, classname, hash, 
INSERT)
+ = (void *) classname;
+
 return NULL;
 found:
 #if JCF_USE_STDIO
Index: gcc/java/jcf-write.c
===================================================================
RCS file: /cvs/gcc/gcc/gcc/java/jcf-write.c,v
retrieving revision 1.99.2.3
diff -c -p -r1.99.2.3 jcf-write.c
*** gcc/java/jcf-write.c	22 Apr 2002 12:53:21 -0000	1.99.2.3
--- gcc/java/jcf-write.c	4 Jun 2002 16:41:20 -0000
*************** write_classfile (clas)
*** 3374,3389 ****
 if (class_file_name != NULL)
 {
! FILE *stream = fopen (class_file_name, "wb");
 if (stream == NULL)
! 	fatal_io_error ("can't open %s for writing", class_file_name);
 jcf_dependency_add_target (class_file_name);
 init_jcf_state (state, work);
 chunks = generate_classfile (clas, state);
 write_chunks (stream, chunks);
 if (fclose (stream))
! 	fatal_io_error ("error closing %s", class_file_name);
 free (class_file_name);
 }
 release_jcf_state (state);
--- 3374,3402 ----
 if (class_file_name != NULL)
 {
! FILE *stream;
! char *temporary_file_name;
!
! /* The .class file is initially written to a ".tmp" file so that
! 	 if multiple instances of the compiler are running at once
! 	 they do not see partially formed class files. */
! temporary_file_name = concat (class_file_name, ".tmp", NULL);
! stream = fopen (temporary_file_name, "wb");
 if (stream == NULL)
! 	fatal_io_error ("can't open %s for writing", temporary_file_name);
 jcf_dependency_add_target (class_file_name);
 init_jcf_state (state, work);
 chunks = generate_classfile (clas, state);
 write_chunks (stream, chunks);
 if (fclose (stream))
! 	fatal_io_error ("error closing %s", temporary_file_name);
! if (rename (temporary_file_name, class_file_name) == -1)
! 	{
! 	 remove (temporary_file_name);
! 	 fatal_io_error ("can't create %s", class_file_name);
! 	}
! free (temporary_file_name);
 free (class_file_name);
 }
 release_jcf_state (state);
Index: gcc/java/jcf.h
===================================================================
RCS file: /cvs/gcc/gcc/gcc/java/jcf.h,v
retrieving revision 1.28.2.2
diff -c -p -r1.28.2.2 jcf.h
*** gcc/java/jcf.h	10 Apr 2002 13:09:01 -0000	1.28.2.2
--- gcc/java/jcf.h	4 Jun 2002 16:41:20 -0000
*************** The Free Software Foundation is independ
*** 63,68 ****
--- 63,76 ----
 #define JCF_word JCF_u4
 #endif
+ /* If we have both "scandir" and "alphasort", we can cache directory
+ listings to reduce the time taken to search the classpath. */
+ #if defined(HAVE_SCANDIR) && defined(HAVE_ALPHASORT)
+ #define JCF_USE_SCANDIR 1
+ #else
+ #define JCF_USE_SCANDIR 0
+ #endif
+
 struct JCF;
 typedef int (*jcf_filbuf_t) PARAMS ((struct JCF*, int needed));
Index: gcc/java/parse.y
===================================================================
RCS file: /cvs/gcc/gcc/gcc/java/parse.y,v
retrieving revision 1.353.2.18
diff -c -p -r1.353.2.18 parse.y
*** gcc/java/parse.y	25 Apr 2002 01:08:29 -0000	1.353.2.18
--- gcc/java/parse.y	4 Jun 2002 16:41:22 -0000
*************** java_expand_classes ()
*** 9046,9055 ****
 for (cur_ctxp = ctxp_for_generation; cur_ctxp; cur_ctxp = 
cur_ctxp->next)
 {
 tree current;
 ctxp = cur_ctxp;
! for (current = ctxp->class_list; current; current = TREE_CHAIN 
(current))
 	{
! 	 current_class = TREE_TYPE (current);
 	 outgoing_cpool = TYPE_CPOOL (current_class);
 	 if (flag_emit_class_files)
 	 write_classfile (current_class);
--- 9046,9075 ----
 for (cur_ctxp = ctxp_for_generation; cur_ctxp; cur_ctxp = 
cur_ctxp->next)
 {
 tree current;
+ tree reversed_class_list = NULL;
+
 ctxp = cur_ctxp;
!
! /* We write out the classes in reverse order. This ensures that
! 	 inner classes are written before their containing classes,
! 	 which is important for parallel builds. Otherwise, the
! 	 class file for the outer class may be found, but the class
! 	 file for the inner class may not be present. In that
! 	 situation, the compiler cannot fall back to the original
! 	 source, having already read the outer class, so we must
! 	 prevent that situation. */
! for (current = ctxp->class_list;
! 	 current;
! 	 current = TREE_CHAIN (current))
! 	reversed_class_list
! 	 = tree_cons (NULL_TREE, current, reversed_class_list);
! ggc_add_tree_root (&reversed_class_list, 1);
!
! for (current = reversed_class_list;
! 	 current;
! 	 current = TREE_CHAIN (current))
 	{
! 	 current_class = TREE_TYPE (TREE_VALUE (current));
 	 outgoing_cpool = TYPE_CPOOL (current_class);
 	 if (flag_emit_class_files)
 	 write_classfile (current_class);
*************** java_expand_classes ()
*** 9061,9066 ****
--- 9081,9088 ----
 	 finish_class ();
 	 }
 	}
+
+ ggc_del_root (&reversed_class_list);
 }
 }
Index: libjava/Makefile.am
===================================================================
RCS file: /cvs/gcc/gcc/libjava/Makefile.am,v
retrieving revision 1.202.2.11
diff -c -p -r1.202.2.11 Makefile.am
*** libjava/Makefile.am	3 May 2002 18:22:15 -0000	1.202.2.11
--- libjava/Makefile.am	4 Jun 2002 16:41:23 -0000
*************** install-exec-hook:
*** 163,207 ****
 	 $(LN_S) libgcjx.la gnu-awt-xlib.la; \
 	fi
! ## Make the .class files depend on the .zip file. This seems
! ## backwards, but is right. This doesn't catch all the .class files,
! ## but that is ok, because the ones it fails to pick up are defined in
! ## a .java file with some other class which is caught. Note that we
! ## only want to create headers for those files which do not have
! ## hand-maintained headers.
! $(built_java_source_files:.java=.class): libgcj-@gcc_version@.jar
! $(java_source_files:.java=.class): libgcj-@gcc_version@.jar
!
! ## The .class files for X will not be included in libgcj.jar, but the
! ## rule for libgcj.jar will cause all out-of-date .class files to be
! ## built. We need this to generate headers for the nat-files.
! $(x_java_source_files:.java=.class): libgcj-@gcc_version@.jar
!
! ## We have the zip file depend on the java sources and not the class
! ## files, because we don't know the names of all the class files.
! ## FIXME: this method fails in a peculiar case: if libgcj.jar is
! ## up-to-date, and foo.class is removed, and bar.java is touched, then
! ## `make libgcj.jar' will not rebuilt foo.class. That's because
! ## libgcj.jar is not out-of-date with respect to foo.java.
! libgcj-@gcc_version@.jar: $(built_java_source_files) $(java_source_files) 
$(x_java_source_files)
! ## Create a list of all Java sources, without exceeding any shell limits.
! 	@: $(shell echo Creating list of files to compile...) $(shell rm -f 
tmp-list || :) $(shell touch tmp-list) $(foreach source,$?,$(shell echo 
$(source) >> tmp-list))
! 	@set fnord $(MAKEFLAGS); amf=$2ドル; fail=no; \
! 	javac="$(JAVAC)"; \
! 	cat tmp-list | (while read f; do \
! 	 echo $$javac $(JCFLAGS) -classpath \'\' -bootclasspath 
$(here):$(srcdir) -d $(here) $$f; \
! 	 $$javac $(JCFLAGS) -classpath '' -bootclasspath $(here):$(srcdir) -d 
$(here) $$f \
! 	 || case "$$amf" in *=*) exit 1;; *k*) fail=yes ;; *) exit 1;; esac; \
! 	done; \
! 	test "$$fail" = no)
! 	-@rm -f tmp-list libgcj-@gcc_version@.jar
 ## Note that we explicitly want to include directory information.
 	find java gnu javax org -type d -o -type f -name '*.class' | \
 	 sed -e '/\/\./d' -e '/\/xlib/d' | \
 	 $(ZIP) cfM0E@ $@
 MOSTLYCLEANFILES = $(javao_files) $(nat_files) $(nat_headers) $(c_files) 
$(x_javao_files) $(x_nat_files) $(x_nat_headers)
! CLEANFILES = tmp-list libgcj-@gcc_version@.jar
 clean-local:
 ## We just remove every .class file that was created.
--- 163,188 ----
 	 $(LN_S) libgcjx.la gnu-awt-xlib.la; \
 	fi
! all_java_source_files = \
! $(java_source_files) \
! $(built_java_source_files) \
! $(x_java_source_files)
!
! all_java_class_files = $(all_java_source_files:.java=.class)
!
! .java.class:
! 	$(JAVAC) $(JCFLAGS) -classpath '' -bootclasspath $(here):$(srcdir) \
! -d $(here) $<
!
! libgcj-@gcc_version@.jar: $(all_java_class_files)
! 	-@rm -f libgcj-@gcc_version@.jar
 ## Note that we explicitly want to include directory information.
 	find java gnu javax org -type d -o -type f -name '*.class' | \
 	 sed -e '/\/\./d' -e '/\/xlib/d' | \
 	 $(ZIP) cfM0E@ $@
 MOSTLYCLEANFILES = $(javao_files) $(nat_files) $(nat_headers) $(c_files) 
$(x_javao_files) $(x_nat_files) $(x_nat_headers)
! CLEANFILES = libgcj-@gcc_version@.jar
 clean-local:
 ## We just remove every .class file that was created.
*************** texinfo: TexinfoDoclet.class
*** 1910,1916 ****
 ## internally by libgcj. We can't make the .o files depend on 
nat_headers,
 ## because in that case we'll force a complete rebuild of
 ## the C++ code whenever any .java file is touched.
! all-recursive: $(nat_headers) $(x_nat_headers)
 ## ################################################################
--- 1891,1899 ----
 ## internally by libgcj. We can't make the .o files depend on 
nat_headers,
 ## because in that case we'll force a complete rebuild of
 ## the C++ code whenever any .java file is touched.
! ## Also force all the class files to build first. This makes them build in
! ## the right order to improve performance.
! all-recursive: $(all_java_class_files) $(nat_headers) $(x_nat_headers)
 ## ################################################################
 

---

• Previous message (by thread): PATCH: Improve Java build times
• Next message (by thread): gcc-3.1 build problem in the java area(?)
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

---

More information about the Java mailing list