gmod-schema Mailing List for Generic Model Organism Database Project

Hello,
Since SourceForge and my ISP were having difficulties at the time I
initially posted this announcement, I wanted to post it again in case it
was missed by anybody. Also, I have a few items to add:
 - The deadline to be sure to get the GMOD meeting rate at the hotel is
May 29. It may be possible to get the meeting rate after then, but
there are no guarantees.
 - There may be funds available to help defray the travel costs for
some attendees; since there is no registration fee for this meeting
though, these funds are limited. Please contact me for more details.
Thanks,
Scott
-------------------------------------------------------------
And now, the original announcement:
I am pleased to announce that the semi-annual GMOD meeting will be held=20
a the National Evolutionary Synthesis Center <http://www.nescent.org/>=20
in Durham, NC on June 29 and 30. In addition to 'typical' topics for=20
discussion (new applications, progress on making applications=20
interoperate), we will discuss how existing applications can be used or=20
modified for evolutionary biologists and what applications could be=20
developed in conjunction with GMOD to facilitate evolutionary model=20
organism databases. I am also accepting suggestions for topics or=20
talks.=20
=20
We've received a discount on hotel rooms at the nearby Millennium Hotel=20
Durham for 79ドル/night. The '6136' group code is used for this discount:=20
=20
https://reservations.synxis.com/LBE/rez.aspx?Hotel=3D11543&Chain=3D5303=3D1=
&group=3D6136&arrive=3D06/28/2006&depart=3D06/30/2006&adult=3D1&child=3D0=20
=20
Note that there is no registration fee for the meeting itself but there=20
is a deadline: you must register by June 20 and there is a limit of 50=20
people.=20
=20
The registration page is at http://www.gmod.org/register.html=20
=20
Thanks and I look forward to seeing you in Durham!=20
=20
Scott=20
--=20
------------------------------------------------------------------------
Scott Cain, Ph. D. ca...@cs...
GMOD Coordinator (http://www.gmod.org/) 216-392-3087
Cold Spring Harbor Laboratory

Hi Mohammad,
I just committed to cvs a fix to the bulk loader and database adapter
that provides a '--nouniquecache' which suppresses the creation of the
uniquename_cache when the loader starts up. I left the default be to
create the cache, because it does speed things up quite a bit. It's
only when you've got a few million rows that the cache becomes so big
that you run into memory problems.
Just to geek out a little bit, I'll describe what I did: fortunately for
me, I'd already written the loader to access the uniquename_cache
through an object method, so that was the only piece of code I had to
touch (other than actually wiring up accepting the command line option)
to implement this. Now, the uniquename_cache method checks if the
--nouniquecache option was set, and if it is, after it checks the
internal cache, queries the database for that uniquename before
returning. It was actually good that I went through this exercise,
because I did find a case where I hadn't specified the return for the
uniquename_cache method, so I don't know what it was returning (it
should have, and probably was, returning false, 0 or undef, but with
perl who knows unless you tell it).
So, just to be clear, the loader still creates uniquename_cache, but if
--nouniquecache is set, it only populates it with uniquenames and
related data from the current GFF3 file. This causes the memory foot
print of the loader to grow while it is processing the file, but not
nearly as large as if it swallowed the whole database when the database
is big.
Scott
On Wed, 2006年05月24日 at 18:04 -0400, Scott Cain wrote:
> Hi Mohammad,
>=20
> Since you are using the bleeding edge chado, I would appreciate it if
> you could respond to schema mailing list so others can be kept up to
> date with what the issues are.
>=20
> I'll answer the last question first: when you want to update your cvs
> directory, you just need to run this cvs command to get any newer files:
>=20
> cvs update -d
>=20
> When you run that in the highest level directory of the cvs repository,
> it will recurse through all of the directories in the repository and
> fetch any new files. One note though: the SourceForge cvs server has
> changed recently, so you'll need to see this node on the gmod website
> for directions for fixing your repository so you'll be able to do the
> update:
>=20
> http://www.gmod.org/node/186
>=20
> Now for the other questions (answered inline below)
>=20
> Scott
>=20
>=20
> On Wed, 2006年05月24日 at 17:17 -0400, Mohammad Fallahi wrote:
> > Hi Scot,
> >=20
> >=20
> > Thanks a lot for the answer.
> >=20
> > One of my colleagues is asking the below question?
> > -------------------------------------------
> > What is the uniquename cache used for?
>=20
> The uniquename is similar to the primary key of the feature table. The
> cache is used to make the loader run faster by prefetching existing
> uniquenames and their feature_ids, type_ids and organism_ids
> (uniquename, type_id and organism_id form a unique constraint on the
> feature table).
>=20
> > Is it to be sure that feature IDs are unique? something else?
>=20
> Yes and yes: if a GFF3 ID is does not conform to the unique constraint
> on the table, the loader will generate a new one. The cache is also
> used in the case where Target/match attributes refer to features that
> are already in the database (so that feature locations can be relative
> to both the source feature (eg, a chromosome) and whatever the hit was
> to (eg, an EST)).
> =20
> >=20
> > I assume that "undef the uniquename cache" means empty it.
>=20
> Yes.
>=20
> > Does he intend to make the change and send you the code?
>=20
> See the initial comments above about updating your cvs repository.
>=20
> > This sounds like a simple fix, but what are the implications?
>=20
> None really. The hash is being emptied after the processing of the GFF3
> file is done and the temp files for loading the database have been
> created (your crash occurred after that point).
> >=20
> > If by doing this, we need to guarantee a priori that our names are
> > unique
> > before loading them, this isn't too hard.
>=20
> No, you don't need to do anything, however I have found a caveat as I've
> been doing testing on this change to the loader: it doesn't appear to
> give much memory back to the OS after the caches are emptied (perl can
> be greedy that way). As it happens, I am working on building a database
> with a few million feature rows as well, so I suspect I may run into the
> same problem very soon, at which point I will be forced to fix this (by
> tying the cache to a temporary BerkeleyDB created by the loader). The
> way things look, I will probably be working on that tomorrow. (perl is
> sucking up 0.5G real ram and growing on my computer that only has 1.25G,
> and I'm no where near done).
> >=20
> > -----------------------
> >=20
> > I really appreciate it if you could answer the above questions.
> >=20
> > By the way, is there any document that explains what is the best way to
> > update my bleeding gmod when there is a change on CVS?
> >=20
> > Thanks,
> > Mohammad
> >=20
> > On 5/24/06 4:30 PM, "Scott Cain" <ca...@cs...> wrote:
> >=20
> > > Hi Mohammad,
> > >=20
> > > I'm wondering if you might be running out of memory. The uniquename
> > > cache could be getting pretty full when you've got that many features=
.
> > > The 'Terminated' message looks like the type of thing the OS might do=
 if
> > > a process is going to bring down the system because it has grown too
> > > big. As a stop gap, I'll undef the uniquename cache after the bulk
> > > files are built (and therefore, the cache is no longer needed). A
> > > longer term solution will be to tie the caches to a BerkeleyDB, which=
 I
> > > will put on my todo list.
> > >=20
> > > I committed the updated loader and Adapter.pm. Please let me know ho=
w
> > > they work.
> > >=20
> > > Scott
> > >=20
> > >=20
> > >=20
> > > On Mon, 2006年05月22日 at 17:21 -0400, Mohammad Fallahi-Sichani wrote:
> > >>=20
> > >> Hi Scott,
> > >>=20
> > >> After loading about 3.4 million featuers (3.5GB data) in small files
> > >> (100MB) into the chado database using gmod_new_bulk_load_gff3.pl, I
> > >> cannot load more data anymore. I was loading the data using
> > >> --skip_vacuum when I faced this problem. I tried it without
> > >> --skip_vacuum and even after performing a "VACUUM FULL ANALYSE" but
> > >> the same problem showed up. Below are the info messages I got:
> > >> -----------------------------------------
> > >> /usr/local/src/schema/chado # ./load/bin/gmod_new_bulk_load_gff3.pl
> > >> --organism X --gfffile segment_bh.gff
> > >> Creating a uniquename cache...done
> > >> Preparing data for inserting into the X database
> > >> (This may take a while ...)
> > >> Loading data into feature table ...
> > >> Loading data into featureloc table ...
> > >> Loading data into feature_relationship table ...
> > >> Loading data into featureprop table ...
> > >> Terminated
> > >> -------------------------------------------
> > >>=20
> > >>=20
> > >> I really appreciate it if you could help with this problem.
> > >>=20
> > >>=20
> > >> Thanks a lot,
> > >> Mohammad,
> > >>=20
> > >>=20
> > >> -----Original Message-----
> > >> From: Scott Cain [mailto:ca...@cs...]
> > >> Sent: Fri 5/19/2006 10:39 AM
> > >> To: Mohammad Fallahi-Sichani
> > >> Cc: gmod schema
> > >> Subject: Re: Optimizing database
> > >>=20
> > >> Yes, that is exactly what skip_vacuum is for, however, it may not he=
lp
> > >> as much as you think overall, since the VACUUM ANALYZE can speed up
> > >> subsequent loads, since each load has to query the database many
> > >> times.
> > >> You may want to experiment with skipping some of the time and allowi=
ng
> > >> it others.
> > >>=20
> > >> Scott
> > >>=20
> > >>=20
> > >> On Fri, 2006年05月19日 at 10:33 -0400, Mohammad Fallahi wrote:
> > >>> Hi Scott,
> > >>>=20
> > >>> I'm loading around 50 gff3 files (100 MB each) into the chado
> > >> database using
> > >>> a script that uses gmod_new_bulk_load_gff3.pl. However, after
> > >> loading each
> > >>> file it takes a long time for "Optimizing database". In order to
> > >> make the
> > >>> loading process faster I prefer to optimize the database only after
> > >> loading
> > >>> the last file. Is this possible by "--skip_vacuum" option?
> > >>>=20
> > >>> Thanks,
> > >>> Mohammad,
> > >>>=20
> > >>>=20
> > >>>=20
> > >> --------------------------------------------------------------------=
----
> > >>>=20
> > >>> Mohammad Fallahi (M.Sc.)
> > >>> Scientific Software Engineer
> > >>> Department of Informatics
> > >>>=20
> > >>> Scripps Florida (The Scripps Research Institute)
> > >>> 5353 Parkside Drive, RF-A
> > >>> Jupiter, FL 33458
> > >>>=20
> > >>> Telephone: 561-799-8919
> > >>> Fax:561-799-8952
> > >>> Email: mfa...@sc...
> > >>>=20
> > >>>=20
> > >> --
> > >> --------------------------------------------------------------------=
----
> > >> Scott Cain, Ph. D.
> > >> ca...@cs...
> > >> GMOD Coordinator (http://www.gmod.org/)
> > >> 216-392-3087
> > >> Cold Spring Harbor Laboratory
> > >>=20
> > >>=20
> > >>=20
> >=20
> > -----------------------------------------------------------------------=
-
> >=20
> > Mohammad Fallahi (M.Sc.)
> > Scientific Software Engineer
> > Department of Informatics
> >=20
> > Scripps Florida (The Scripps Research Institute)
> > 5353 Parkside Drive, RF-A
> > Jupiter, FL 33458
> >=20
> > Telephone: 561-799-8919
> > Fax:561-799-8952
> > Email: mfa...@sc...
> >=20
> >=20
--=20
------------------------------------------------------------------------
Scott Cain, Ph. D. ca...@cs...
GMOD Coordinator (http://www.gmod.org/) 216-392-3087
Cold Spring Harbor Laboratory

Hi Mohammad,
Since you are using the bleeding edge chado, I would appreciate it if
you could respond to schema mailing list so others can be kept up to
date with what the issues are.
I'll answer the last question first: when you want to update your cvs
directory, you just need to run this cvs command to get any newer files:
 cvs update -d
When you run that in the highest level directory of the cvs repository,
it will recurse through all of the directories in the repository and
fetch any new files. One note though: the SourceForge cvs server has
changed recently, so you'll need to see this node on the gmod website
for directions for fixing your repository so you'll be able to do the
update:
 http://www.gmod.org/node/186
Now for the other questions (answered inline below)
Scott
On Wed, 2006年05月24日 at 17:17 -0400, Mohammad Fallahi wrote:
> Hi Scot,
>=20
>=20
> Thanks a lot for the answer.
>=20
> One of my colleagues is asking the below question?
> -------------------------------------------
> What is the uniquename cache used for?
The uniquename is similar to the primary key of the feature table. The
cache is used to make the loader run faster by prefetching existing
uniquenames and their feature_ids, type_ids and organism_ids
(uniquename, type_id and organism_id form a unique constraint on the
feature table).
> Is it to be sure that feature IDs are unique? something else?
Yes and yes: if a GFF3 ID is does not conform to the unique constraint
on the table, the loader will generate a new one. The cache is also
used in the case where Target/match attributes refer to features that
are already in the database (so that feature locations can be relative
to both the source feature (eg, a chromosome) and whatever the hit was
to (eg, an EST)).
=20
>=20
> I assume that "undef the uniquename cache" means empty it.
Yes.
> Does he intend to make the change and send you the code?
See the initial comments above about updating your cvs repository.
> This sounds like a simple fix, but what are the implications?
None really. The hash is being emptied after the processing of the GFF3
file is done and the temp files for loading the database have been
created (your crash occurred after that point).
>=20
> If by doing this, we need to guarantee a priori that our names are
> unique
> before loading them, this isn't too hard.
No, you don't need to do anything, however I have found a caveat as I've
been doing testing on this change to the loader: it doesn't appear to
give much memory back to the OS after the caches are emptied (perl can
be greedy that way). As it happens, I am working on building a database
with a few million feature rows as well, so I suspect I may run into the
same problem very soon, at which point I will be forced to fix this (by
tying the cache to a temporary BerkeleyDB created by the loader). The
way things look, I will probably be working on that tomorrow. (perl is
sucking up 0.5G real ram and growing on my computer that only has 1.25G,
and I'm no where near done).
>=20
> -----------------------
>=20
> I really appreciate it if you could answer the above questions.
>=20
> By the way, is there any document that explains what is the best way to
> update my bleeding gmod when there is a change on CVS?
>=20
> Thanks,
> Mohammad
>=20
> On 5/24/06 4:30 PM, "Scott Cain" <ca...@cs...> wrote:
>=20
> > Hi Mohammad,
> >=20
> > I'm wondering if you might be running out of memory. The uniquename
> > cache could be getting pretty full when you've got that many features.
> > The 'Terminated' message looks like the type of thing the OS might do i=
f
> > a process is going to bring down the system because it has grown too
> > big. As a stop gap, I'll undef the uniquename cache after the bulk
> > files are built (and therefore, the cache is no longer needed). A
> > longer term solution will be to tie the caches to a BerkeleyDB, which I
> > will put on my todo list.
> >=20
> > I committed the updated loader and Adapter.pm. Please let me know how
> > they work.
> >=20
> > Scott
> >=20
> >=20
> >=20
> > On Mon, 2006年05月22日 at 17:21 -0400, Mohammad Fallahi-Sichani wrote:
> >>=20
> >> Hi Scott,
> >>=20
> >> After loading about 3.4 million featuers (3.5GB data) in small files
> >> (100MB) into the chado database using gmod_new_bulk_load_gff3.pl, I
> >> cannot load more data anymore. I was loading the data using
> >> --skip_vacuum when I faced this problem. I tried it without
> >> --skip_vacuum and even after performing a "VACUUM FULL ANALYSE" but
> >> the same problem showed up. Below are the info messages I got:
> >> -----------------------------------------
> >> /usr/local/src/schema/chado # ./load/bin/gmod_new_bulk_load_gff3.pl
> >> --organism X --gfffile segment_bh.gff
> >> Creating a uniquename cache...done
> >> Preparing data for inserting into the X database
> >> (This may take a while ...)
> >> Loading data into feature table ...
> >> Loading data into featureloc table ...
> >> Loading data into feature_relationship table ...
> >> Loading data into featureprop table ...
> >> Terminated
> >> -------------------------------------------
> >>=20
> >>=20
> >> I really appreciate it if you could help with this problem.
> >>=20
> >>=20
> >> Thanks a lot,
> >> Mohammad,
> >>=20
> >>=20
> >> -----Original Message-----
> >> From: Scott Cain [mailto:ca...@cs...]
> >> Sent: Fri 5/19/2006 10:39 AM
> >> To: Mohammad Fallahi-Sichani
> >> Cc: gmod schema
> >> Subject: Re: Optimizing database
> >>=20
> >> Yes, that is exactly what skip_vacuum is for, however, it may not help
> >> as much as you think overall, since the VACUUM ANALYZE can speed up
> >> subsequent loads, since each load has to query the database many
> >> times.
> >> You may want to experiment with skipping some of the time and allowing
> >> it others.
> >>=20
> >> Scott
> >>=20
> >>=20
> >> On Fri, 2006年05月19日 at 10:33 -0400, Mohammad Fallahi wrote:
> >>> Hi Scott,
> >>>=20
> >>> I'm loading around 50 gff3 files (100 MB each) into the chado
> >> database using
> >>> a script that uses gmod_new_bulk_load_gff3.pl. However, after
> >> loading each
> >>> file it takes a long time for "Optimizing database". In order to
> >> make the
> >>> loading process faster I prefer to optimize the database only after
> >> loading
> >>> the last file. Is this possible by "--skip_vacuum" option?
> >>>=20
> >>> Thanks,
> >>> Mohammad,
> >>>=20
> >>>=20
> >>>=20
> >> ----------------------------------------------------------------------=
--
> >>>=20
> >>> Mohammad Fallahi (M.Sc.)
> >>> Scientific Software Engineer
> >>> Department of Informatics
> >>>=20
> >>> Scripps Florida (The Scripps Research Institute)
> >>> 5353 Parkside Drive, RF-A
> >>> Jupiter, FL 33458
> >>>=20
> >>> Telephone: 561-799-8919
> >>> Fax:561-799-8952
> >>> Email: mfa...@sc...
> >>>=20
> >>>=20
> >> --
> >> ----------------------------------------------------------------------=
--
> >> Scott Cain, Ph. D.
> >> ca...@cs...
> >> GMOD Coordinator (http://www.gmod.org/)
> >> 216-392-3087
> >> Cold Spring Harbor Laboratory
> >>=20
> >>=20
> >>=20
>=20
> ------------------------------------------------------------------------
>=20
> Mohammad Fallahi (M.Sc.)
> Scientific Software Engineer
> Department of Informatics
>=20
> Scripps Florida (The Scripps Research Institute)
> 5353 Parkside Drive, RF-A
> Jupiter, FL 33458
>=20
> Telephone: 561-799-8919
> Fax:561-799-8952
> Email: mfa...@sc...
>=20
>=20
--=20
------------------------------------------------------------------------
Scott Cain, Ph. D. ca...@cs...
GMOD Coordinator (http://www.gmod.org/) 216-392-3087
Cold Spring Harbor Laboratory

Hi Mohammad,
I'm wondering if you might be running out of memory. The uniquename
cache could be getting pretty full when you've got that many features.
The 'Terminated' message looks like the type of thing the OS might do if
a process is going to bring down the system because it has grown too
big. As a stop gap, I'll undef the uniquename cache after the bulk
files are built (and therefore, the cache is no longer needed). A
longer term solution will be to tie the caches to a BerkeleyDB, which I
will put on my todo list.
I committed the updated loader and Adapter.pm. Please let me know how
they work.
Scott
On Mon, 2006年05月22日 at 17:21 -0400, Mohammad Fallahi-Sichani wrote:
>=20
> Hi Scott,
>=20
> After loading about 3.4 million featuers (3.5GB data) in small files
> (100MB) into the chado database using gmod_new_bulk_load_gff3.pl, I
> cannot load more data anymore. I was loading the data using
> --skip_vacuum when I faced this problem. I tried it without
> --skip_vacuum and even after performing a "VACUUM FULL ANALYSE" but
> the same problem showed up. Below are the info messages I got:
> -----------------------------------------
> /usr/local/src/schema/chado # ./load/bin/gmod_new_bulk_load_gff3.pl
> --organism X --gfffile segment_bh.gff
> Creating a uniquename cache...done
> Preparing data for inserting into the X database
> (This may take a while ...)
> Loading data into feature table ...
> Loading data into featureloc table ...
> Loading data into feature_relationship table ...
> Loading data into featureprop table ...
> Terminated
> -------------------------------------------
>=20
>=20
> I really appreciate it if you could help with this problem.
>=20
>=20
> Thanks a lot,
> Mohammad,
>=20
>=20
> -----Original Message-----
> From: Scott Cain [mailto:ca...@cs...]
> Sent: Fri 5/19/2006 10:39 AM
> To: Mohammad Fallahi-Sichani
> Cc: gmod schema
> Subject: Re: Optimizing database
>=20
> Yes, that is exactly what skip_vacuum is for, however, it may not help
> as much as you think overall, since the VACUUM ANALYZE can speed up
> subsequent loads, since each load has to query the database many
> times.
> You may want to experiment with skipping some of the time and allowing
> it others.
>=20
> Scott
>=20
>=20
> On Fri, 2006年05月19日 at 10:33 -0400, Mohammad Fallahi wrote:
> > Hi Scott,
> >
> > I'm loading around 50 gff3 files (100 MB each) into the chado
> database using
> > a script that uses gmod_new_bulk_load_gff3.pl. However, after
> loading each
> > file it takes a long time for "Optimizing database". In order to
> make the
> > loading process faster I prefer to optimize the database only after
> loading
> > the last file. Is this possible by "--skip_vacuum" option?
> >
> > Thanks,
> > Mohammad,
> >
> >
> >
> ------------------------------------------------------------------------
> >
> > Mohammad Fallahi (M.Sc.)
> > Scientific Software Engineer
> > Department of Informatics
> >
> > Scripps Florida (The Scripps Research Institute)
> > 5353 Parkside Drive, RF-A
> > Jupiter, FL 33458
> >
> > Telephone: 561-799-8919
> > Fax:561-799-8952
> > Email: mfa...@sc...
> >
> >
> --
> ------------------------------------------------------------------------
> Scott Cain, Ph. D.
> ca...@cs...
> GMOD Coordinator (http://www.gmod.org/)
> 216-392-3087
> Cold Spring Harbor Laboratory
>=20
>=20
>=20
--=20
------------------------------------------------------------------------
Scott Cain, Ph. D. ca...@cs...
GMOD Coordinator (http://www.gmod.org/) 216-392-3087
Cold Spring Harbor Laboratory

Hi Scott (and anyone who is interested),
If you checkout the main trunk of Modware this ought to work with the main 
trunk (future 0.01 release?) of GMOD.
cvs -z3 
-d:pserver:ano...@gm...:/cvsroot/gmod-ware co -P 
modware
Let me know if there are any problems!
Eric
At 10:53 AM 5/17/2006, Scott Cain wrote:
>Hi Eric,
>
>Do you have any guesses on how this will work with the upcoming 0.01
>release? I'm guessing it relies on the Class::DBI classes, so
>presumably it will probably work fine, right?
>
>Scott
>
>
>On Tue, 2006年05月16日 at 14:05 -0500, Eric Just wrote:
> > Hi Everyone,
> >
> > We are announcing a new Sourceforge Project called Modware that may be of
> > interest to you. It is an object-oriented API written in Perl that
> > creates in-memory object representations of biological features stored 
> in a
> > Chado database.
> >
> > For download please visit our Sourceforge project page:
> > http://sourceforge.net/projects/gmod-ware
> >
> > For API documentation and some short examples of selected use cases visit
> > our project home page:
> > http://gmod-ware.sourceforge.net/
> >
> > This software is adapted from the production middleware code that 
> dictyBase
> > uses. Modware 0.1 requires the latest stable GMOD release: 0.003 be
> > installed. We are currently calling it a release candidate and if we get
> > some feedback will call it an official release if there are no major
> > install bugs (we've installed it only on two different machines). If you
> > would like a version that works on the latest CVS version of GMOD, let me
> > know and I'll expedite getting that out the door.
> >
> > Lastly, please use the direct download version, we have not fully 
> recovered
> > from the recent Sourceforge CVS issues.
> >
> > Please try the software out and let us know what you think!
> >
> >
> > Sincerely,
> > Eric Just and Sohel Merchant
> >
> > e-just at northwestern.edu
> > s-merchant at northwestern.edu
> >
> >
> > ============================================
> >
> > Eric Just
> > e-...@no...
> > dictyBase Programmer
> > Center for Genetic Medicine
> > Northwestern University
> > http://dictybase.org
> >
> > ============================================
> >
> >
> >
> >
> > -------------------------------------------------------
> > Using Tomcat but need to do more? Need to support web services, security?
> > Get stuff done quickly with pre-integrated technology to make your job 
> easier
> > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> > _______________________________________________
> > Gmod-announce mailing list
> > Gmo...@li...
> > https://lists.sourceforge.net/lists/listinfo/gmod-announce
>--
>------------------------------------------------------------------------
>Scott Cain, Ph. D. ca...@cs...
>GMOD Coordinator (http://www.gmod.org/) 216-392-3087
>Cold Spring Harbor Laboratory
>
>
>
>-------------------------------------------------------
>Using Tomcat but need to do more? Need to support web services, security?
>Get stuff done quickly with pre-integrated technology to make your job easier
>Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
>http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>_______________________________________________
>Gmod-architecture mailing list
>Gmo...@li...
>https://lists.sourceforge.net/lists/listinfo/gmod-architecture
============================================
Eric Just
e-...@no...
dictyBase Programmer
Center for Genetic Medicine
Northwestern University
http://dictybase.org
============================================ 

Yes, that is exactly what skip_vacuum is for, however, it may not help
as much as you think overall, since the VACUUM ANALYZE can speed up
subsequent loads, since each load has to query the database many times.
You may want to experiment with skipping some of the time and allowing
it others.
Scott
On Fri, 2006年05月19日 at 10:33 -0400, Mohammad Fallahi wrote:
> Hi Scott,
>=20
> I'm loading around 50 gff3 files (100 MB each) into the chado database us=
ing
> a script that uses gmod_new_bulk_load_gff3.pl. However, after loading ea=
ch
> file it takes a long time for "Optimizing database". In order to make th=
e
> loading process faster I prefer to optimize the database only after loadi=
ng
> the last file. Is this possible by "--skip_vacuum" option?
>=20
> Thanks,
> Mohammad,
>=20
>=20
> ------------------------------------------------------------------------
>=20
> Mohammad Fallahi (M.Sc.)
> Scientific Software Engineer
> Department of Informatics
>=20
> Scripps Florida (The Scripps Research Institute)
> 5353 Parkside Drive, RF-A
> Jupiter, FL 33458
>=20
> Telephone: 561-799-8919
> Fax:561-799-8952
> Email: mfa...@sc...
>=20
>=20
--=20
------------------------------------------------------------------------
Scott Cain, Ph. D. ca...@cs...
GMOD Coordinator (http://www.gmod.org/) 216-392-3087
Cold Spring Harbor Laboratory

Hi Scott,
Just checking if you are you going to include something that lets you load 
primary_ids from the gff3 loader in the next version? (this email 
contains the previous discussion). I think there were basically two proposals:
1. Pass a flag --ID2dbxrefid=<db.name> and the ID field in the GFF3 is 
converted to the dbxref_id in the feature table. Maybe you wouldn't even 
have to specify the db.name though since it can (and does) grab it from the 
config file if it is not specifed, right?
2. No extra flag but any dbxref that references db.name of database being 
loaded gets loaded as a dbxref_id in the feature table instead of the 
feature_dbxref table.
I think I slightly prefer number 2 as it is a little better in that it 
makes no assumption on how people are using the ID field and does not 
require an extra command line flag, although option 1 is also acceptable.
Eric
At 08:22 AM 4/26/2006, Eric Just wrote:
>At 01:12 PM 4/25/2006, Scott Cain wrote:
>>Hi Eric,
>>
>>I cc'ed this to the schema mailing list as well.
>>
>>How would you propose to implement this? Would one provide a command
>>line option like --ID2dbxrefid=<db.name>, and then for each feature with
>>an ID, construct a dbxref with that db_id, the ID as the accession and
>>then put that dbxref_id in feature.dbxref_id? I think something like
>>that could be done without too much work. Certainly it has to be a
>>non-default option, as frequently the ID will not be meaningful as a
>>dbxref. Also, it will assume that db.name already exists in the
>>database.
>
>I think that works well. Another option would be to have two arguments:
>
>--ID2dbxrefid true --dbname dictyBase
>
>seems like a dbname argument could really be helpful in other places? If 
>not, I think the proposed method is great. One last option, would be to 
>only specify dbname and then any dbxrefs with that dbname would go into 
>the feature.dbxref field instead of linking dbxrefs the current way.
>
>
>>Presumably, if one instead wanted to use a feature attribute (for
>>example "primary_dbxref=abcdded", that could be done too, but I think it
>>should be done via a subclassing of Bio::GMOD::DB::Adapter. (Though the
>>bulk loader will have to be modified to pass unknown commandline flags
>>to the adaptor for it to deal with.)
>
>I think the previous method is simpler in that it uses what is available 
>in GFF
>
>
>>Any suggestions on a better command line flag? I don't much care for
>>the one I used up there.
>
>
>Thanks for looking into this.
>
>Eric
>
>
>>Scott
>>
>>
>>On Mon, 2006年04月24日 at 12:34 -0500, Eric Just wrote:
>> > Hi there,
>> >
>> > I was wondering if it was at all possible to get an ID linked into the
>> > feature table using the dbxref_id field in the feature table when loading
>> > data with the bulk loader. It would be nice to be able to optionally use
>> > the ID tag to populate a features primary dbxref. Alternatively, one 
>> could
>> > specify a particular dbxref that could get loaded into this field or to
>> > Does this sound like a reasonable request? Our database and middleware
>> > makes use of this field -- I am trying to tweak our middleware so that it
>> > works on data loaded with the bulk loader and I have a workaround but I
>> > thought that I'd ask first.
>> >
>> > Thanks,
>> >
>> > Eric
>> >
>> > ============================================
>> >
>> > Eric Just
>> > e-...@no...
>> > dictyBase Programmer
>> > Center for Genetic Medicine
>> > Northwestern University
>> > http://dictybase.org
>> >
>> > ============================================
>> >
>> >
>> >
>> >
>> > -------------------------------------------------------
>> > Using Tomcat but need to do more? Need to support web services, security?
>> > Get stuff done quickly with pre-integrated technology to make your job 
>> easier
>> > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
>> > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>> > _______________________________________________
>> > Gmod-devel mailing list
>> > Gmo...@li...
>> > https://lists.sourceforge.net/lists/listinfo/gmod-devel
>>--
>>------------------------------------------------------------------------
>>Scott Cain, Ph. D. ca...@cs...
>>GMOD Coordinator (http://www.gmod.org/) 216-392-3087
>>Cold Spring Harbor Laboratory
>>
>>
>>
>>-------------------------------------------------------
>>Using Tomcat but need to do more? Need to support web services, security?
>>Get stuff done quickly with pre-integrated technology to make your job easier
>>Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
>>http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>>_______________________________________________
>>Gmod-schema mailing list
>>Gmo...@li...
>>https://lists.sourceforge.net/lists/listinfo/gmod-schema
>
>============================================
>
>Eric Just
>e-...@no...
>dictyBase Programmer
>Center for Genetic Medicine
>Northwestern University
>http://dictybase.org
>
>============================================
>
>
>
>-------------------------------------------------------
>Using Tomcat but need to do more? Need to support web services, security?
>Get stuff done quickly with pre-integrated technology to make your job easier
>Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
>http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>_______________________________________________
>Gmod-schema mailing list
>Gmo...@li...
>https://lists.sourceforge.net/lists/listinfo/gmod-schema
============================================
Eric Just
e-...@no...
dictyBase Programmer
Center for Genetic Medicine
Northwestern University
http://dictybase.org
============================================ 

Mohammad,
No, the --noexon option is for GFF3 files that have both CDS and exon
features in them (so that redundant exons are not created). The problem
with loading CDS features the way I did before is that they were not
really being loaded as CDS features but as 'CDS_parts', which isn't a SO
term. Converting the CDSes to exons and polypeptides ensures SO
compliance.
Scott
On Thu, 2006年05月18日 at 10:33 -0400, Mohammad Fallahi wrote:
> Hi Scott,
> 
> Thanks a lot for your help. The data we are trying to load into the chado
> database includes only the CDS_predicted and not exons. We are using below
> algorithms:
> 
> 1. FGENESH
> 2. GENSCAN
> 3. GLIMMER
> 
> Which none of them includes utrs. Now I'm just wondering, in our case,
> whether using --noexons will be sufficient to load the CDS data properly or
> I need to use the old loading script?
> 
> Thanks,
> Mohammad,
> 
> 
> On 5/17/06 11:06 PM, "Scott Cain" <ca...@cs...> wrote:
> 
> > Hi Mohammad,
> > 
> > I am cc'ing this to the schema mailing list since this is a topic that
> > is likely to be covered more than once in the near future...
> > 
> > You certainly are right up on the bleeding edge! What you've seen is
> > actually the desired behavior for chado. CDS features are converted to
> > exons and a polypeptide feature (you can recover the CDS feature by
> > intersecting the polypeptide (which indicates where start and stop of
> > translation occur) and the exons). I also provide a --noexon option in
> > the loader if your GFF file has both CDS and exon features, and then it
> > only uses the CDS features to determine the bounds of the polypeptide
> > feature.
> > 
> > So here's the bleeding part: I am actively working on this loading code
> > and the associated GBrowse adaptor. Also needed but not yet begun is a
> > database function to return a set of CDS features for a transcript
> > (where it does the intersection as described above). An additional
> > outstanding problem is that I need to also deal with converting 5' and
> > 3' utrs.
> > 
> > For loading the output of a gene predictor, I don't think there will be
> > much of a problem.
> > 
> > Scott
> > 
> > On Wed, 2006年05月17日 at 20:06 -0400, Mohammad Fallahi wrote:
> >> Hi Scott,
> >> 
> >> I really appreciate it if you could answer the below question:
> >> 
> >> I've installed the latest gmod from cvs and also I'm using your new
> >> gmod_new_bulk_... Loader. After loading our CDS_predicted features into the
> >> chado database, these features are being recognized as exons (inferred from
> >> the CDS feature line in Gbrowse!). However, you know that there is
> >> difference between CDS and exons and I'm just wondering why this is
> >> happening in the latest gmod/loader combo?
> >> 
> >> Thanks,
> >> Mohammad,
> >> 
> >> ------------------------------------------------------------------------
> >> 
> >> Mohammad Fallahi (M.Sc.)
> >> Scientific Software Engineer
> >> Department of Informatics
> >> 
> >> Scripps Florida (The Scripps Research Institute)
> >> 5353 Parkside Drive, RF-A
> >> Jupiter, FL 33458
> >> 
> >> Telephone: 561-799-8919
> >> Fax:561-799-8952
> >> Email: mfa...@sc...
> >> 
> >> 
> 
> ------------------------------------------------------------------------
> 
> Mohammad Fallahi (M.Sc.)
> Scientific Software Engineer
> Department of Informatics
> 
> Scripps Florida (The Scripps Research Institute)
> 5353 Parkside Drive, RF-A
> Jupiter, FL 33458
> 
> Telephone: 561-799-8919
> Fax:561-799-8952
> Email: mfa...@sc...
> 
> 
-- 
------------------------------------------------------------------------
Scott Cain, Ph. D. ca...@cs...
GMOD Coordinator (http://www.gmod.org/) 216-392-3087
Cold Spring Harbor Laboratory

Hi Mohammad,
I am cc'ing this to the schema mailing list since this is a topic that
is likely to be covered more than once in the near future...
You certainly are right up on the bleeding edge! What you've seen is
actually the desired behavior for chado. CDS features are converted to
exons and a polypeptide feature (you can recover the CDS feature by
intersecting the polypeptide (which indicates where start and stop of
translation occur) and the exons). I also provide a --noexon option in
the loader if your GFF file has both CDS and exon features, and then it
only uses the CDS features to determine the bounds of the polypeptide
feature.
So here's the bleeding part: I am actively working on this loading code
and the associated GBrowse adaptor. Also needed but not yet begun is a
database function to return a set of CDS features for a transcript
(where it does the intersection as described above). An additional
outstanding problem is that I need to also deal with converting 5' and
3' utrs.
For loading the output of a gene predictor, I don't think there will be
much of a problem.
Scott
On Wed, 2006年05月17日 at 20:06 -0400, Mohammad Fallahi wrote:
> Hi Scott,
> 
> I really appreciate it if you could answer the below question:
> 
> I've installed the latest gmod from cvs and also I'm using your new
> gmod_new_bulk_... Loader. After loading our CDS_predicted features into the
> chado database, these features are being recognized as exons (inferred from
> the CDS feature line in Gbrowse!). However, you know that there is
> difference between CDS and exons and I'm just wondering why this is
> happening in the latest gmod/loader combo?
> 
> Thanks,
> Mohammad,
> 
> ------------------------------------------------------------------------
> 
> Mohammad Fallahi (M.Sc.)
> Scientific Software Engineer
> Department of Informatics
> 
> Scripps Florida (The Scripps Research Institute)
> 5353 Parkside Drive, RF-A
> Jupiter, FL 33458
> 
> Telephone: 561-799-8919
> Fax:561-799-8952
> Email: mfa...@sc...
> 
> 
-- 
------------------------------------------------------------------------
Scott Cain, Ph. D. ca...@cs...
GMOD Coordinator (http://www.gmod.org/) 216-392-3087
Cold Spring Harbor Laboratory

Hi Scott,
We have a version that works with the upcoming release. It will have to be 
released separately because the names of the Class::DBI classes have 
changed (i.e. Chado::Feature is now Chado::DBI::Chado::Feature). I was 
planning on releasing version 0.2 (compatible with GMOD version 0.01) 
around the same time as you release GMOD. I was just wary of releasing 
software based on a developer branch in CVS. Once I get our CVS repository 
back to normal (I still have to commit my changes since the blackout and 
merge into the main trunk), I can distribute instructions on how to get a 
version that works with the latest GMOD through CVS.
Eric
At 10:53 AM 5/17/2006, Scott Cain wrote:
>Hi Eric,
>
>Do you have any guesses on how this will work with the upcoming 0.01
>release? I'm guessing it relies on the Class::DBI classes, so
>presumably it will probably work fine, right?
>
>Scott
>
>
>On Tue, 2006年05月16日 at 14:05 -0500, Eric Just wrote:
> > Hi Everyone,
> >
> > We are announcing a new Sourceforge Project called Modware that may be of
> > interest to you. It is an object-oriented API written in Perl that
> > creates in-memory object representations of biological features stored 
> in a
> > Chado database.
> >
> > For download please visit our Sourceforge project page:
> > http://sourceforge.net/projects/gmod-ware
> >
> > For API documentation and some short examples of selected use cases visit
> > our project home page:
> > http://gmod-ware.sourceforge.net/
> >
> > This software is adapted from the production middleware code that 
> dictyBase
> > uses. Modware 0.1 requires the latest stable GMOD release: 0.003 be
> > installed. We are currently calling it a release candidate and if we get
> > some feedback will call it an official release if there are no major
> > install bugs (we've installed it only on two different machines). If you
> > would like a version that works on the latest CVS version of GMOD, let me
> > know and I'll expedite getting that out the door.
> >
> > Lastly, please use the direct download version, we have not fully 
> recovered
> > from the recent Sourceforge CVS issues.
> >
> > Please try the software out and let us know what you think!
> >
> >
> > Sincerely,
> > Eric Just and Sohel Merchant
> >
> > e-just at northwestern.edu
> > s-merchant at northwestern.edu
> >
> >
> > ============================================
> >
> > Eric Just
> > e-...@no...
> > dictyBase Programmer
> > Center for Genetic Medicine
> > Northwestern University
> > http://dictybase.org
> >
> > ============================================
> >
> >
> >
> >
> > -------------------------------------------------------
> > Using Tomcat but need to do more? Need to support web services, security?
> > Get stuff done quickly with pre-integrated technology to make your job 
> easier
> > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> > _______________________________________________
> > Gmod-announce mailing list
> > Gmo...@li...
> > https://lists.sourceforge.net/lists/listinfo/gmod-announce
>--
>------------------------------------------------------------------------
>Scott Cain, Ph. D. ca...@cs...
>GMOD Coordinator (http://www.gmod.org/) 216-392-3087
>Cold Spring Harbor Laboratory
>
>
>
>-------------------------------------------------------
>Using Tomcat but need to do more? Need to support web services, security?
>Get stuff done quickly with pre-integrated technology to make your job easier
>Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
>http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>_______________________________________________
>Gmod-architecture mailing list
>Gmo...@li...
>https://lists.sourceforge.net/lists/listinfo/gmod-architecture
============================================
Eric Just
e-...@no...
dictyBase Programmer
Center for Genetic Medicine
Northwestern University
http://dictybase.org
============================================ 

Hi Eric,
Do you have any guesses on how this will work with the upcoming 0.01
release? I'm guessing it relies on the Class::DBI classes, so
presumably it will probably work fine, right?
Scott
On Tue, 2006年05月16日 at 14:05 -0500, Eric Just wrote:
> Hi Everyone,
> 
> We are announcing a new Sourceforge Project called Modware that may be of 
> interest to you. It is an object-oriented API written in Perl that 
> creates in-memory object representations of biological features stored in a 
> Chado database.
> 
> For download please visit our Sourceforge project page:
> http://sourceforge.net/projects/gmod-ware
> 
> For API documentation and some short examples of selected use cases visit 
> our project home page:
> http://gmod-ware.sourceforge.net/
> 
> This software is adapted from the production middleware code that dictyBase 
> uses. Modware 0.1 requires the latest stable GMOD release: 0.003 be 
> installed. We are currently calling it a release candidate and if we get 
> some feedback will call it an official release if there are no major 
> install bugs (we've installed it only on two different machines). If you 
> would like a version that works on the latest CVS version of GMOD, let me 
> know and I'll expedite getting that out the door.
> 
> Lastly, please use the direct download version, we have not fully recovered 
> from the recent Sourceforge CVS issues.
> 
> Please try the software out and let us know what you think!
> 
> 
> Sincerely,
> Eric Just and Sohel Merchant
> 
> e-just at northwestern.edu
> s-merchant at northwestern.edu
> 
> 
> ============================================
> 
> Eric Just
> e-...@no...
> dictyBase Programmer
> Center for Genetic Medicine
> Northwestern University
> http://dictybase.org
> 
> ============================================ 
> 
> 
> 
> 
> -------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> Gmod-announce mailing list
> Gmo...@li...
> https://lists.sourceforge.net/lists/listinfo/gmod-announce
-- 
------------------------------------------------------------------------
Scott Cain, Ph. D. ca...@cs...
GMOD Coordinator (http://www.gmod.org/) 216-392-3087
Cold Spring Harbor Laboratory

Hi Everyone,
We are announcing a new Sourceforge Project called Modware that may be of 
interest to you. It is an object-oriented API written in Perl that 
creates in-memory object representations of biological features stored in a 
Chado database.
For download please visit our Sourceforge project page:
http://sourceforge.net/projects/gmod-ware
For API documentation and some short examples of selected use cases visit 
our project home page:
http://gmod-ware.sourceforge.net/
This software is adapted from the production middleware code that dictyBase 
uses. Modware 0.1 requires the latest stable GMOD release: 0.003 be 
installed. We are currently calling it a release candidate and if we get 
some feedback will call it an official release if there are no major 
install bugs (we've installed it only on two different machines). If you 
would like a version that works on the latest CVS version of GMOD, let me 
know and I'll expedite getting that out the door.
Lastly, please use the direct download version, we have not fully recovered 
from the recent Sourceforge CVS issues.
Please try the software out and let us know what you think!
Sincerely,
Eric Just and Sohel Merchant
e-just at northwestern.edu
s-merchant at northwestern.edu
============================================
Eric Just
e-...@no...
dictyBase Programmer
Center for Genetic Medicine
Northwestern University
http://dictybase.org
============================================ 

Hello again,
I was waiting for an email from SF admin (because they said they would
send one), but it appears that the gmod cvs server is back up. You
still need to fix an checked out repositories as I indicated below.
Scott
On Fri, 2006年05月12日 at 11:51 -0400, Scott Cain wrote:
> Hello,
> 
> As many of you may have noticed, the GMOD SourceForge cvs server has
> been inaccessible for several days. The folks at SF indicate that the
> new server will be available later today, but to use the repository, you
> will need to make changes to any local repository you have. The name of
> the server has changed; it was cvs.sourceforge.net but now will be
> gmod.cvs.sourceforge.net. To fix this, you can either blow away your
> old repository and check out a new one, or you should be able to execute
> this find command in your local repository:
> 
> find . -name "Root" -exec perl -pi -e 's/\@cvs\.sourceforge/\@gmod.cvs.sourceforge/' '{}' ';'
> 
> I'll send out another email later today when I've confirmed that the cvs
> server is back up and this find command fixes my local repository.
> 
> Scott
> 
-- 
------------------------------------------------------------------------
Scott Cain, Ph. D. ca...@cs...
GMOD Coordinator (http://www.gmod.org/) 216-392-3087
Cold Spring Harbor Laboratory

Hi Mohammad,
I cc'ed the schema mailing list as this is a question of general
interest.
There are several things you can do to make process more lines of GFF.
First off, what version of gmod/chado are you using? If you are using
0.003, it might be better to update to the current cvs development; I am
approaching a 0.01 release which makes lots of things better, including
loading GFF3 data. If you have cvs HEAD, you will find
chado/load/bin/new_bulk_load_gff3.PLS which you can convert to a
runnable script by executing `perl new_bulk_load_gff3.PLS` which will
create chado/load/bin/gmod_new_bulk_load_gff3.pl. (Shortly I am going
to be replacing the existing bulk loader with this one and have the
Makefile install it, but it is quite usable at this point.)
Options you can use to make it run longer:
 --notransact Since Postgres is trying to load all of these tables in
one very big transaction, that is where it is running out of memory. By
specifying notransact, each of the tables will load as individual
transactions (less safe but requires less memory). (Though I doubt that
this would help with a file so large anyway.)
 --inserts Instead of creating COPY FROM STDIN files, the bulk
loader will create individual INSERT statements for each piece of data.
While this generally takes much longer to insert, it will pretty much
always work.
Finally, you might want to consider breaking your file up into smaller
pieces. When I load Gramene data, I have a short perl script that will
break up the file into manageable chunks that are logically related:
known genes, gene predictions, and various types of similarity analyses.
Scott
On Fri, 2006年05月12日 at 13:41 -0400, Mohammad Fallahi wrote:
> Hi Scott,
> 
> I'm having a memory problem after trying to load a large gff3 file (4GB)
> which includes 4.6 million features (with no sequence) into the Chado
> database.
> 
> I got an error message after a very long time:
> --------------
> 
> Preparing data for inserting into the C1 database
> (This may take a while ...)
> *** malloc: vm_allocate(size=8421376) failed (error code=3)
> *** malloc[17281]: error: Can't allocate region
> Out of memory!
> 
> -----------------
> 
> I really appreciate it if you could share your experience in resolving these
> kind of problems.
> 
> Thanks a lot,
> Mohammad,
> 
> 
> 
> ------------------------------------------------------------------------
> 
> Mohammad Fallahi (M.Sc.)
> Scientific Software Engineer
> Department of Informatics
> 
> Scripps Florida (The Scripps Research Institute)
> 5353 Parkside Drive, RF-A
> Jupiter, FL 33458
> 
> Telephone: 561-799-8919
> Fax:561-799-8952
> Email: mfa...@sc...
> 
> 
-- 
------------------------------------------------------------------------
Scott Cain, Ph. D. ca...@cs...
GMOD Coordinator (http://www.gmod.org/) 216-392-3087
Cold Spring Harbor Laboratory

Thank you very much for you help. I got the gene information for one species (e.g., Dmel or Dpse).
Now I want to get all orthologues between Dmel and Dpse. How can I do that query? For example (eve gene), the result like this:
FBgn0000606 eve FBgn0020474 Dpse\eve
It seems the postgresql dump file "chado_dmel_r4_3_16" also includes the annotations from other species. I downloaded the postgresql dump file "chado_dmel_r42_dpse_r20_1_reporting" too, this file also has the information of Dmel. So what is the difference except the version? Or where can I find the document about this?
Chris, which CVS do you mean? Could you give me a link?
Long
chris mungall <cj...@fr...> wrote: 
Thanks Pinglei
I have added a view 'protein_coding_gene' to implicit-feature- 
views.sql. This should be included in the db build as standard in the 
next chado release - for now you can get it from cvs (hmm, as soon as 
sourceforge developer cvs is working again and I can commit it!)
Once loaded, you should just be able to do things like
 SELECT count(*) FROM protein_coding_gene;
(the fact that this view name is identical to the SO type of the same 
name is not coincidental!)
This view has a dependency on the sofa layer (currently in cvs but 
again will be built by default in the next release). The sofa layer 
provides a view for every type in SOFA.
If anyone else has common queries they're not quite sure how to do - 
please, send them to the list!
And anyone who has a standard set of SQL they use - please, add them 
as views! This is by far the most robust way to do this kind of 
thing, you insulate yourself from changes in coding practices, and 
you give yourself the option of materializing the views for speed 
later on.
Cheers
Chris
On May 11, 2006, at 6:54 AM, Pinglei Zhou wrote:
>
>>
>> I downloaded the chado file "chado_dmel_r4_3_16" from flybase and 
>> restored it to my local computer. By
> running the command:
>>
>> chado_dmel_r4_3_16=> select feature_id, uniquename, name from 
>> feature where organism_id=1 and type_id=219;
>>
>> I can get all of genes (organism_id=1 is dmel, type_id=219 is the 
>> "gene" in SO). But my questions is: how
> can I get all of genes which are protein coding genes? I mean is 
> there a field in the tables of feature
> module in Chado to indicate the gene is a protein coding gene, or 
> is there an indirect subquery to do this?
>
> All coding and non-coding genes have type as gene, it will define 
> it on transcript leverl, for all coding
> genes, its transcript will have type as 'mRNA', non-coding will 
> have other types, such as 'pseudogene',
> 'tRNA'....
>
> this query should return all coding genes:
> select g.uniquename, g.name from feature g, feature t, 
> feature_relationship fr, cvterm cg, cvterm ct where
> g.feature_id=fr.object_id and g.type_id=cg.cvterm_id and 
> cg.name='gene' and fr.subject_id=t.feature_id and
> t.type_id=ct.cvterm_id and ct.name='mRNA'
>
>
> pinglei
>
>
>
> -------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, 
> security?
> Get stuff done quickly with pre-integrated technology to make your 
> job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache 
> Geronimo
> http://sel.as-us.falkag.net/sel? 
> cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> Gmod-schema mailing list
> Gmo...@li...
> https://lists.sourceforge.net/lists/listinfo/gmod-schema
-------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Gmod-schema mailing list
Gmo...@li...
https://lists.sourceforge.net/lists/listinfo/gmod-schema
			
---------------------------------
Yahoo! Mail goes everywhere you do. Get it on your phone.

Hello,
As many of you may have noticed, the GMOD SourceForge cvs server has
been inaccessible for several days. The folks at SF indicate that the
new server will be available later today, but to use the repository, you
will need to make changes to any local repository you have. The name of
the server has changed; it was cvs.sourceforge.net but now will be
gmod.cvs.sourceforge.net. To fix this, you can either blow away your
old repository and check out a new one, or you should be able to execute
this find command in your local repository:
 find . -name "Root" -exec perl -pi -e 's/\@cvs\.sourceforge/\@gmod.cvs.sourceforge/' '{}' ';'
I'll send out another email later today when I've confirmed that the cvs
server is back up and this find command fixes my local repository.
Scott
-- 
------------------------------------------------------------------------
Scott Cain, Ph. D. ca...@cs...
GMOD Coordinator (http://www.gmod.org/) 216-392-3087
Cold Spring Harbor Laboratory

Hello,
I am pleased to announce that the semi-annual GMOD meeting will be held
a the National Evolutionary Synthesis Center <http://www.nescent.org/>
in Durham, NC on June 29 and 30. In addition to 'typical' topics for
discussion (new applications, progress on making applications
interoperate), we will discuss how existing applications can be used or
modified for evolutionary biologists and what applications could be
developed in conjunction with GMOD to facilitate evolutionary model
organism databases. I am also accepting suggestions for topics or
talks.
We've received a discount on hotel rooms at the nearby Millennium Hotel
Durham for 79ドル/night. The '6136' group code is used for this discount:
 https://reservations.synxis.com/LBE/rez.aspx?Hotel=11543&Chain=5303&lang=1&group=6136&arrive=06/28/2006&depart=06/30/2006&adult=1&child=0
Note that there is no registration fee for the meeting itself but there
is a deadline: you must register by June 20 and there is a limit of 50
people.
The registration page is at http://www.gmod.org/register.html
Thanks and I look forward to seeing you in Durham!
Scott
-- 
------------------------------------------------------------------------
Scott Cain, Ph. D. ca...@cs...
GMOD Coordinator (http://www.gmod.org/) 216-392-3087
Cold Spring Harbor Laboratory

Infinex Ventures Inc. (INFX)
Current Price: 0.79
The Rally has begun
Watch this one like a hawk, this report is sent because the potential is 
incredible
H U G E N E W S read below
S T R O N G B U Y 
COMPANY OVERVIEW
Aggressive and energetic, Infinex boasts a dynamic and diversified 
portfolio of operations across North America, with an eye on 
international expansion. 
Grounded in natural resource exploration, Inifinex also offers investors 
access to exciting new developments in the high-tech sector and the 
booming international real estate market. Our market based experience, 
tenacious research techniques, and razor sharp analytical skills allow 
us to leverage opportunities in emerging markets and developing 
technologies. 
Identifying these opportunities in the earliest stages allows us to 
accelerate business development and fully realize the company.s true 
potential. Maximizing overall profitability and in turn enhancing 
shareholder value. 
Current Press Release
Infinex Announces Extension to Its Agreement in Chile
LAS VEGAS, NV, May 9 /PRNewswire-FirstCall/ - Infinex Ventures Inc. 
(INFX:OB - News; "the Company") and its Board of Directors are pleased 
to announce that the Company has received an extension (90 days) to its 
Agreement for the due diligence period, in an effort to fully verify the 
offered title and all additional documentation, including but not 
limited to, Trial C-1912- 2001 at the 14th Civil Court of Santiago and 
Criminal Trial 1160-2002 at the 19th Court of Crime of Santiago of 
Chile, Ministry of Mines of Chile over its sole and exclusive right to 
acquire a 50% interest in the Tesoro 1-12 Mining Claims.
Infinex Announces Joint Venture and Option Agreement Extension
LAS VEGAS, May 5 /PRNewswire-FirstCall/ - Infinex Ventures Inc. (INFX:OB 
- "the Company") and its Board of Directors are please to announce that 
the Company has been granted an extension of 120 days to fulfill its 
contractual obligations under the Joint Venture and Option Agreement 
dated June 14, 2004 on the Texada Island "Yew Gr0up" Mining Claims:
This is as sure as it gets, GET IN NOW
experts Fatima employment's feathertop discriminates launderer activity Valparaiso acids imputed contrivances germ USAF fray acknowledgments execute Moliere demitting becloud inconvertible inferences adulthood frugally alienating fruition edicts amounters enshroud displeasing McCracken durability

Thanks Pinglei
I have added a view 'protein_coding_gene' to implicit-feature- 
views.sql. This should be included in the db build as standard in the 
next chado release - for now you can get it from cvs (hmm, as soon as 
sourceforge developer cvs is working again and I can commit it!)
Once loaded, you should just be able to do things like
 SELECT count(*) FROM protein_coding_gene;
(the fact that this view name is identical to the SO type of the same 
name is not coincidental!)
This view has a dependency on the sofa layer (currently in cvs but 
again will be built by default in the next release). The sofa layer 
provides a view for every type in SOFA.
If anyone else has common queries they're not quite sure how to do - 
please, send them to the list!
And anyone who has a standard set of SQL they use - please, add them 
as views! This is by far the most robust way to do this kind of 
thing, you insulate yourself from changes in coding practices, and 
you give yourself the option of materializing the views for speed 
later on.
Cheers
Chris
On May 11, 2006, at 6:54 AM, Pinglei Zhou wrote:
>
>>
>> I downloaded the chado file "chado_dmel_r4_3_16" from flybase and 
>> restored it to my local computer. By
> running the command:
>>
>> chado_dmel_r4_3_16=> select feature_id, uniquename, name from 
>> feature where organism_id=1 and type_id=219;
>>
>> I can get all of genes (organism_id=1 is dmel, type_id=219 is the 
>> "gene" in SO). But my questions is: how
> can I get all of genes which are protein coding genes? I mean is 
> there a field in the tables of feature
> module in Chado to indicate the gene is a protein coding gene, or 
> is there an indirect subquery to do this?
>
> All coding and non-coding genes have type as gene, it will define 
> it on transcript leverl, for all coding
> genes, its transcript will have type as 'mRNA', non-coding will 
> have other types, such as 'pseudogene',
> 'tRNA'....
>
> this query should return all coding genes:
> select g.uniquename, g.name from feature g, feature t, 
> feature_relationship fr, cvterm cg, cvterm ct where
> g.feature_id=fr.object_id and g.type_id=cg.cvterm_id and 
> cg.name='gene' and fr.subject_id=t.feature_id and
> t.type_id=ct.cvterm_id and ct.name='mRNA'
>
>
> pinglei
>
>
>
> -------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, 
> security?
> Get stuff done quickly with pre-integrated technology to make your 
> job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache 
> Geronimo
> http://sel.as-us.falkag.net/sel? 
> cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> Gmod-schema mailing list
> Gmo...@li...
> https://lists.sourceforge.net/lists/listinfo/gmod-schema

Hi Long,
 
>There is a section about the gene identifier in Flybase 
(http://www.flybase.org/annot/dmel-release4-notes.html#id):
>
>In Release 3.0 and 3.1, protein-coding genes were given 'CG' identifiers of the form CGnnnnn. 
For non-protein-coding genes, such as tRNAs, snRNAs, snoRNAs, microRNAs, miscellaneous 
non-coding RNAs, and pseudogenes, 'CR' idenifiers of the form CRnnnnn were assigned.
>
>It seems that from 4.3, flybase does NOT use ID=CGnnnnn or ID=CRnnnnn form in GFF3 files, 
instead using ID=FBgnxxxxxxx. 
>
>In Chado, field "uniquename" in table "feauture" of sequence module is used to keep the 
FBgnxxxxxxx, field "name" to keep the common accepted name (CG/CR if there is no common accepted 
name) for that gene. My question is "is there a field to keep the CG/CR indentifiers in 
Chado"? If yes, the query of protein coding genes will be easier.
You are right for our uniquename change for some historic reason. CG/CR is used for sequence 
gene model, while FBgnnnnnnn is unique identifier for genetic object which may or may not have 
sequence gene model associated with it. We are in the process of integrating those two database 
into one (will be in our release5.1, assume be out to public this summer). The first step is to 
change feature.uniquename from CG/CR to FBgnnnnnnnn, and store those CG/CR in dbxref(still CG 
for coding and CR for non-coding gene). so another way to see if coding gene is: 
select g.uniquename, g.name, accession from feature g, feature_dbxref fd, dbxref d, cvterm c 
where g.type_id=c.cvterm_id and c.name='gene' and g.feature_id=fd.feature_id and 
fd.dbxref_id=d.dbxref_id and fd.is_current='true' and (d.accession like 'CG_____' or d.accession 
like 'CG____') ;
pinglei
>Pinglei Zhou <zh...@mo...> wrote: 
>>
>All coding and non-coding genes have type as gene, it will define it on transcript leverl, for 
all coding 
>genes, its transcript will have type as 'mRNA', non-coding will have other types, such as 
'pseudogene', 
>'tRNA'....
>
>this query should return all coding genes:
>select g.uniquename, g.name from feature g, feature t, feature_relationship fr, cvterm cg, 
cvterm ct where 
>g.feature_id=fr.object_id and g.type_id=cg.cvterm_id and cg.name='gene' and 
fr.subject_id=t.feature_id and 
>t.type_id=ct.cvterm_id and ct.name='mRNA'
>
>
>pinglei
>
>
>
>		
>---------------------------------
>Get amazing travel prices for air and hotel in one click on Yahoo! FareChase 

Thanks a lot, pinglei.
There is a section about the gene identifier in Flybase (http://www.flybase.org/annot/dmel-release4-notes.html#id):
In Release 3.0 and 3.1, protein-coding genes were given 'CG' identifiers of the form CGnnnnn. For non-protein-coding genes, such as tRNAs, snRNAs, snoRNAs, microRNAs, miscellaneous non-coding RNAs, and pseudogenes, 'CR' idenifiers of the form CRnnnnn were assigned.
It seems that from 4.3, flybase does NOT use ID=CGnnnnn or ID=CRnnnnn form in GFF3 files, instead using ID=FBgnxxxxxxx. 
In Chado, field "uniquename" in table "feauture" of sequence module is used to keep the FBgnxxxxxxx, field "name" to keep the common accepted name (CG/CR if there is no common accepted name) for that gene. My question is "is there a field to keep the CG/CR indentifiers in Chado"? If yes, the query of protein coding genes will be easier.
Long
Pinglei Zhou <zh...@mo...> wrote: 
>
All coding and non-coding genes have type as gene, it will define it on transcript leverl, for all coding 
genes, its transcript will have type as 'mRNA', non-coding will have other types, such as 'pseudogene', 
'tRNA'....
this query should return all coding genes:
select g.uniquename, g.name from feature g, feature t, feature_relationship fr, cvterm cg, cvterm ct where 
g.feature_id=fr.object_id and g.type_id=cg.cvterm_id and cg.name='gene' and fr.subject_id=t.feature_id and 
t.type_id=ct.cvterm_id and ct.name='mRNA'
pinglei
		
---------------------------------
Get amazing travel prices for air and hotel in one click on Yahoo! FareChase 

 
>
>I downloaded the chado file "chado_dmel_r4_3_16" from flybase and restored it to my local computer. By 
running the command:
>
>chado_dmel_r4_3_16=> select feature_id, uniquename, name from feature where organism_id=1 and type_id=219;
>
>I can get all of genes (organism_id=1 is dmel, type_id=219 is the "gene" in SO). But my questions is: how 
can I get all of genes which are protein coding genes? I mean is there a field in the tables of feature 
module in Chado to indicate the gene is a protein coding gene, or is there an indirect subquery to do this?
All coding and non-coding genes have type as gene, it will define it on transcript leverl, for all coding 
genes, its transcript will have type as 'mRNA', non-coding will have other types, such as 'pseudogene', 
'tRNA'....
this query should return all coding genes:
select g.uniquename, g.name from feature g, feature t, feature_relationship fr, cvterm cg, cvterm ct where 
g.feature_id=fr.object_id and g.type_id=cg.cvterm_id and cg.name='gene' and fr.subject_id=t.feature_id and 
t.type_id=ct.cvterm_id and ct.name='mRNA'
pinglei

Hi,
I downloaded the chado file "chado_dmel_r4_3_16" from flybase and restored it to my local computer. By running the command:
chado_dmel_r4_3_16=> select feature_id, uniquename, name from feature where organism_id=1 and type_id=219;
I can get all of genes (organism_id=1 is dmel, type_id=219 is the "gene" in SO). But my questions is: how can I get all of genes which are protein coding genes? I mean is there a field in the tables of feature module in Chado to indicate the gene is a protein coding gene, or is there an indirect subquery to do this?
Long
		
---------------------------------
Blab-away for as little as 1¢/min. Make PC-to-Phone Calls using Yahoo! Messenger with Voice.

Hi colleagues,
The Generation Challenge Programme here is working on wrapping Chado
using Hibernate and connecting it to our scientific domain model. I've
asked the pertinent members of my team here to pay attention to this
effort at the Sanger.
Richard
-----Original Message-----
From: gmo...@li...
[mailto:gmo...@li...] On Behalf Of
gmo...@li...
Sent: Wednesday, May 10, 2006 11:08 AM
To: gmo...@li...
Subject: Gmod-schema digest, Vol 1 #708 - 1 msg
Send Gmod-schema mailing list submissions to
	gmo...@li...
To subscribe or unsubscribe via the World Wide Web, visit
	https://lists.sourceforge.net/lists/listinfo/gmod-schema
or, via email, send a message with subject or body 'help' to
	gmo...@li...
You can reach the person managing the list at
	gmo...@li...
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Gmod-schema digest..."
Today's Topics:
 1. Re: Chado data access API (Scott Cain)
--__--__--
Message: 1
Subject: Re: [Gmod-schema] Chado data access API
From: Scott Cain <ca...@cs...>
To: chris mungall <cj...@fr...>
Cc: Tim Carver <tj...@sa...>,
 gmod schema <gmo...@li...>
Date: 2006年5月09日 10:07:26 -0400
Hi Tim,
That is very good--I'm glad to hear it. Would you also consider sending
an announcement to gmo...@li... as well?
Thanks,
Scott
On Mon, 2006年05月08日 at 19:10 -0700, chris mungall wrote:
> This is great Tim!
>=20
> you may want to post this on gmod-architecture too (though I suspect =20
> everyone there may be on here)
>=20
> On May 8, 2006, at 8:53 AM, Tim Carver wrote:
>=20
> >
> > This is just to let people know that we have been developing a Java
> > data
> > access API. Currently it can use either JDBC or iBatis. We have =20
> > Artemis now
> > reading from and writing to a chado database using this API and we =20
> > intend to
> > develop this further for artemis and our other software tools. It =20
> > still is
> > in development but is available from the Artemis CVS (see
> > http://www.sanger.ac.uk/Software/Artemis/).
> >
> > It would be interesting to hear if other people have a generic Java
> > API's or
> > if you would like more details on this.
> >
> > Thanks
> > Tim
> >
> >
> >
> >
> > -------------------------------------------------------
> > Using Tomcat but need to do more? Need to support web services, =20
> > security?
> > Get stuff done quickly with pre-integrated technology to make your =20
> > job easier
> > Download IBM WebSphere Application Server v.1.0.1 based on Apache =20
> > Geronimo
> > http://sel.as-us.falkag.net/sel?=20
> > cmd=3Dlnk&kid=3D120709&bid=3D263057&dat=3D121642
> > _______________________________________________
> > Gmod-schema mailing list
> > Gmo...@li...
> > https://lists.sourceforge.net/lists/listinfo/gmod-schema
>=20
>=20
>=20
> -------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services,
security?
> Get stuff done quickly with pre-integrated technology to make your job
easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache
Geronimo
>
http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D120709&bid=3D263057&dat=3D=
121642
> _______________________________________________
> Gmod-schema mailing list
> Gmo...@li...
> https://lists.sourceforge.net/lists/listinfo/gmod-schema
--=20
------------------------------------------------------------------------
Scott Cain, Ph. D. ca...@cs...
GMOD Coordinator (http://www.gmod.org/) 216-392-3087
Cold Spring Harbor Laboratory
--__--__--
_______________________________________________
Gmod-schema mailing list
Gmo...@li...
https://lists.sourceforge.net/lists/listinfo/gmod-schema
End of Gmod-schema Digest

Hi Tim,
That is very good--I'm glad to hear it. Would you also consider sending
an announcement to gmo...@li... as well?
Thanks,
Scott
On Mon, 2006年05月08日 at 19:10 -0700, chris mungall wrote:
> This is great Tim!
> 
> you may want to post this on gmod-architecture too (though I suspect 
> everyone there may be on here)
> 
> On May 8, 2006, at 8:53 AM, Tim Carver wrote:
> 
> >
> > This is just to let people know that we have been developing a Java 
> > data
> > access API. Currently it can use either JDBC or iBatis. We have 
> > Artemis now
> > reading from and writing to a chado database using this API and we 
> > intend to
> > develop this further for artemis and our other software tools. It 
> > still is
> > in development but is available from the Artemis CVS (see
> > http://www.sanger.ac.uk/Software/Artemis/).
> >
> > It would be interesting to hear if other people have a generic Java 
> > API's or
> > if you would like more details on this.
> >
> > Thanks
> > Tim
> >
> >
> >
> >
> > -------------------------------------------------------
> > Using Tomcat but need to do more? Need to support web services, 
> > security?
> > Get stuff done quickly with pre-integrated technology to make your 
> > job easier
> > Download IBM WebSphere Application Server v.1.0.1 based on Apache 
> > Geronimo
> > http://sel.as-us.falkag.net/sel? 
> > cmd=lnk&kid=120709&bid=263057&dat=121642
> > _______________________________________________
> > Gmod-schema mailing list
> > Gmo...@li...
> > https://lists.sourceforge.net/lists/listinfo/gmod-schema
> 
> 
> 
> -------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> Gmod-schema mailing list
> Gmo...@li...
> https://lists.sourceforge.net/lists/listinfo/gmod-schema
-- 
------------------------------------------------------------------------
Scott Cain, Ph. D. ca...@cs...
GMOD Coordinator (http://www.gmod.org/) 216-392-3087
Cold Spring Harbor Laboratory