Keyboard Shortcuts

File
u :up to issue
m :publish + mail comments
M :edit review message
j / k :jump to file after / before current file
J / K :jump to next file with a comment after / before current file
Side-by-side diff
i :toggle intra-line diffs
e :expand all comments
c :collapse all comments
s :toggle showing all comments
n / p :next / previous diff chunk or comment
N / P :next / previous comment
<Up> / <Down> :next / previous line
<Enter> :respond to / edit current comment
d :mark current comment as done
Issue
u :up to list of issues
m :publish + mail comments
j / k :jump to patch after / before current patch
o / <Enter> :open current patch in side-by-side view
i :open current patch in unified diff view
Issue List
j / k :jump to issue after / before current issue
o / <Enter> :open current issue
# : close issue
Comment/message editing
<Ctrl> + s or <Ctrl> + Enter :save comment
<Esc> :cancel edit
Rietveld Code Review Tool
Help | Bug tracker | Discussion group | Source code | Sign in
(488)
Issues Repositories Search
Open Issues | Closed Issues | All Issues | Sign in with your Google Account to create issues and add comments

Issue 7137054: Use import instead of subprocess for charm proof.

Can't Edit
Can't Publish+Mail
Start Review
Created:
12 years, 11 months ago by j.c.sackett
Modified:
12 years, 11 months ago
Reviewers:
curtis , mp+143790, abentley-home , hazmat
Visibility:
Public.
Use import instead of subprocess for charm proof. To speed up proof, we use the newly created proof lib in charm-tools instead of a call out to "charm proof" via subprocess. * Replace config.CHARM_PROOF_BIN with config.CHARM_PROOF_PATH, a path to the library's parent dir. * Added patch_path and unpatch_path, which (within a try...finally block) safely manage modifying the path during the job's `run` method. * Removed exception handling that required subprocess management, as we know longer have that; this sadly leaves us with only the "Exception" clause around the call to proof_charm, since we can't capture the data from the subprocess stream. * Updates the checks for a functioning binary with checks that the lib path exists. * Updated the error messages and tests. As a note; the previous version of this set CHARM_PROOF_PATH to $charm-tool-dir/script/lib, and imported proof. As the name of the job is proof, and both methods are "run" this led to a really fun bug. We now import lib.proof as prooflib, and point the CHARM_PROOF_PATH to the parent directory of the proof lib. This works just fine. https://code.launchpad.net/~jcsackett/charmworld/use-libified-proof/+merge/143790 (do not edit description out of merge proposal)

Patch Set 1 #

Patch Set 2 : Use import instead of subprocess for charm proof. #

Total comments: 6

Patch Set 3 : Use import instead of subprocess for charm proof. #

Total comments: 6

Patch Set 4 : Use import instead of subprocess for charm proof. #

Patch Set 5 : Use import instead of subprocess for charm proof. #

Patch Set 6 : Use import instead of subprocess for charm proof. #

Total comments: 1
Created: 12 years, 11 months ago
Download [raw] [tar.bz2]
Unified diffs Side-by-side diffs Delta from patch set Stats (+62 lines, -58 lines) Patch
A [revision details] View 1 2 3 4 5 1 chunk +2 lines, -0 lines 0 comments Download
M charmworld/jobs/config.py View 1 2 1 chunk +1 line, -1 line 0 comments Download
M charmworld/jobs/proof.py View 1 2 3 4 5 1 chunk +50 lines, -48 lines 1 comment Download
M charmworld/jobs/tests/test_proof.py View 1 2 3 1 chunk +9 lines, -9 lines 0 comments Download
Total messages: 16
|
j.c.sackett
Please take a look.
12 years, 11 months ago (2013年01月17日 22:19:24 UTC) #1
Please take a look.
Sign in to reply to this message.
j.c.sackett
Please take a look.
12 years, 11 months ago (2013年01月17日 22:20:09 UTC) #2
Please take a look.
Sign in to reply to this message.
hazmat
some concerns about the runtime sys.path manipulation as well as the use of globals in ...
12 years, 11 months ago (2013年01月18日 15:01:13 UTC) #3
some concerns about the runtime sys.path manipulation as well as the use of
globals in proof/lint.
https://codereview.appspot.com/7137054/diff/3001/charmworld/jobs/proof.py
File charmworld/jobs/proof.py (right):
https://codereview.appspot.com/7137054/diff/3001/charmworld/jobs/proof.py#new...
charmworld/jobs/proof.py:21: sys.path.append(config.CHARM_PROOF_PATH)
this should be done once per process not once per charm with manipulations, and
it should probably also verify that the proof isn't already on the path.
the use of globals here is quite unfortunate, can the librarizing of proof not
generate a return value and obviate the need for globals?
Sign in to reply to this message.
curtis
lgtm. I have a suggestion that may make the operation more robust. https://codereview.appspot.com/7137054/diff/3001/charmworld/jobs/proof.py File charmworld/jobs/proof.py ...
12 years, 11 months ago (2013年01月18日 15:01:24 UTC) #4
lgtm. I have a suggestion that may make the operation more robust.
https://codereview.appspot.com/7137054/diff/3001/charmworld/jobs/proof.py
File charmworld/jobs/proof.py (right):
https://codereview.appspot.com/7137054/diff/3001/charmworld/jobs/proof.py#new...
charmworld/jobs/proof.py:26: return lint
Can we place this in a try-finally block to ensure the path is sane?
Sign in to reply to this message.
hazmat
one more concern. https://codereview.appspot.com/7137054/diff/3001/charmworld/jobs/proof.py File charmworld/jobs/proof.py (right): https://codereview.appspot.com/7137054/diff/3001/charmworld/jobs/proof.py#newcode23 charmworld/jobs/proof.py:23: proof.run(charm['branch_dir']) also this needs a try/except ...
12 years, 11 months ago (2013年01月18日 15:02:20 UTC) #5
one more concern.
https://codereview.appspot.com/7137054/diff/3001/charmworld/jobs/proof.py
File charmworld/jobs/proof.py (right):
https://codereview.appspot.com/7137054/diff/3001/charmworld/jobs/proof.py#new...
charmworld/jobs/proof.py:23: proof.run(charm['branch_dir'])
also this needs a try/except check.. there are some charms that proof will just
blow up with error on.
Sign in to reply to this message.
j.c.sackett
https://codereview.appspot.com/7137054/diff/3001/charmworld/jobs/proof.py File charmworld/jobs/proof.py (right): https://codereview.appspot.com/7137054/diff/3001/charmworld/jobs/proof.py#newcode21 charmworld/jobs/proof.py:21: sys.path.append(config.CHARM_PROOF_PATH) On 2013年01月18日 15:01:13, hazmat wrote: > this should ...
12 years, 11 months ago (2013年01月18日 15:24:13 UTC) #6
https://codereview.appspot.com/7137054/diff/3001/charmworld/jobs/proof.py
File charmworld/jobs/proof.py (right):
https://codereview.appspot.com/7137054/diff/3001/charmworld/jobs/proof.py#new...
charmworld/jobs/proof.py:21: sys.path.append(config.CHARM_PROOF_PATH)
On 2013年01月18日 15:01:13, hazmat wrote:
> this should be done once per process not once per charm with manipulations,
and
> it should probably also verify that the proof isn't already on the path.
Dig.
> 
> the use of globals here is quite unfortunate, can the librarizing of proof not
> generate a return value and obviate the need for globals?
I suppose; however, the whole script was setup with globals in mind, and I
didn't want to spend more time on it. I suppose I could, but it will require a
good bit more rearchitecting.
https://codereview.appspot.com/7137054/diff/3001/charmworld/jobs/proof.py#new...
charmworld/jobs/proof.py:23: proof.run(charm['branch_dir'])
On 2013年01月18日 15:02:20, hazmat wrote:
> also this needs a try/except check.. there are some charms that proof will
just
> blow up with error on.
Ok; just a pass on except Exception, or is there a particular exception it
throws and handling we should do?
https://codereview.appspot.com/7137054/diff/3001/charmworld/jobs/proof.py#new...
charmworld/jobs/proof.py:26: return lint
On 2013年01月18日 15:01:24, curtis wrote:
> Can we place this in a try-finally block to ensure the path is sane?
We could, but if we're modifying the path manipulation to be per job run rather
than per proof, there's less churn in the first place.
Also, I don't think anything else that happens in this run should modify the
path, so as long as we're handling our own cleanup this should be safe, right?
Sign in to reply to this message.
j.c.sackett
Please take a look.
12 years, 11 months ago (2013年01月18日 22:35:20 UTC) #7
Please take a look.
Sign in to reply to this message.
hazmat
https://codereview.appspot.com/7137054/diff/5002/charmworld/jobs/proof.py File charmworld/jobs/proof.py (right): https://codereview.appspot.com/7137054/diff/5002/charmworld/jobs/proof.py#newcode42 charmworld/jobs/proof.py:42: try: in general this sort of usage lock/unlock is ...
12 years, 11 months ago (2013年01月22日 14:19:40 UTC) #8
https://codereview.appspot.com/7137054/diff/5002/charmworld/jobs/proof.py
File charmworld/jobs/proof.py (right):
https://codereview.appspot.com/7137054/diff/5002/charmworld/jobs/proof.py#new...
charmworld/jobs/proof.py:42: try:
in general this sort of usage lock/unlock is perfect for a context manager. the
exception catch here for a list append feels a bit over much vs just using a
boolean return value.
ideally this patch is also doing the import, so we can check/guard importerror
and report in the same block. else with the current structure, the whole queue
would be processed with errors even if we couldn't import the prooflib.
Sign in to reply to this message.
abentley-home
https://codereview.appspot.com/7137054/diff/5002/charmworld/jobs/proof.py File charmworld/jobs/proof.py (right): https://codereview.appspot.com/7137054/diff/5002/charmworld/jobs/proof.py#newcode15 charmworld/jobs/proof.py:15: raise OSError('[Errno 2] No such file or directory') Please ...
12 years, 11 months ago (2013年01月22日 15:08:40 UTC) #9
https://codereview.appspot.com/7137054/diff/5002/charmworld/jobs/proof.py
File charmworld/jobs/proof.py (right):
https://codereview.appspot.com/7137054/diff/5002/charmworld/jobs/proof.py#new...
charmworld/jobs/proof.py:15: raise OSError('[Errno 2] No such file or
directory')
Please use a different, perhaps custom exception. Saying "No such file" when
there may be a file is confusing at best.
https://codereview.appspot.com/7137054/diff/5002/charmworld/jobs/proof.py#new...
charmworld/jobs/proof.py:30: for l in lint:
Please use more descriptive variable names, e.g. "line".
Sign in to reply to this message.
j.c.sackett
I've addressed all comments, and done quite a bit of experimenting today to see if ...
12 years, 11 months ago (2013年01月22日 21:41:49 UTC) #10
I've addressed all comments, and done quite a bit of experimenting today to see
if there was a way to restructure this to use a contextmanager, as that has been
a persistent comment. In short, it cannot be done, but I directly address that
question in in-line reply.
https://codereview.appspot.com/7137054/diff/5002/charmworld/jobs/proof.py
File charmworld/jobs/proof.py (right):
https://codereview.appspot.com/7137054/diff/5002/charmworld/jobs/proof.py#new...
charmworld/jobs/proof.py:15: raise OSError('[Errno 2] No such file or
directory')
On 2013年01月22日 15:08:40, abentley wrote:
> Please use a different, perhaps custom exception. Saying "No such file" when
> there may be a file is confusing at best.
With your suggestion for moving unpatch to a separate try block, rather than the
previous try/except/else/finally, we can restructure this to simply return a
boolean rather than using the exception. I did still use your suggestion of
os.stat to ensure the path exists.
It should be noted, that I'm putting the unpatch call inside a try/finally,
rather than a try/except, as I don't want to handle silence errors that I don't
expect--I simply want to ensure unpatch is called regardless of the proofing
process.
https://codereview.appspot.com/7137054/diff/5002/charmworld/jobs/proof.py#new...
charmworld/jobs/proof.py:30: for l in lint:
On 2013年01月22日 15:08:40, abentley wrote:
> Please use more descriptive variable names, e.g. "line".
Done.
https://codereview.appspot.com/7137054/diff/5002/charmworld/jobs/proof.py#new...
charmworld/jobs/proof.py:42: try:
On 2013年01月22日 14:19:40, hazmat wrote:
> in general this sort of usage lock/unlock is perfect for a context manager.
the
> exception catch here for a list append feels a bit over much vs just using a
> boolean return value.
A try/finally is necessary to make certain we restore the path if we've altered
it; a context manager doesn't always execute the exit case in the case of an
unhandled exception.
> ideally this patch is also doing the import, so we can check/guard importerror
> and report in the same block. else with the current structure, the whole queue
> would be processed with errors even if we couldn't import the prooflib.
Because we can't safely restructure this with a context manager, we can't have
patch_path do the import, as the import is then lost on the return. However, we
can change it so that if patch fails, we cease execution of `run`, which ends
the proofing run.
Sign in to reply to this message.
j.c.sackett
Please take a look.
12 years, 11 months ago (2013年01月22日 21:42:52 UTC) #11
Please take a look.
Sign in to reply to this message.
j.c.sackett
Please take a look.
12 years, 11 months ago (2013年01月22日 22:13:06 UTC) #12
Please take a look.
Sign in to reply to this message.
abentley-home
On 2013年01月22日 22:13:06, j.c.sackett wrote: > Please take a look. lgtm
12 years, 11 months ago (2013年01月22日 22:14:46 UTC) #13
On 2013年01月22日 22:13:06, j.c.sackett wrote:
> Please take a look.
lgtm
Sign in to reply to this message.
hazmat
On 2013年01月22日 22:13:06, j.c.sackett wrote: > Please take a look. glad to see the context ...
12 years, 11 months ago (2013年01月23日 13:29:44 UTC) #14
On 2013年01月22日 22:13:06, j.c.sackett wrote:
> Please take a look.
glad to see the context manager got implemented.
my previous review comment regarding the lack of importerror checking causes the
entire queue to be processed in error is still valid.
unknown errors of using charm proof against a charm should get recorded as an
error in the output data for the charm.
as an example of using the context manager to return the lib that allows a guard
against the import error.
http://paste.ubuntu.com/1562870/ 
Sign in to reply to this message.
j.c.sackett
Please take a look.
12 years, 11 months ago (2013年01月23日 15:42:23 UTC) #15
Please take a look.
Sign in to reply to this message.
hazmat
https://codereview.appspot.com/7137054/diff/18001/charmworld/jobs/proof.py File charmworld/jobs/proof.py (right): https://codereview.appspot.com/7137054/diff/18001/charmworld/jobs/proof.py#newcode66 charmworld/jobs/proof.py:66: log.exception( almost there. as noted in previous review comment. ...
12 years, 11 months ago (2013年01月24日 14:59:03 UTC) #16
https://codereview.appspot.com/7137054/diff/18001/charmworld/jobs/proof.py
File charmworld/jobs/proof.py (right):
https://codereview.appspot.com/7137054/diff/18001/charmworld/jobs/proof.py#ne...
charmworld/jobs/proof.py:66: log.exception(
almost there. as noted in previous review comment. an error/exception in running
proof lint on a charm should result in us recording an error against the charm.
else a potentially badly broken charm, looks like its green wrt to lint.
Sign in to reply to this message.
|
This is Rietveld f62528b

AltStyle によって変換されたページ (->オリジナル) /