Workflows
User submitting debug info
Basically:
get samples
run a script
submit that directory
This is detailed in tools/ffmpeg/readme_run_tests.txt
.
Changing code in pyglet ffmpeg subsystem
Preparation like in readme_run_tests.txt, optionally install the library bokeh
(http://bokeh.pydata.org/en/latest/index.html) for visualization support.
The basic flow goes as:
It is possible to return to a previous session to request additional reports:
configure.py activate <session>
report.py ...
You can list the known sessions for the current samples_dir with:
Important
All this debugging machinery depends on a detailed and accurate capture of
media_player related state, currently in examples/media_player.py and
pyglet.media.player.
Modifications in those modules may require matching modifications in
pyglet/media/sources/instrumentation.py, and further propagation to other
modules.
Changing the debug code for pyglet ffmpeg
For initial debugging of debug code, where there are misspellings and trivial
errors to weed out, creating a new session for each run_test_suite.py run may
be inconvenient.
The flag dev_debug
can be set to true in the session configuration file;
this will allow to rewrite the session.
Keep in mind that some raw data will be stale or misleading:
The ones captured at session creation time (currently pyglet.info and
pyglet_changeset)
The collected crashes info (new crashes will not be seen)
If media_player.py crashes before doing any writing, the state recording
will be the previous recording.
The reports using that stale raw data will obviously report stale data.
So it is a good idea to switch to a normal workflow as soon as posible
(simply creating a new session and deleting the special session).
Session
If playlist_file
is not specified, then all files in samples_dir, except
for the files with extension “.dbg”, “.htm”, “.html”, “.json”, “.log”, “.pkl”,
“.py”, “.txt” will make the implicit playlist; subdirectories of samples_dir
will not be explored.
If a playlist_file
is specified, then it should contain one sample name
per line; a sanity check will be performed ensuring no blacklisted extension
is used, and that the sample exists in samples_dir.
Once the playlist_file
is used in configure.py new
a copy is writen to
the session raw data directory, and this copy will be the authoritative
playlist for the session; playlist_file
can be deleted if desired.
Specifying a playlist is useful in development to restrict the tests to
samples relevant to the feature or issue under work.
The session name will be used to create a samples_dir subdir to store the test
results, hence it should be different of previous sessions names, and it must
not contain slashes, /
, backslashes \
or characters forbidden in
directory names.
Active session
Most commands and subcommands target the currently active session.
A session becomes active when
The current implementation relies in two pieces of data to determine the
active session
the environment variable pyglet_mp_samples_dir
specifies samples_dir,
the directory where all the media samples reside. Under the current
paths schema is also where session data will be stored, one subdir per
session.
a file activation.json
in samples_dir storing the name for the
current active session.
Notice that the second precludes running two commands in parallel targeting
two different sessions in the same sample_dir.
The concept of active session plus the enforced path schema avoids the need to
provide paths at each command invocation, making for less errors, easier docs
and less typing.
Commands Summary
Primary commands
They are the ones normally used by developers
configure.py
, mp.py
: session creation, activation, protection, status
and list all.
run_test_suite.py
: plays session’s samples, reports results.
report.py
: produces the specified report for the specified sample.
timeline.py
: translates the event stream to a stream of media_player
state, useful to pass to other software.
bokeh_timeline.py
: visualization of data collected for the specified
sample.
Helper commands
Somehow an artifact of run_test_suite.py
development, can help in testing
the debugging subsystem. run_test_suite.py
is basically playmany.py +
retry_crashed.py + summarize.py
. When trying to change run_test_suite.py
it is easier to first adapt the relevant helper.
playmany.py
: plays active session samples, recording media_player state
along the play.
retry_crashed.py
: plays again samples that have been seen always
crashing, hoping to get a recording with no crash. Motivated by early tests on
Ubuntu, where sometimes (but not always) a sample will crash the media_player.
summarize.py
: using the raw data produced by the two previous commands
elaborates some reports, aiming to give an idea of how well the run was and
what samples should be investigated.
Commands detailed
bokeh_timeline.py
Usage:
Renders media player’s internal state graphically using bokeh.
Arguments:
The output will be written to session’s output dir under
reports/sample.timeline.html
.
Notice the plot can be zoomed live with the mouse wheel, but you must click
the button that looks as a distorted OP; it also does pan with mouse drag.
Example:
bokeh_timeline.py small.mp4
will write the output to report/small.mp4.timeline.html
.
compare.py
Usage:
compare.py --reldir=relpath other_session
Builds a reports comparing the active session with other_session.
Outputs to samples_dir/relpath/comparison_<session>_<other_session>.txt
.
mp.py
alias for configure.py
playmany.py
Usage:
Uses media_player to play a sequence of samples and record debug info.
A session must be active, see command configure.py
If the active configuration has disallowed dbg overwrites it will do nothing.
If a playlist was provided at session creation, then only the samples in the
playlist will be played, otherwise all files in samples_dir
.
report.py
Usage:
report.py sample report_name
Generates a report from the debugging info recorded while playing sample.
Arguments:
sample: sample to report
report_name: desired report, one of
"anomalies": Start, end and interesting events
"all": All data is exposed as text
"counter": How many occurrences of each defect
The report will be written to session’s output dir under
reports/sample.report_name.txt
.
Example:
report anomalies small.mp4
will write the report anomalies to report/small.mp4.anomalies.txt
.
The authoritative list of reports available comes from
reports.available_reports
retry_crashed.py
Usage:
retry_crashed.py [--clean] [max_retries]
Inspects the raw data collected to get the list of samples that crashed the
last time they were played.
Then it replays those samples, recording new raw data for them.
The process is repeated until all samples has a recording with no crashes or
the still crashing samples were played max_tries
times in this command
run.
Notice that only samples recorded as crashing in the last run are retried.
A configuration must be active, see command configure.py
.
Besides the updated debug recordings, a state is build and saved:
total_retries: total retries attempted, including previous runs
sometimes_crashed: list of samples that crashed one time but later
completed a play
always_crashed: list of samples that always crashed
Options:
--clean: discards crash data collected in a previous run
max_retries: defaults to 5
run_test_suite.py
Usage:
run_test_suite.py [samples_dir]
Plays media samples with the pyglet media_player, recording debug information
for each sample played and writing reports about the data captured.
Arguments:
samples_dir: directory with the media samples to play
If no samples_dir is provided the active session is the target.
If an explicit playlist was specified when creating the session, then only the
samples in the playlist will be played, otherwise all samples in samples_dir
will be played.
If sample_dir is provided, a session named testrun_00
(_01
, _02
,
… if that name was taken) will be created, with no explicit playlist, and
then the command operates as in the previous case.
Output files will be into:
samples_dir/session/dbg : binary capture of media_player events, other raw
data captured
samples_dir/session/reports : human readable reports
Note
This script will refuse to overwrite an existing test_run results
.
Output files will be into subdirectories:
samples_dir/test_run/dbg
Each sample will generate a sample.dbg
file storing the sequence of
player debug events seen while playing the sample.
It is simply a pickle of a list of tuples, each tuple an event.
There are not meant for direct human use, but to run some analyzers to
render useful reports.
A crash_retries.pkl
file, a pickle of
(total_retries, sometimes_crashed, still_crashing) <-> (int, set, set)
.
A pyglet.info
captured at session creation to track hw & sw.
A pyglet hg revision captured at session creation.
samples_dir/test_run/reports
Human readable outputs, described in command summarize.py
Later a user can generate visualizations and additional reports that will
be stored in this directory
summarize.py
Usage:
Summarizes the session info collected with playmany
and retry_crashes
.
A configuration must be active, see command configure.py
.
If a playlist was provided at session creation, then only the samples in the
playlist will be played, otherwise all files in samples_dir.
Produces human readable reports, constructed from the .dbg files.
Output will be in
samples_dir/test_run/reports
The files in that directory will be
00_summary.txt
, which provides:
03_pyglet_info.txt
, pyglet.info
output giving OS, python version,
etc (as captured at session creation).
04_pyglet_hg_revision.txt
, pyglet hg revision if running from a repo
clone, non writen if no repo (as captured at session creation).
sample_name.all.txt
and sample_name.anomalies.txt
for each sample that
played non perfect.
sample_name.all.txt
has all info in the sample_name.dbg
in human
readable form, that is, the sequence of player’s internal events along the
play.
sample_name.anomalies.txt
is a reduced version of the .all
.
variant: normal events are not shown, only anomalies.
timeline.py
Usage:
timeline.py sample [output_format]
Renders the media player’s debug info to a format more suitable to postprocess
in a spreadsheets or other software, particularly to get a data visualization.
See output details in the manual.
Arguments:
sample: sample to report
output_format : one of { "csv", "pkl"}, by default saves as .pkl (pickle)
The output will be written to session’s output dir under
reports/sample.timeline.[.pkl or .csv]
.
Example:
will write the output to report/small.mp4.timeline.pkl
.
Note
.csv
sample is currently not implemented.