| Nov | DEC | Jan |
| 06 | ||
| 2019 | 2020 | 2021 |
COLLECTED BY
Collection: github.com
Add TypeScript alternative64c5b2e
require "scientist" class MyWidget def allows?(user) experiment = Scientist::Default.new "widget-permissions" experiment.use { model.check_user?(user).valid? } # old way experiment.try { user.can?(:read, model) } # new way experiment.run end endWrap a
use block around the code's original behavior, and wrap try around the new behavior. experiment.run will always return whatever the use block returns, but it does a bunch of stuff behind the scenes:
●It decides whether or not to run the try block,
●Randomizes the order in which use and try blocks are run,
●Measures the durations of all behaviors in seconds,
●Compares the result of try to the result of use,
●Swallow and record exceptions raised in the try block when overriding raised, and
●Publishes all this information.
The use block is called the control. The try block is called the candidate.
Creating an experiment is wordy, but when you include the Scientist module, the science helper will instantiate an experiment and call run for you:
require "scientist" class MyWidget include Scientist def allows?(user) science "widget-permissions" do |experiment| experiment.use { model.check_user(user).valid? } # old way experiment.try { user.can?(:read, model) } # new way end # returns the control value end endIf you don't declare any
try blocks, none of the Scientist machinery is invoked and the control value is always returned.
try blocks don't run yet and none of the results get published. Replace the default experiment implementation to control execution and reporting:
require "scientist/experiment" class MyExperiment include Scientist::Experiment attr_accessor :name def initialize(name) @name = name end def enabled? # see "Ramping up experiments" below true end def raised(operation, error) # see "In a Scientist callback" below p "Operation '#{operation}' failed with error '#{error.inspect}'" super # will re-raise end def publish(result) # see "Publishing results" below p result end endNow calls to the
science helper will load instances of MyExperiment.
==. To override this behavior, use compare to define how to compare observed values instead:
class MyWidget include Scientist def users science "users" do |e| e.use { User.all } # returns User instances e.try { UserService.list } # returns UserService::User instances e.compare do |control, candidate| control.map(&:login) == candidate.map(&:login) end end end end
context method to add to or retrieve the context for an experiment:
science "widget-permissions" do |e| e.context :user => user e.use { model.check_user(user).valid? } e.try { user.can?(:read, model) } end
context takes a Symbol-keyed Hash of extra data. The data is available in Experiment#publish via the context method. If you're using the science helper a lot in a class, you can provide a default context:
class MyWidget include Scientist def allows?(user) science "widget-permissions" do |e| e.context :user => user e.use { model.check_user(user).valid? } e.try { user.can?(:read, model) } end end def destroy science "widget-destruction" do |e| e.use { old_scary_destroy } e.try { new_safe_destroy } end end def default_scientist_context { :widget => self } end endThe
widget-permissions and widget-destruction experiments will both have a :widget key in their contexts.
before_run method:
# Code under test modifies this in-place. We want to copy it for the # candidate code, but only when needed: value_for_original_code = big_object value_for_new_code = nil science "expensive-but-worthwhile" do |e| e.before_run do value_for_new_code = big_object.deep_copy end e.use { original_code(value_for_original_code) } e.try { new_code(value_for_new_code) } end
User instances, but when researching a mismatch, all you care about is the logins. You can define how to clean these values in an experiment:
class MyWidget include Scientist def users science "users" do |e| e.use { User.all } e.try { UserService.list } e.clean do |value| value.map(&:login).sort end end end endAnd this cleaned value is available in observations in the final published result:
class MyExperiment include Scientist::Experiment # ... def publish(result) result.control.value # [<User alice>, <User bob>, <User carol>] result.control.cleaned_value # ["alice", "bob", "carol"] end endNote that the
#clean method will discard the previous cleaner block if you call it again. If for some reason you need to access the currently configured cleaner block, Scientist::Experiment#cleaner will return the block without further ado. (This probably won't come up in normal usage, but comes in handy if you're writing, say, a custom experiment runner that provides default cleaners.)
ignore method. You may include more than one block if needed:
def admin?(user) science "widget-permissions" do |e| e.use { model.check_user(user).admin? } e.try { user.can?(:admin, model) } e.ignore { user.staff? } # user is staff, always an admin in the new system e.ignore do |control, candidate| # new system doesn't handle unconfirmed users yet: control &&!candidate &&!user.confirmed_email? end end endThe ignore blocks are only called if the values don't match. If one observation raises an exception and the other doesn't, it's always considered a mismatch. If both observations raise different exceptions, that is also considered a mismatch.
run_if block. If this returns false, the experiment will merely return the control value. Otherwise, it defers to the experiment's configured enabled? method.
class DashboardController include Scientist def dashboard_items science "dashboard-items" do |e| # only run this experiment for staff members e.run_if { current_user.staff? } # ... end end
enabled? method in your Scientist::Experiment implementation.
class MyExperiment include Scientist::Experiment attr_accessor :name, :percent_enabled def initialize(name) @name = name @percent_enabled = 100 end def enabled? percent_enabled > 0 && rand(100) < percent_enabled end # ... endThis code will be invoked for every method with an experiment every time, so be sensitive about its performance. For example, you can store an experiment in the database but wrap it in various levels of caching such as memcache or per-request thread-locals.
publish(result) method, and can publish data however you like. For example, timing data can be sent to graphite, and mismatches can be placed in a capped collection in redis for debugging later.
The publish method is given a Scientist::Result instance with its associated Scientist::Observations:
class MyExperiment include Scientist::Experiment # ... def publish(result) # Store the timing for the control value, $statsd.timing "science.#{name}.control", result.control.duration # for the candidate (only the first, see "Breaking the rules" below, $statsd.timing "science.#{name}.candidate", result.candidates.first.duration # and counts for match/ignore/mismatch: if result.matched? $statsd.increment "science.#{name}.matched" elsif result.ignored? $statsd.increment "science.#{name}.ignored" else $statsd.increment "science.#{name}.mismatched" # Finally, store mismatches in redis so they can be retrieved and examined # later on, for debugging and research. store_mismatch_data(result) end end def store_mismatch_data(result) payload = { :name => name, :context => context, :control => observation_payload(result.control), :candidate => observation_payload(result.candidates.first), :execution_order => result.observations.map(&:name) } key = "science.#{name}.mismatch" $redis.lpush key, payload $redis.ltrim key, 0, 1000 end def observation_payload(observation) if observation.raised? { :exception => observation.exception.class, :message => observation.exception.message, :backtrace => observation.exception.backtrace } else { # see "Keeping it clean" above :value => observation.cleaned_value } end end end
raise_on_mismatches class attribute when you include Scientist::Experiment. Only do this in your test suite!
To raise on mismatches:
class MyExperiment include Scientist::Experiment # ... implementation end MyExperiment.raise_on_mismatches = trueScientist will raise a
Scientist::Experiment::MismatchError exception if any observations don't match.
Scientist::Experiment::MismatchError:
class CustomMismatchError < Scientist::Experiment::MismatchError def to_s message = "There was a mismatch! Here's the diff:" diffs = result.candidates.map do |candidate| Diff.new(result.control, candidate) end.join("\n") "#{message}\n#{diffs}" end end
science "widget-permissions" do |e| e.use { Report.find(id) } e.try { ReportService.new.fetch(id) } e.raise_with CustomMismatchError endThis allows for pre-processing on mismatch error exception messages.
tryoruse block, including some where rescuing may cause unexpected behavior (like SystemExitorScriptError). To rescue a more restrictive set of exceptions, modify the RESCUES list:
# default is [Exception] Scientist::Observation::RESCUES.replace [StandardError]
publish, compare, or clean, the raised method is called with the symbol name of the internal operation that failed and the exception that was raised. The default behavior of Scientist::Default is to simply re-raise the exception. Since this halts the experiment entirely, it's often a better idea to handle this error and continue so the experiment as a whole isn't canceled entirely:
class MyExperiment include Scientist::Experiment # ... def raised(operation, error) InternalErrorTracker.track! "science failure in #{name}: #{operation}", error end endThe operations that may be handled here are: ●
:clean - an exception is raised in a clean block
●:compare - an exception is raised in a compare block
●:enabled - an exception is raised in the enabled? method
●:ignore - an exception is raised in an ignore block
●:publish - an exception is raised in the publish method
●:run_if - an exception is raised in a run_if block
enabled? and run_if determine when a candidate runs, it's impossible to guarantee that it will run every time. For this reason, Scientist is only safe for wrapping methods that aren't changing data.
When using Scientist, we've found it most useful to modify both the existing and new systems simultaneously anywhere writes happen, and verify the results at read time with science. raise_on_mismatches has also been useful to ensure that the correct data was written during tests, and reviewing published mismatches has helped us find any situations we overlooked with our production data at runtime. When writing to and reading from two systems, it's also useful to write some data reconciliation scripts to verify and clean up production data alongside any running experiments.
try and use blocks run sequentially in random order. As such, any data upon which your code depends may change before the second block is invoked, potentially yielding a mismatch between the candidate and control return values. To calibrate your expectations with respect to false negatives arising from systemic conditions external to your proposed changes, consider starting with an experiment in which both the try and use blocks invoke the control method. Then proceed with introducing a candidate.
enabled? method, you can use it to silently and carefully test new code paths and ignore the results altogether. You can do this by setting ignore { true }, or for greater efficiency, compare { true }.
This will still log mismatches if any exceptions are raised, but will disregard the values entirely.
try blocks:
require "scientist" class MyWidget include Scientist def allows?(user) science "widget-permissions" do |e| e.use { model.check_user(user).valid? } # old way e.try("api") { user.can?(:read, model) } # new service API e.try("raw-sql") { user.can_sql?(:read, model) } # raw query end end endWhen the experiment runs, all candidate behaviors are tested and each candidate observation is compared with the control in turn.
try blocks, omit a use, and pass a candidate name to run:
experiment = MyExperiment.new("various-ways") do |e| e.try("first-way") { ... } e.try("second-way") { ... } end experiment.run("second-way")The
science helper also knows this trick:
science "various-ways", run: "first-way" do |e| e.try("first-way") { ... } e.try("second-way") { ... } end
fabricate_durations_for_testing_purposes method, and Scientist will report these in Scientist::Observation#duration instead of the actual execution times.
science "absolutely-nothing-suspicious-happening-here" do |e| e.use { ... } # "control" e.try { ... } # "candidate" e.fabricate_durations_for_testing_purposes( "control" => 1.0, "candidate" => 0.5 ) end
fabricate_durations_for_testing_purposes takes a Hash of duration values, keyed by behavior names. (By default, Scientist uses "control" and "candidate", but if you override these as shown in Trying more than one thingorNo control, just candidates, use matching names here.) If a name is not provided, the actual execution time will be reported instead.
Like Scientist::Experiment#cleaner, this probably won't come up in normal usage. It's here to make it easier to test code that extends Scientist.
Scientist.run:
Scientist.run "widget-permissions" do |e| e.use { model.check_user(user).valid? } e.try { user.can?(:read, model) } end
script/test runs the unit tests. All development dependencies are installed automatically. Scientist requires Ruby 2.3 or newer.