Background SmackDown

Resque vs. Sidekiq

Resque vs. Sidekiq

Functional Differences

Forks vs. Spoons Threads

Installing

  # Gemfile
  gem 'resque'
  gem 'sidekiq'
  

  $ bundle install
  

Using Resque

  Resque.enqueue(ResqueWorker, job.id)
  

  class ResqueWorker
    @queue = :jobs

    def self.perform(job_id)
      start_time = Time.now
      job = Job.find job_id
      job.process(start_time)
    end
  end
  

  $ COUNT=8 QUEUE=* rake resque:workers
  

Using Sidekiq

  SidekiqWorker.perform_async(job.id)
  

  class SidekiqWorker
    include Sidekiq::Worker

    def perform(job_id)
      start_time = Time.now
      job = Job.find job_id
      job.process(start_time)
    end
  end
  

  $ sidekiq -c 8
  

Job - The "control"

  class Job < ActiveRecord::Base
    def process(start_time)
      self.started_at = start_time
      process_impl
      self.ended_at = Time.now
      save!
    end

    def process_impl
      # do nothing.  Subclasses will do stuff.
    end
  end
  

  class CpuJob < Job
    def process_impl
      BCrypt::Password.create(SecureRandom.hex(200),:cost => 14)
    end
  end
  

  class IoJob < Job
    require 'open-uri'
    def process_impl
      open('http://localhost:5000/')
    end
  end
  

  class MixedJob < Job
    def process_impl
      open('http://localhost:5000/?wait_time=1')
      BCrypt::Password.create(SecureRandom.hex(200),:cost => 14)
      open('http://localhost:5000/?wait_time=1')
    end
  end
  

Slownatra

  require 'sinatra/base'
  class Slownatra < Sinatra::Base
    DEFAULT_WAIT = 2
    get '/' do
      @wait_time = params[:wait_time] || DEFAULT_WAIT
      @wait_time = Random.rand(10) if @wait_time == "random"
      sleep(@wait_time.to_i)
      erb :index
    end
  end
  

Un-Scientific Methodology

1. Start worker(s)
2. Create batch
3. Create jobs
4. Queue jobs

What about memory use?

Sidekiq 8 workers : At Rest

    $ sidekiq -c 8

    $ ps -o rss,pcpu,command -p $(pgrep -f sidekiq -d',')
   RSS  %CPU COMMAND
 78352   0.0 sidekiq 2.17.0 background_smackdown [0 of 8 busy]
  

Sidekiq 8 Workers : While Working

    $ sidekiq -c 8
    $ ps -o rss,pcpu,command -p $(pgrep -f sidekiq -d',')
   RSS  %CPU COMMAND
 77116 107.3 sidekiq 2.17.0 background_smackdown [8 of 8 busy]
  

Resque 8 Workers : At Rest

    $ COUNT=8 QUEUE=* rake resque:workers

    $ ps -o rss,pcpu,command -p $(pgrep -f resque -d',')
   RSS  %CPU COMMAND
 54124   0.0 ruby /Users/jgreen/.rvm/gems/ruby-2.0.0-p353/bin/rake resque:workers
 75688   0.0 resque-1.25.1: Waiting for *  
 75656   0.0 resque-1.25.1: Waiting for *  
 77284   0.0 resque-1.25.1: Waiting for *  
 76576   0.0 resque-1.25.1: Waiting for *  
 77144   0.0 resque-1.25.1: Waiting for *  
 76688   0.0 resque-1.25.1: Waiting for *  
 77620   0.0 resque-1.25.1: Waiting for *  
 74792   0.0 resque-1.25.1: Waiting for *
  

Resque 8 Workers : At Rest

    $ COUNT=8 QUEUE=* rake resque:workers

    $ ps -o rss,pcpu,command -p $(pgrep -f resque -d',')
   RSS  %CPU COMMAND
 54124   0.0 ruby /Users/jgreen/.rvm/gems/ruby-2.0.0-p353/bin/rake resque:workers
 76292   5.8 resque-1.25.1: Forked 8004 at 1386358588  
 76344   5.3 resque-1.25.1: Forked 8007 at 1386358588  
 77736   6.6 resque-1.25.1: Forked 8016 at 1386358588  
 77004   5.7 resque-1.25.1: Forked 8000 at 1386358588  
 77444   6.3 resque-1.25.1: Forked 8008 at 1386358588  
 77072   6.6 resque-1.25.1: Forked 8009 at 1386358588  
 78124   5.4 resque-1.25.1: Forked 8003 at 1386358588  
 75276   6.0 resque-1.25.1: Forked 8006 at 1386358588  
 27772   8.8 resque-1.25.1: Processing jobs since 1386358588 [ResqueWorker]  
 27484   6.0 resque-1.25.1: Processing jobs since 1386358588 [ResqueWorker]  
 26752   5.7 resque-1.25.1: Processing jobs since 1386358588 [ResqueWorker]  
 23876   5.7 resque-1.25.1: Processing jobs since 1386358588 [ResqueWorker]  
 25040   5.2 resque-1.25.1: Processing jobs since 1386358588 [ResqueWorker]  
 24412   5.4 resque-1.25.1: Processing jobs since 1386358588 [ResqueWorker]
  

Verdict?

Not so fast!

Can we minimize the divide?

Resque tweaks

  # Gemfile
  gem 'resque-multi-job-forks'
  

  $ JOBS_PER_FORK=1000 COUNT=8 QUEUE=* rake resque:workers
  

Sidekiq tweaks

  $ sidekiq -c 1
  $ sidekiq -c 1
  $ sidekiq -c 1
  $ sidekiq -c 1
  $ sidekiq -c 1
  $ sidekiq -c 1
  $ sidekiq -c 1
  $ sidekiq -c 1
  

An interesting mistake

  class CpuJob < Job
    def process_impl
      (1..25000).inject(:*) || 1
    end
  end
  

Thanks For Watching!