A presentation at Saint P Ruby Meetup in in St Petersburg, Russia by Vladimir Dementyev
Component B Component A Component C Engine-ering Rails Applications Vladimir Dementyev, Saint-P Ruby 2019
We* are Doomed * Rails developers
Can you escape your destiny? V.Yurlov, for “The war of the worlds” by H.G.Wells
Components 👍
Monolith 😞
Hanami
Elixir
Shopify: Modular Monolith
SHOPIFY all others
Rails
The Book?
What we’ve done Or how we decided to go with engines
Co-living rentals service
Our mission • They had a property and lease managements system (admin panel) • They needed a community app for users • They wanted to keep everything in the same Rails app
Community • • • • Events Perks Chat Billing
Phase #1: Namespaces app/ controllers/ chat/… models/ chat/… jobs/ chat/…
Namespaces • Quick start • Fake isolation
Our mission evolves • There is a property and lease management application (admin panel) • We need a community app for users • We (they) want to keep everything in the same Rails app • And we (again, they) want to re-use the app’s code in the future for a side-project
Phase #2: Engines & Gems app/… engines/ chat/ app/ controllers/… … lib/… gems/…
The Modular Monolith: Rails Architecture
rails plugin new \ my_engine !—mountable Engems Building Rails apps from engines and gems
Crash Course in Engines $ rails plugin new my_engine \ !—mountable # or —full create create create create … create README.md Rakefile Gemfile my_engine.gemspec Engine is a gem
Crash Course in Engines my_engine/ app/ controllers/… … config/routes.rb lib/ my_engine/engine.rb my_engine.rb Added to paths
Crash Course in Engines # my_engine/lib/my_engine/engine.rb module MyEngine class Engine < !::Rails!::Engine # only in mountable engines isolate_namespace MyEngine end end
Crash Course in Engines # my_engine/config/routes.rb MyEngine!::Engine.routes.draw do get “/best_ruby_conference”, to: redirect(“https:!//spbrubyconf.ru”) end
Crash Course in Engines # <root>/Gemfile gem “my_engine”, path: “my_engine”
Crash Course in Engines # <root>/config/routes.rb Rails.application.routes.draw do mount MyEngine!::Engine !=> “/my_engine” end
Crash Course in Engines $ rake routes Prefix Verb … my_engine URI Pattern Controller#Action /my_engine MyEngine!::Engine Routes for MyEngine!::Engine: best_ruby_conference GET /best_ruby_conference(:format)
The end
Common Engines common-events perks_by connect_by active-storage-proxy chat_by meet_by manage_by main app
Dependencies • How to use non-Rubygems deps? • How to share common deps? • How to sync versions?
Path/Git problem # engines/my_engine/Gemfile gem “local-lib”, path: “!../local-lib” gem “git-lib”, github: “palkan/git-lib” # <root>/Gemfile gem “my_engine”, path: “engines/my_engine” gem “local-lib”, path: “!../local-lib” gem “git-lib”, github: “palkan/git-lib”
eval_gemfile # engines/my_engine/Gemfile eval_gemfile “./Gemfile.runtime” # engines/my_engine/Gemfile.runtime gem “local-lib”, path: “!../local-lib” gem “git-lib”, github: “palkan/git-lib” # <root>/Gemfile gem “my_engine”, path: “engines/my_engine” eval_gemfile “engines/my_engine/Gemfile.runtime”
eval_gemfile
Shared Gemfiles gemfiles/ profilers.gemfile rails.gemfile gem “stackprof”, “0.2.12” gem “ruby-prof”, “0.17.0” gem “memory_profiler”, “0.9.12” gem “rails”, “6.0.0.rc1”
Shared Gemfiles # engines/my_engine/Gemfile eval_gemfile “!../gemfiles/rails.runtime” eval_gemfile “!../gemfiles/profilers.runtime” # <root>/Gemfile eval_gemfile “gemfiles/rails.runtime” eval_gemfile “gemfiles/profilers.runtime”
Keep Versions N’Sync gem ‘transdeps’ A gem to find inconsistent dependency versions in component-based Ruby apps. NOT VER IFIED
DB vs. Engines • How to manage migrations? • How to write seeds? • How to namespace tables?
Migrations. Option #1 Install migrations: rails my_engine:install:migrations 🤔
Migrations. Option #2 “Mount“ migrations: # engines/my_engine/lib/my_engine/engine.rb initializer “my_engine.migrations” do |app| app.config.paths[“db/migrate”].concat( config.paths[“db/migrate”].expanded ) # For checking pending migrations ActiveRecord!::Migrator.migrations_paths += config.paths[“db/migrate”].expanded.flatten end 😺
Seeds # <root>/db/seed.rb ActiveRecord!::Base.transaction do ConnectBy!::Engine.load_seed PerksBy!::Engine.load_seed ChatBy!::Engine.load_seed MeetBy!::Engine.load_seed end
table_name_prefix class CreateConnectByInterestTags < ActiveRecord!::Migration include ConnectBy!::MigrationTablePrefix def change create_table :interest_tags do |t| t.string :name, null: false end end end
table_name_prefix module ConnectBy module MigrationTablePrefix def table_prefix ConnectBy.table_name_prefix end def table_name_options(config = ActiveRecord!::Base) { table_name_prefix: “!#{table_prefix}!#{config.table_name_prefix}”, table_name_suffix: config.table_name_suffix } end end end
table_name_prefix
Factories # engines/my_engine/lib/my_engine/engine.rb initializer “my_engine.factories” do |app| factories_path = root.join(“spec”, “factories”) Custom load hook ActiveSupport.on_load(:factory_bot) do require “connect_by/ext/factory_bot_dsl” FactoryBot.definition_file_paths.unshift factories_path end end
Factories: Aliasing using ConnectBy!::FactoryBotDSL Takes engine namespace into account FactoryBot.define do # Uses ConnectBy.factory_name_prefix + “city” as the name factory :city do sequence(:name) { |n| Faker!::Address.city + ” (!#{n})”} trait :private do visibility { :privately_visible } end end end
Factories: Aliasing # spec/support/factory_aliases.rb unless ConnectBy.factory_name_prefix.empty? RSpec.configure do |config| config.before(:suite) do FactoryBot.factories.map do |factory| next unless factory.name.to_s.starts_with?(ConnectBy.factory_name_prefix) FactoryBot.factories.register( factory.name.to_s.sub(/^!#{ConnectBy.factory_name_prefix}/, “”).to_sym, factory ) end end end end
Factories: Aliasing • Use short (local) name when testing the engine • Use long (namespaced) when using in other engines
Shared Contexts my_engine/ lib/ my_engine/ testing/ shared_contexts/… shared_examples/… testing.rb
Shared Contexts # other_engine/spec/rails_helper/rb require “connect_by/testing/shared_contexts” require “connect_by/testing/shared_examples”
How to test engines?
Testing. Option #1 Using a full-featured Dummy app: spec/ dummy/ app/ controllers/ … config/ db/ test/ … 😕
Testing. Option #2 gem ‘combustion’ A library to help you test your Rails Engines in a simple and effective manner, instead of creating a full Rails application in your spec or test folder. 🙃
Combustion begin Combustion.initialize! :active_record, :active_job do config.logger = Logger.new(nil) config.log_level = :fatal config.autoloader = :zeitwerk config.active_storage.service = :test config.active_job.queue_adapter = :test end rescue !=> e # Fail fast if application couldn’t be loaded $stdout.puts “Failed to load the app: !#{e.message}\n” \ “!#{e.backtrace.take(5).join(“\n”)}” exit(1) end
Combustion spec/ internal/ config/ storage.yml db/ schema.rb
Combustion • Load only required Rails frameworks • No boilerplate, only the files you need • Automatically re-runs migrations for every test run
CI commands: engem: description: Run engine/gem build parameters: target: type: string steps: - run: Skip if no relevant changes name: “[!<< parameters.target !>>] bundle install” command: | .circleci/is-dirty !<< parameters.target !>> !|| \ bundle exec bin/engem !<< parameters.target !>> build
.circleci/is-dirty pattern = File.join(!dir!, “!../{engines,gems}//.gemspec”) gemspecs = Dir.glob(pattern).map do |gemspec_file| Gem!::Specification.load(gemspec_file) end names = gemspecs.each_with_object({}) do |gemspec, hash| hash[gemspec.name] = [] end # see next slide
.circleci/is-dirty tree = gemspecs.each_with_object(names) do |gemspec, hash| deps = Set.new(gemspec.dependencies.map(&:name)) + Set.new(gemspec.development_dependencies.map(&:name)) local_deps = deps & Set.new(names.keys) local_deps.each do |local_dep| hash[local_dep] !<< gemspec.name end end # invert tree to show which gem depends on what tree.each_with_object(Hash.new { |h, k| h[k] = Set.new }) do |(name, deps), index| deps.each { |dep| index[dep] !<< name } index end
.circleci/is-dirty def dirty_libraries changed_files = git diff $(git merge-base origin/master HEAD) !--name-only .split(“\n”) raise “failed to get changed files” unless $!?.success? changed_files.each_with_object(Set.new) do |file, changeset| if file =~ %r{^(engines|gems)/([^/]+)} changeset !<< Regexp.last_match[2] end changeset end end
engines: executor: rails steps: - attach_workspace: at: . - engem: target: connect_by - engem: target: perks_by - engem: target: chat_by - engem: target: manage_by - engem: target: meet_by CI
Dev Tools • rails plugins new is too simple • Use generators: rails g engine
Generators # lib/generators/engine/engine_generator.rb class EngineGenerator < Rails!::Generators!::NamedBase source_root File.expand_path(“templates”, !dir!) def create_engine directory(“.”, “engines/!#{name}”) end end
Dev Tools • rails plugins new is too simple • Use generators: • Manage engines: rails g engine bin/engem
bin/engem # run a specific test $ ./bin/engem connect_by rspec spec/models/connect_by/city.rb:5
bundle install, rubocop and rspec by default $ ./bin/engem connect_by buildEngine Engine Main app
How to modify other engine’s entities?
Base & Behaviour • Base Rails classes within an engine MUST be configurable • They also MAY require to have a certain “interface”
Base & Behaviour module ConnectBy class ApplicationController < Engine.config.application_controller.constantize raise “Must include ConnectBy!::ControllerBehaviour” unless self < ConnectBy!::ControllerBehaviour raise “Must implement #current_user method” unless instance_methods.include?(:current_user) end end
Base & Behaviour module ConnectBy class ApplicationController < Engine.config.application_controller.constantize raise “Must include ConnectBy!::ControllerBehaviour” unless self < ConnectBy!::ControllerBehaviour raise “Must implement #current_user method” unless instance_methods.include?(:current_user) end end Configurable
Base & Behaviour # config/application.rb config.connect_by.application_controller = “MyApplicationController”
Base & Behaviour module ConnectBy class ApplicationController < Engine.config.application_controller.constantize raise “Must include ConnectBy!::ControllerBehaviour” unless self < ConnectBy!::ControllerBehaviour raise “Must implement #current_user method” unless instance_methods.include?(:current_user) end end “Interface”
Base & Behaviour class ApplicationController < ActionController!::API include ConnectBy!::ControllerBehaviour include JWTAuth include RavenContext if defined?(Raven) end
Modify Models • Prefer extensions over patches • Use load hooks (to support autoload/ reload)
Load Hooks # engines/connect_by/app/models/connect_by/city.rb module ConnectBy class City < ActiveRecord!::Base # !!… ActiveSupport.run_load_hooks( “connect_by/city”, self ) end end
Load Hooks # engines/meet_by/lib/meet_by/engine.rb initializer “meet_by.extensions” do ActiveSupport.on_load(“connect_by/city”) do include !::MeetBy!::Ext!::ConnectBy!::City end end
Load Hooks # engines/meet_by/app/models/ext/connect_by/city.rb module MeetBy module Ext module ConnectBy module City extend ActiveSupport!::Concern included do has_many :events, class_name: “MeetBy!::Events!::Member”, inverse_of: :user, foreign_key: :user_id, dependent: :destroy end end end end end
How to communicate between engines?
Problem • Some “events” trigger actions in multiple engines • E.g. , user registration triggers chat membership initialization, manager notifications, etc.
• But user registration “lives” in connect_by and have no idea about chats, managers, whatever 🤔
Solution • Events • Events • Events
Tools • Hanami Events • Rails Events Store • Wisper (?) • dry-events (?)
Railsy RES
Railsy RES vs RES • Class-independent event types • Uses RES as interchangeable adapter • Better testing tools • Less verbose API and convention over configuration
Railsy RES class ProfileCompleted < Railsy!::Events!::Event # (optional) event identifier is used for transmitting events # to subscribers. # # By default, identifier is equal to name.underscore.gsub('/', '.'). self.identifier = “profile_completed” # Add attributes accessors attributes :user_id # Sync attributes only available for sync subscribers sync_attributes :user end
Railsy RES event = ProfileCompleted.new(user_id: user.id) # or with metadata event = ProfileCompleted.new( user_id: user.id, metadata: { ip: request.remote_ip } ) # then publish the event Railsy!::Events.publish(event)
Railsy RES initializer “my_engine.subscribe_to_events” do ActiveSupport.on_load “railsy-events” do |store| # async subscriber is invoked from background job, # enqueued after the current transaction commits store.subscribe MyEventHandler, to: ProfileCreated # anonymous handler (could only be synchronous) store.subscribe(to: ProfileCreated, sync: true) do |event| # do something end # subscribes to ProfileCreated automatically store.subscribe OnProfileCreated!::DoThat end end Convention over configuration
Railsy RES engines/ connect_by/ events/ connect_by/ users/ registered.rb chat_by/ subscribers/ connect_by/ users/ on_registered/ create_chat_account.rb
What we implement in the main app?
Main App • Authentication
Authentication module ConnectBy class ApplicationController < Engine.config.application_controller.constantize raise “Must include ConnectBy!::ControllerBehaviour” unless self < ConnectBy!::ControllerBehaviour raise “Must implement #current_user method” unless instance_methods.include?(:current_user) end end Engines don’t care about how do you obtain the current user
Main App • Authentication • Feature/system tests • Locales and mailers templates • Instrumentation and exception handling • Configuration
Gems Or stop putting everything into lib/ folder
Gems • Shared non-application specific code between engines • Isolated tests • Ability to share between applications in the future (e.g. , GitHub Package Registry)
Gems gems/ common-rubocop/ common-testing/ common-graphql/ …
common-rubocop • Standard RuboCop configuration (based on standard gem)
• RuboCop plugins (e.g. , rubocoprspec)
• Custom cops
common-rubocop # .rubocop.yml inherit_gem: common-rubocop: config/base.yml
common-testing • RSpec tools bundle (factory_bot, faker, test-prof, formatters)
• Common spec_helper.rb and rails_helper.rb
common-testing # rails_helper.rb require “common/testing/rails_configuration”
common-graphql • Base classes (object, mutation) • Additional scalar types • Batch loaders • Testing helpers
Why engines? And why not
Why engines? • Modular monolith instead of monolith monsters or micro-services hell • Code (and tests) isolation • Loose coupling (no more spaghetti logic)
Why engines? • Easy to upgrade frameworks (e.g. , Rails) component by component • Easy to re-use logic in another app • …and we can migrate to micro-services in the future (if we’re brave enough)
Why not engines? • Not so good third-party gems support • Not so good support within Rails itself
Thanks! Спасибо!
Rails applications tend to grow and turn into massive monoliths–that’s a natural evolution of a Rails app, isn’t it?
What happens next is you starting looking for an architectural solution to keep the codebase maintainable. Microservices? If you brave enough…
Rails ecosystem already has a right tool for the job: engines. With the help of engines, you can split your application into independent parts combined under the same root application–the same way rails gem combines all its sub-frameworks, which are engines too, by the way.
Curious how to do that? Come to hear how we’ve engine-ified our Rails monolith and what difficulties we faced along the way.
The following resources were mentioned during the presentation or are useful additional information.
for free. You
can too.