ruby - Kiprosh Blogs

Type-checking in Ruby 3 using RBS

Navaneeth Krishnan P — Wed, 05 Jul 2023 07:17:29 GMT

Ruby 3 has introduced a new syntax language for dynamic typing called RBS. In short, it is a language that we can use to describe the data types used in a Ruby class. We can define the data type of variables and the return type of methods used in a class using RBS.

Why do we need type-checking?

Since Ruby is a dynamically typed language, we don't need to define the data type of the variables we are using. Ruby automatically assigns a type based on the variable's value at runtime. Let's take the below class as an example.

# basic_math.rb

class BasicMath
  def initialize(num1, num2)
    @num1 = num1
    @num2 = num2
  end

  def first_less_than_second?
    @num1 < @num2
  end

  def add
    @num1 + @num2
  end
end

If we run the below code it will return false as expected since integer 1000 is not less than integer 2.

puts BasicMath.new(1000, 2).first_less_than_second?
 
=> false

We did not define that num1 and num2 are integers. Ruby interpreter automatically assigned integer type to them at runtime. And the method first_less_than_second? checks if its first argument is less than the second one. But let's call the same method with strings as parameters.

puts BasicMath.new('1000', '2').first_less_than_second?
 
=> true

Here the output is true because num1 and num2 are strings. But for this method we expected the output to be false. This error happened because we missed doing a string-to-integer conversion on the parameters. A type-checking system will help us identify such errors.

How to write an RBS file?

Let's write an RBS file for the above BasicMath class.

# basic_math.rbs

class BasicMath
  @num1: Integer
  @num2: Integer

  def initialize: (Integer num1, Integer num2) -> void
  def first_less_than_second?: -> bool
  def add: -> Integer
end

For the variables @num1 and @num2, we define its type using the : symbol.
For methods, we write the method name followed by a : symbol and then specify the types of each method argument inside the () bracket. Then we define the return type using a -> symbol.

How to run an RBS file?

We only defined the structure of the BasicMath class in our RBS file. We still need to run it against a type checker to verify if the types are correct in our class. RBS is only a language that defines the structure of a class. It cannot do type-checking on its own. Let's use a popular type checker called steep to test our class against the RBS file.

Setup steep gem

Install steep gem using gem install steep.
Run steep init to generate the configuration file for steep.
This will generate a configuration file called Steepfile.
Add the following code to the Steepfile.

target :app do
  check "lib"
  signature "sig"

  library "set", "pathname"
end

Add the directory of the .rb files in check option and directory of .rbs files in signature option.
In this example, I am adding .rb files in lib directory and .rbs files in sig directory.
Make sure the file names of the respective .rb and .rbs files are the same.

Type-checking using Steep

Run steep using the command steep check. This will perform type-checking on all the Ruby files in lib directory, using their respective RBS files in sig directory.
If the type-checking passes it will return the below message.

steep check

# Type checking files:

....................................................................................

No type error detected. 🫖

If there is any mismatch in the types, it will throw an error message. For example, in BasicMath class, let's change the return value of first_less_than_second? to a String. In our RBS file, we have defined that the output of this method will be a Boolean.

# lib/basic_math.rb

class BasicMath
  def initialize(num1, num2)
    @num1 = num1
    @num2 = num2
  end

  def first_less_than_second?
    @num1 < @num2
    'yes'
  end

  def add
    @num1 + @num2
  end
end

Now if we run steep check, it will return an error message. This is because according to the RBS definition first_less_than_second? method is expected to return a Boolean value.

steep check

# Type checking files:

.......................................................................F............

lib/rbs_test.rb:7:6: [error] Cannot allow method body have type `::String` because declared as type `bool`
│   ::String <: bool
│     ::String <: (true | false)
│       ::String <: true
│
│ Diagnostic ID: Ruby::MethodBodyTypeMismatch
│
└   def first_less_than_second?
        ~~~~~~~~~~~~~~~~~~~~~~~

Detected 1 problem from 1 file

TypeProf

TypeProf is a type analysis tool that comes built-in with Ruby 3. It can automatically detect data types in a class and return a rough RBS file for it.
Just run the command typeprof and it will return a rough RBS file. Let's try running the typeprof command on the above BasicMath class.

typeprof lib/basic_math.rb

This will return the following result.

# TypeProf 0.21.3

# Classes
class BasicMath
  @num1: untyped
  @num2: untyped

  def initialize: (untyped num1, untyped num2) -> void
  def first_less_than_second?: -> untyped
  def add: -> untyped
end

If we check the returned RBS code, we can see that TypeProf assigned untyped as the data type in some places. This is because TypeProf couldn't determine the correct data type at those places. But it did return a skeletal structure for the RBS file which we can edit according to the way we want. TypeProf is still an experimental tool, so it doesn't always return an accurate RBS file.

Conclusion

We saw how we can write an RBS file for a class and do type-checking using Steep. We also saw how we can use TypeProf to generate a rough RBS file for a class. Since RBS allows us to add type definitions, it can help in catching type errors in our code. So it is a good practice to write RBS for our Ruby code.

References

All about "Data" Simple Immutable Value Objects in Ruby 3.2

Vishal Jain — Fri, 24 Mar 2023 07:30:50 GMT

In Ruby 3.2, a new class Data was introduced as a way to define simple immutable value objects. A value object is a type of object that represents a value in a program, such as a point in 2D space or a date. The main advantage of value objects is that they are easy to understand, simple to use, and can improve the readability and maintainability of code. The proposal to add Data class was accepted by Matz on the Ruby forum here.

How does it work?

Using the newly defined class Data we can create a simple immutable object. These objects are designed to be small, self-contained, and represent a single concept in your application.
The class definition includes the name of the class, as well as a list of instance variables that the object will contain. Here's an example:

class Point < Data.define(:x, :y)
end

# Can be initialized using Positional or Keyword arguments
point = Point.new(3, 4) OR Point.new(x: 3, y: 4) OR Point[3, 4] OR Point[x: 3, y: 4]
=> #

# But using both Positional and Keyword arguments together will not work
Point.new(2, y: 3)
=> `new`: wrong number of arguments (given 2, expected 0) (ArgumentError)

Once an instance of the object is created, its instance variables cannot be changed. This makes it easy to understand about the object's state and prevents bugs that can arise from unexpected changes to the object.

However, if we need to change any one instance variable by keeping the other variable same we can do that using with method.

point2 = point.with(x: 10)
=> #

Since Data objects are immutable, with method creates a copy of the object to update the arguments.
_{Note: If the with method is called with no arguments, the receiver is returned as-is and no new copy is created.}

Data.define also accepts an optional block that can be used to define custom methods for the immutable object. These methods can provide additional functionality to the object without compromising its immutability.

class Point < Data.define(:x, :y)
  def distance_from_origin
    Math.sqrt(x**2 + y**2)
  end
end

point = Point.new(3, 4)
=> #

point.distance_from_origin
=> 5.0

Why not Struct?

While Struct can also be used to define objects, there are some reasons why Data might be a better choice in certain situations.

First, Data provides some safety checks that Struct does not. For example, Data prevents an object from being created with a missing argument, while Struct allows this. This can help to prevent bugs and improve code safety.

Here's an example:

Measure = Struct.new(:amount, :unit)

measure = Measure.new(30) # This works, but will fail with Data
=> #

Point.new(x: 3) # Missing argument will raise error for Data
=> `initialize': missing keyword: :y (ArgumentError)

In this example, we've defined a Measure struct that contains two instance variables: amount and unit. However, when we create a new instance of the object, we only provide one argument. This is allowed by Struct, but it can lead to bugs if the code assumes that all the instance variables are present.

Second, Data is a more explicit way of defining simple immutable objects. The Data class makes it clear that the object is intended to be immutable, while with Struct it's less clear.

Final Thoughts

Data is a powerful tool for defining simple immutable objects in Ruby 3.2. While it may not be suitable for all situations, its safety, performance, and clarity benefits make it a strong choice in many cases. By understanding how Data works and its tradeoffs compared to other techniques, such as Struct, you can make an informed decision about when to use it in your own code.

References

Ruby 3.2.0 enhances Regexp performance and security with ReDoS protections

Krishna Singh — Wed, 01 Mar 2023 08:45:54 GMT

What is ReDoS?

Regular expression Denial of Service (ReDoS) is a security vulnerability that can occur in a regular expression (regex) when the regex is applied to a long string. This attack is designed to make a system or network unavailable to its intended users.

An example occurrence of a ReDoS

Imagine that a website has a form that accepts user input and uses a regex to validate the input. The regex is designed to only allow alphanumeric characters in the input, so it looks like this: /^[a-zA-Z0-9]+$/.
An attacker could potentially craft a string of input that consists of a very long sequence of characters, such as this:
'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'.
When this string is passed through the regex, it will take a very long time to validate, potentially causing the system to become unresponsive or crash. This would prevent legitimate users from accessing the website, effectively denying them service.
To prevent this attack, it is important to use regexes that are designed to be efficient and not susceptible to ReDoS attacks. This can involve using certain regex constructs and patterns that are known to be efficient, and avoiding certain constructs and patterns that can cause regexes to be slow or vulnerable to ReDoS attacks. It is also important to test regexes for efficiency and security before deploying them to the production environment.

How to prevent ReDoS in Ruby < 3.2.0 ?

To prevent ReDoS attacks in a Ruby on Rails application, there are several steps that you can take:

Avoid using the match method in your regexes, as this method can be slow and can lead to ReDoS attacks. Instead, use the scan or grep methods, which are faster and more efficient.
Avoid using regexes with nested quantifiers such as (a+)* or with unbounded repetitions such as a* or a+ in your Rails application. These types of regexes can be very slow and can lead to ReDoS attacks.
Use the /o option in your regexes to prevent them from being compiled multiple times. This can improve the efficiency of your regexes and can prevent ReDoS attacks.
Test your regexes for efficiency and security before deploying them in a production environment. This can help you identify any potential ReDoS vulnerabilities and can allow you to fix them before they can be exploited by an attacker.

Regexp improvements introduced in Ruby 3.2.0 to prevent ReDoS.

Ruby 3.2.0 introduced two improvements that significantly mitigate ReDoS.

Improved Regexp matching algorithm

Since Ruby 3.2, the matching algorithm for Regexp has been significantly enhanced by using a memoization technique:

This technique improves the performance of regexp matching by allowing most regexp matches to be completed in linear time.
This means that the time it takes to complete a regexp match will be directly proportional to the length of the input string, and will not increase exponentially as the input string gets longer.
This can help prevent ReDoS attacks, which are a type of security vulnerability that can occur when a regexp is applied to a long string of input.
Also, this optimization may consume memory proportional to the input length for each match. This means that the amount of memory used by the optimization will increase as the length of the input string increases.
This should not cause any practical problems because the memory allocation is usually delayed, and a normal regexp match should consume at most 10 times as much memory as the input length.

Example:-

With 3.1.0

  :001 > require 'benchmark'
  => true
  :002 > Benchmark.realtime { /^a*b?a*$/ =~ "a" * 50000 + "x" }
  => 24.171763999998802

With 3.2.0

  :001 > require 'benchmark'
  => true
  :002 > Benchmark.realtime { /^a*b?a*$/ =~ "a" * 50000 + "x" }
  => 0.007867999986046925

Preventing ReDoS attacks with regexp timeouts

This allows you to specify a timeout for regexp matching. Two different APIs that can be used to set a timeout for regexp matching are:

Regexp.timeout=

This is the process-global configuration of timeout for regexp matching. It allows you to specify a timeout that will apply to all regexp matches in your Ruby application. For example, you can use it like this:
```
  # Set a timeout of 1 second for regexp matching
  Regexp.timeout = 1.0

  regexp = Regexp.new("^[a-zA-Z0-9]+$")

  # Perform a regexp match
  regexp.scan(string)
```
In this example, the Regexp.timeout global configuration is set to 1 second. This means that any regexp match performed in the application will have a timeout of 1 second. If the match takes longer than 1 second to complete, it will raise a Regexp::TimeoutError.
timeout keyword of Regexp.new

This API allows you to specify a timeout for a specific regexp object. This is useful when you want to use different timeout settings for different regexps in your application. For example, you can use it like this:
```
  # Create a regexp with a timeout of 1 second
  regexp = Regexp.new("^[a-zA-Z0-9]+$", timeout: 1.0)

  # Perform a regexp match
  regexp.scan(string)
```
In this example, the timeout keyword is used with the Regexp.new method to specify a timeout of 1 second for the regexp object. This means that any match performed with this regexp object will have a timeout of 1 second. If the match takes longer than 1 second to complete, it will raise a Regexp::TimeoutError.

References

Rails 7.1 supports infinite ranges for Active Record Validators

Athira Kadampatta — Wed, 04 Jan 2023 07:30:00 GMT

Ruby ranges are very handy and efficient when you want to work with a sequence of numbers or characters bound by start and end values. But we could come across situations where we want ranges that either have no end value or start value. Ruby 2.6 introduced the concept of an endless range and later Ruby 2.7 brought in the beginless range.

Infinite Ranges in Ruby

As mentioned in the official Ruby docs, a beginless range and endless range represents a semi-infinite range.
A beginless range would look like (..3) Or (...3) while endless range looks like (3..) Or (3...) and effectively defines the maximum or minimum boundary of the range.

Infinite Ranges in Rails

Infinite ranges are supported in Rails while fetching records from the database as seen below:

Loading development environment (Rails 7.1.0.alpha)
irb(main):013:0> Event.where(date:  (..Time.now))
  Event Load (0.5ms)  SELECT "events".* FROM "events" WHERE "events"."date" <= ? /* loading for pp */ LIMIT ?  [["date", "2022-12-11 12:56:31.866828"], ["LIMIT", 11]]
=>
[#,
 #]

The above query fetched all the past event records, i.e the ones with the date value less than the current time.

Loading development environment (Rails 7.1.0.alpha)
irb(main):014:0> Event.where(date:  (Time.now..))
  Event Load (0.2ms)  SELECT "events".* FROM "events" WHERE "events"."date" >= ? /* loading for pp */ LIMIT ?  [["date", "2022-12-11 12:56:43.438138"], ["LIMIT", 11]]
=>
[#,
 #]

The above query fetched all the future event records, i.e the ones with the date value greater than the current time.

In addition to using ranges for fetching records, we generally use them in Rails with the Active Record length and inclusion validators.

With Rails < 7.1.0-alpha

If we use infinite range in a validator like:

class Event < ApplicationRecord
  ...
  validates_length_of :name, in: (..30)
  ...
end

it throws an error as follows:

Loading development environment (Rails 7.0.1)
irb(main):001:0> Event
Traceback (most recent call last):
        3: from (irb):1
        2: from app/models/event.rb:4:in `'
        1: from app/models/event.rb:7:in `'
RangeError (cannot get the minimum of beginless range)

The only way to validate is by passing maximum or minimum value to the validates method as follows:

validates_length_of :name, maximum: 30, allow_blank: true

With Rails >= 7.1.0-alpha

With the two PRs that were merged recently - PR#45123 and PR#45138, it is possible to pass beginless/endless ranges to the length and inclusivity validators with the in/within options.

  # Allows date to be either today's or any future date but not a past date
  validates_inclusion_of :date, in: -> { (Date.today..) }

  # Allows name to be nil or of a max length of 30 characters
  validates_length_of :name, in: (..30)

  # Mandates description to be of minimum 10 characters with no max limit
  validates_length_of :description, in: (10..)

Please note: These changes are available in the alpha phase of Rails 7.1.0 at the time of writing this article. To get the 7.1.0-alpha version, you have to specify the rails main branch in Gemfile -

    # Gemfile
    gem 'rails', github: 'rails/rails', branch: 'main'

References

Introduction to Ractors in Ruby 3

Navaneeth Krishnan P — Wed, 30 Nov 2022 07:39:07 GMT

Ruby 3 has introduced an experimental feature called Ractors(previously known as Guilds). Ractor is an actor model abstraction for Ruby. It allows us to execute many code blocks parallelly. Before going into Ractors, let's discuss what an actor model is.

What is an actor model?

In the actor model, everything is an actor, like everything is an object in the object-oriented model. Each actor is isolated from other actors. So an actor cannot access the state of another actor, which prevents multiple processes from modifying the same data in concurrent systems. Actors interact with the outside environment using messages. Due to these reasons, the actor model is preferred in concurrent computational systems. Ractor is an implementation of the actor model in ruby.

Creating a Ractor

A Ractor can be created using Ractor.new { }.

3.0.0 :001 > first_ractor = Ractor.new { puts 'Hello world! I am a ractor' }
:267: warning: Ractor is experimental, and the behavior may change in future versions of Ruby! Also there are many implementation issues.
Hello world! I am a ractor
 => #

A Ractor can be also created using a name.

3.0.0 :001 > first_ractor = Ractor.new(name: 'ractor_1') { nil }
 => # 
3.0.0 :002 > first_ractor.name
 => "ractor_1"

Ractors cannot access objects created outside their scope.

3.0.0 :001 > animal = 'Elephant'
 => "Elephant" 
3.0.0 :002 > Ractor.new { puts "#{animal} is a mammal" }
Traceback (most recent call last):
        5: from /Users/user_name/.rvm/rubies/ruby-3.0.0/bin/irb:23:in `'
        4: from /Users/user_name/.rvm/rubies/ruby-3.0.0/bin/irb:23:in `load'
        3: from /Users/user_name/.rvm/rubies/ruby-3.0.0/lib/ruby/gems/3.0.0/gems/irb-1.3.0/exe/irb:11:in `'
        2: from (irb):2:in `'
        1: from :267:in `new'
ArgumentError (can not isolate a Proc because it accesses outer variables (animal).)

But we can pass outside objects as parameters to a ractor.

3.0.0 :001 > animal = 'Elephant'
 => "Elephant" 
3.0.0 :002 > Ractor.new animal do |a|
3.0.0 :003 >   puts "#{a} is a mammal"
3.0.0 :004 > end
Elephant is a mammal
 => #

Messaging in Ractors

Ractors communicate using a messaging system.
There are two types of messaging:

1. Push-type messaging

In this type, the sender knows the receiver of the message. But the receiver does not know the sender.
This is done using send() method to send the message and receive method to receive the message.

3.0.0 :001 > receiver = Ractor.new do
3.0.0 :002 >   message = Ractor.receive
3.0.0 :003 >   puts "received message is #{message}"
3.0.0 :004 > end
 => # 
3.0.0 :005 > message = 'Hi!'
 => "Hi!" 
3.0.0 :006 > receiver.send(message)
received message is Hi!
 => #

Here the receiver has an active incoming port and it will remain active until it receives a message.

2. Pull-type messaging

In this type, the sender does not know the receiver. But the receiver knows from which sender it will receive the message.
This is done using yield() method to send the message and take method to receive the message.

3.0.0 :001 > sender = Ractor.new do
3.0.0 :002 >   message = 'Hi!'
3.0.0 :003 >   Ractor.yield(message)
3.0.0 :004 > end
 => # 
3.0.0 :005 > message = sender.take
 => "Hi!" 
3.0.0 :006 > puts "received message is #{message}"
received message is Hi!
 => nil

Here the sender has an active outgoing port and it will remain active until the message is received. If yield method is not specified, then the last return object is returned.

For thread safety, only immutable objects are regarded as shareable objects in ractors. We can check if an object is shareable or not using Ractor.shareable?() method. Some examples of shareable objects are numbers, boolean, modules, classes, other ractors.
Strings are unsharable since they are mutable. .freeze method can be called on the string to make it immutable and thereby shareable.

3.0.0 :001 > Ractor.shareable?('I am a string')
 => false 
3.0.0 :002 > Ractor.shareable?('I am a string'.freeze)
 => true

We can also make an object shareable using Ractor.make_shareable() method.

3.0.0 :001 > mutable_object = 'I am unshareable'
 => "I am unshareable" 
3.0.0 :002 > Ractor.shareable?(mutable_object)
 => false 
3.0.0 :003 > Ractor.make_shareable(mutable_object)
 => "I am unshareable" 
3.0.0 :004 > Ractor.shareable?(mutable_object)
 => true

If unshareable objects are sent via ractors, a deep copy of the object is first created and then this deep copy is sent.

3.0.0 :001 > receiver = Ractor.new do
3.0.0 :002 >   message = receive
3.0.0 :003 >   puts message.object_id
3.0.0 :004 > end
 => # 
3.0.0 :005 > mutable_object = 'I am unshareable'
 => "I am unshareable" 
3.0.0 :006 > puts mutable_object.object_id
260
 => nil 
3.0.0 :007 > receiver.send(mutable_object)
 => # 
280

In the above example, object_id of the mutable_object changed at the receiver. It is because a deep copy of the mutable_object was created and sent. We can prevent sending deep copies by directly moving an unshareable object to a ractor. This is done by adding move: true parameter in send() method. But the object becomes inaccessible outside the ractor.

3.0.0 :001 > receiver = Ractor.new do
3.0.0 :002 >   message = receive
3.0.0 :003 >   puts message
3.0.0 :004 > end
 => # 
3.0.0 :005 > mutable_object = 'I am unshareable'
 => "I am unshareable" 
3.0.0 :006 > puts mutable_object.inspect
"I am unshareable"
 => nil 
3.0.0 :007 > receiver.send(mutable_object, move: true)
I am unshareable
 => # 
3.0.0 :008 > puts mutable_object.inspect
Traceback (most recent call last):
        5: from /Users/user_name/.rvm/rubies/ruby-3.0.0/bin/irb:23:in `'
        4: from /Users/user_name/.rvm/rubies/ruby-3.0.0/bin/irb:23:in `load'
        3: from /Users/user_name/.rvm/rubies/ruby-3.0.0/lib/ruby/gems/3.0.0/gems/irb-1.3.0/exe/irb:11:in `'
        2: from (irb):8:in `'
        1: from (irb):8:in `method_missing'
Ractor::MovedError (can not send any methods to a moved object)

Example - Movie seat booking using ractors

Let us assume that there are 50 seats for a movie. We can define a ractor seats that will provide movie seats on a first come first serve basis.

3.0.0 :001 > seats = Ractor.new {
3.0.0 :002 >   seat_no = 1
3.0.0 :003 >   50.times do
3.0.0 :004 >     Ractor.yield "Your seat number is #{seat_no}"
3.0.0 :005 >     seat_no += 1
3.0.0 :006 >   end
3.0.0 :007 >   'No more seats'
3.0.0 :008 > }
 => #

Now if a person needs a ticket he can call the take method in seats ractor and it will provide a seat.

3.0.0 :009 > ticket1 = Ractor.new seats do |seats|
3.0.0 :010 >   seat = seats.take
3.0.0 :011 >   puts seat
3.0.0 :012 > end
Your seat number is 1
 => # 
3.0.0 :013 > ticket2 = Ractor.new seats do |seats|
3.0.0 :014 >   seat = seats.take
3.0.0 :015 >   puts seat
3.0.0 :016 > end
Your seat number is 2
 => # 
3.0.0 :017 > ticket3 = Ractor.new seats do |seats|
3.0.0 :018 >   seat = seats.take
3.0.0 :019 >   puts seat
3.0.0 :020 > end
Your seat number is 3
 => #

This way many users can book seats concurrently using ractors. An outgoing port is created in the seats ractor using yield method for each seat. The last return value in the block 'No more seats' is returned if take method is called after all the 50 tickets are sold. If the take method is called again we get a port closed error.

3.0.0 :xxx > ticket51 = Ractor.new seats do |seats|
3.0.0 :xxx >     seat = seats.take
3.0.0 :xxx >     puts seat
3.0.0 :xxx > end
No more seats
 => # 
3.0.0 :xxx > ticket52 = Ractor.new seats do |seats|
3.0.0 :xxx >     seat = seats.take
3.0.0 :xxx >     puts seat
3.0.0 :xxx > end
# terminated with exception (report_on_exception is true):
:694:in `take': The outgoing-port is already closed (Ractor::ClosedError)
        from (irb):xx:in `block in '
 => #

Conclusion

Ractors can be very useful if we want to add concurrency to our code. Each ractor maintains a private state and they use message sharing for communication. So it helps us to avoid concurrency issues like race conditions. But ractors are still an experimental feature in Ruby 3. So it's best to not use them for large-scale concurrent applications. Future versions of ruby might polish this feature more and make it a permanent feature. Until then we can use ractors to write small-scale concurrent code.

References

Selenium 4 CDP Integration With Capybara

Darshan Shah — Tue, 04 Oct 2022 07:30:15 GMT

The Chrome DevTools Protocol (CDP) is a tool for inspecting and debugging Chrome-based browsers. CDP APIs offer abilities to alter network requests, get cookies, delete cache, and do many other operations. These APIs are also useful for automation testing when we need to perform DevTools related operations.

The Selenium CDP network object offers methods to monitor the page's network activity. In this article, we'll look at some operations that can be performed on headers, cookies, and browser cache using Selenium 4 CDP with Capybara and Ruby.

To get started, we will install the selenium-webdriver and the selenium-devtools gems. The version of these gems must match the version of the Chrome browser.

To configure Selenium 4 CDP with Capybara, we add the following steps:

require 'capybara/rspec'
require 'selenium-webdriver'
require 'selenium/devtools'

Selenium::WebDriver::Chrome::Service.driver_path = 'C:\Users\chromedriver.exe'
Capybara.default_driver = :selenium
Capybara.register_driver :selenium do |app|
  Capybara::Selenium::Driver.new(
    app,
    browser: :chrome
  )
end

Headers

Network tracking must be enabled in order to carry out operations related to the network. This can be done by using the network object's enable method. After this, we can attach the header to the HTTP request by passing the header values to the set_extra_http_headers method provided by the network object. The code would look like:

it 'adds header to HTTP request' do
  # Activate network settings
  page.driver.browser.devtools.network.enable

  # Attach headers to the request using setExtraHTTPHeaders
  page.driver.browser.devtools.network.set_extra_http_headers(headers: {
    'header_key_1' => 'header_value_1',
    'header_key_2' => 'header_value_2'
  })

  # Visit the desired page
  visit('https://desired-page-url.com')
end

Cookies

To attach the cookie to the website, visit the required page and pass the cookie parameters using set_cookies method. The code snippet would explain how to attach multiple cookies:

it 'adds cookie to a page' do
  # Visit the desired page
  visit('https://desired-page-url.com')

  # Attach the cookie to the page
  page.driver.browser.devtools.network.set_cookies(cookies: [
    {
       name: 'cookie_1_name',
       value: 'cookie_1_value',
       domain: 'page domain'
    },
    {
       name: 'cookie_2_name',
       value: 'cookie_2_value',
       domain: 'page domain'
    }
  ])
end

get_all_cookies method is used to retrieve all the browser cookies.

it 'gets all cookies present on the browser' do
  # Visit the page
  visit('https://desired-page-url.com')
  # Retrieve all the browser cookies
  # page.driver.browser.devtools.network.get_all_cookies = > {"id" => 5, "result" => {"cookies"=>[{"name"=>"cookie_name", ...}]}}
  cookie_array = page.driver.browser.devtools.network.get_all_cookies['result']['cookies']
end

To retrieve browser cookies for a specific page, get_cookies method can be used by passing the page URL.

page.driver.browser.devtools.network.get_cookies(urls: ['https://desired-page-url.com'])

Cache

Before running an automation test it's always a good idea to clear the browser cache. This stops the browser from utilizing cached versions of websites while performing the tests. Selenium 4 CDP offers the option to clear the browser cache and disable the browser cache to prevent the browser from using the cached copy of the page.

To delete cache for a webpage the clear_browser_cache method is used.

it 'clears browser cache' do
  page.driver.browser.devtools.network.clear_browser_cache
end

The set_cache_disabled method is used to prevent the browser from using cache for a webpage. To use this method, we first enable the network setting similar to the header section. Then we call the set_cache_disabled method by passing the cache_disabled flag as true.

it 'Disable Browser Cache' do
  # Activate network settings
  page.driver.browser.devtools.network.enable
  # Disable broswer cache
  page.driver.browser.devtools.network.set_cache_disabled(cache_disabled: true)
  # Visit the page
  visit('https://desired-page-url.com')
end

This was just a quick rundown of how to use DevTools with Capybara and Selenium Driver. In our project, we intended to add a header to access the web-page, verify scenarios involving cookie presence, and carry out cache-related tasks. Selenium 4 CDP was helpful to us in achieving our objective, and we hope it will be for you as well.

References

Embedding of Scenario Outline with Data Table in Cucumber

Darshan Shah — Fri, 01 Jul 2022 07:30:25 GMT

In one of our automation projects using Cucumber and Capybara, there was a use case where we needed to automate the process of submitting a multi-field form and checking the provided form details on the next page. We considered parameterization using Data Table and Scenario Outline provided by Cucumber. In this article, we'll take a look at why and how we combined Scenario Outline with Data Table to optimize the feature file.

We began with Scenario Outline because we only had four fields: First Name, Last Name, Password, Email, and we wanted to run the same scenario with different datasets. However, we were later given a new requirement that included additional fields such as Location, Age, and Phone Number, increasing the number of input parameters. As a result, the scenario and step definitions became more complex. This is how we can write a test case, which can be easily extended for new field requirements. (For reference we have included only the first four fields in the examples)

Scenario Outline: Validate User Registration Form
  Given I am on the registration page
  When I submit the form with the following details "", "", "", ""
  Then the details should be displayed on my profile page

  Examples:
    |FirstName |LastName |Password  |Email             |
    |John      |Doe      |testPass1 |jdoe@email.com    |
    |Anne      |Smith    |testPass2 |asmith@email.com  |
    |Mike      |Stewart  |testPass3 |mstewart@email.com|

Feature File

Given('I am on the registration page') do
  log 'I am on registration page'
end

When('I submit the form with the following details {string}, {string}, {string}, {string}') do |first_name, last_name, password, email|
  # Storing the input parameters in instance variable to access it in subsequent steps
  @first_name = first_name
  @last_name = last_name
  @email = email

  # Fill the registration form
  fill_in('First Name', with: @first_name)
  fill_in('Last Name', with: @last_name)
  fill_in('Password', with: password)
  fill_in('Email', with: @email)
end

Then('the details should be displayed on my profile page') do
  expect(find('#firstname').value).to eq(@first_name)
  expect(find('#lastname').value).to eq(@last_name)
  expect(find('#email').value).to eq(@email)
end

Step Definition

We considered using Data Table because we needed to send multiple parameters for a scenario. In contrast to Scenario Outline, Data Table passes all datasets in a single step, and to retrieve dataset details we had to loop through the data map. As an alternative to iterating through the data map, we could repeat the scenario as many times as the dataset.

Scenario: Validate User Registration Form for User 1
  Given I am on the registration page
  When I submit the form with the following details
    |FirstName |LastName |Password  |Email         |
    |John      |Doe      |testPass1 |jdoe@email.com|
  Then the details should be displayed on my profile page

Scenario: Validate User Registration Form for User 2
  Given I am on the registration page
  When I submit the form with the following details
    |FirstName |LastName |Password  |Email           |
    |Anne      |Smith    |testPass2 |asmith@email.com|
  Then the details should be displayed on my profile page

Scenario: Validate User Registration Form for User 3
  Given I am on the registration page
  When I submit the form with the following details
    |FirstName |LastName |Password  |Email             |
    |Mike      |Stewart  |testPass3 |mstewart@email.com|
  Then the details should be displayed on my profile page

Feature File

To avoid the repetition of the Scenarios we passed the Reference Header of the Example Table as a parameter to the Data Table.

Scenario Outline: Validate User Registration Form
  Given I am on the registration page
  When I submit the form with the following details
  |FirstName  |LastName  |Password  |Email  |
  |||||
  Then the details should be displayed on my profile page

  Examples:
    |FirstName |LastName |Password  |Email             |
    |John      |Doe      |testPass1 |jdoe@email.com    |
    |Anne      |Smith    |testPass2 |asmith@email.com  |
    |Mike      |Stewart  |testPass3 |mstewart@email.com|

Feature File

In the step definition, we retrieved the input parameter provided as the first row of the Data Map. We used the same hash in the subsequent steps by assigning it to an instance variable.

Given('I am on the registration page') do
  log 'I am on registration page'
end

When('I submit the form with the following details') do |user_details_map|
  # Fetch the user detail passed as the first argument of the data map
  # user_details_map = { hashes: [{"FirstName"=>"John", "LastName"=>"Doe",
  # "Password"=>"testPass1", "Email"=>"jdoe@email.com"}]}
  @user_detail = user_details_map.hashes[0]
  # Fill the registration form
  fill_in('First Name', with: @user_detail['FirstName'])
  fill_in('Last Name', with: @user_detail['LastName'])
  fill_in('Password', with: @user_detail['Password'])
  fill_in('Email', with: @user_detail['Email'])
end

Then('the details should be displayed on my profile page') do
  expect(find('#firstname').value).to eq(@user_detail['FirstName'])
  expect(find('#lastname').value).to eq(@user_detail['LastName'])
  expect(find('#email').value).to eq(@user_detail['email'])
end

Step Definition

In the above example, the scenario will run three times as we have three rows in the Example Table.

By combining the scenario outline with the data table, we were able to DRY our code and make it more readable at the same time.

References

Advanced features provided by the new debug gem in Ruby

Manoj Saun — Thu, 14 Apr 2022 09:54:08 GMT

debug is Ruby's new default debugger included in Ruby 3.1. This new debugger has replaced byebug in Rails 7. Not only does debug provide us with a wide range of functionality, but it also provides some advanced features.

In this article, we will explore and understand a few advanced features of debug.

1. Seamless integration with VSCode

Below steps can be followed to integrate debug gem in VSCode.

Install extension VSCode rdbg Ruby Debugger - Visual Studio Marketplace in your VSCode.
Open the file you want to debug in the VSCode.
Register the breakpoint by clicking on the line you want to set the breakpoint and press the F9 key.
Start the debugging by pressing the F5 key or by selecting Start debugging in the Run menu.

2. Postmortem debugging to debug the dead Ruby process

When you debug a program and an exception is raised, the debugger will exit immediately. To avoid this, we can make use of the postmortem feature.

Let's say we have a simple program to calculate the percentage.

  # computation.rb
  def percentage(numerator, denominator)
    (numerator / denominator) * 100.0
  end

  # call the above function by passing the denominator as 0
  percentage(10, 0)

When you run the program with the debugger and continue the debugging process, the debugger will suspend immediately throwing ZeroDivisionError.

  # start the debugging using `rdbg` command
  19:22:35:~/Users/Examples  >% rdbg computation.rb
  [1, 6] in computation.rb
  =>   1| def percentage(numerator, denominator)
       2|   (numerator / denominator) * 100.0
       3| end
       4|
       5|
       6| percentage(10, 0)
  (rdbg)

  # continue the debugging process
  (rdbg) continue
  Traceback (most recent call last):
    2: from computation.rb:6:in `'
    1: from computation.rb:2:in `percentage'
  computation.rb:2:in `/': divided by 0 (ZeroDivisionError)

  # debugging process gets suspended
  19:54:15:~/Users/Examples

To remain in the debugging mode even after the exception is raised, we can use the postmortem feature. Run the program with the debugger, then set the configuration with the command config set postmortem true.

Start the program in debugging mode.

  # start the debugging using `rdbg` command
  20:02:38:~/Users/Examples  >% rdbg computation.rb
  [1, 6] in computation.rb
  =>   1| def def percentage(numerator, denominator)
       2|   (numerator / denominator) * 100.0
       3| end
       4|
       5|
       6| percentage(10, 0)
 (rdbg)

Enable the postmortem feature.

  # command to enable postmortem
  (rdbg) config set postmortem true
  postmortem = true

Continue the program and the exception will be raised, but the debugger won't be suspended.

  # continue the debugging process
  (rdbg) continue    # continue command
  Enter postmortem mode with #
    computation.rb:2:in `/'
    computation.rb:2:in `percentage'
    computation.rb:6:in `'

  # debugger is still active
    (rdbg:postmortem)

Backtrace the issue with bt command.

  # command to backtrace
  (rdbg:postmortem) bt
  =>#0  [C] Integer#/ at computation.rb:2
    #1  Object#percentage(numerator=10, denominator=0) at computation.rb:2
    #2   at computation.rb:6
  (rdbg:postmortem)

3. Supports record & replay debugging

This feature allows recording the execution information. Using this we can go back to the execution again by using step back command. This will help us to check the last state of the program before the breakpoint.

Let's take a simple example that calculates simple interest. Add a breakpoint at the start of the function.

  # computation.rb
  def calculate_simple_interest
    debugger

    puts 'Enter investment:'
    investment = Integer(gets.chomp)

    puts 'Enter interest rate:'
    interest_rate = Integer(gets.chomp)

    puts 'Enter number of years:'
    time = Integer(gets.chomp)

    interest = (investment * interest_rate * time) / 100
    puts "Interest amount is: #{interest}"
  end

  # call to the above function
  calculate_simple_interest

Run the program and start the debugging.

  # start the debugging using `rdbg` command
  18:09:24:/Users/Examples  >% rdbg computation.rb
  [1, 10] in computation.rb
  =>   1| def calculate_simple_interest
       2|   debugger
       3|
       4|   puts 'Enter investment:'
       5|   investment = Integer(gets.chomp)
       6|
       7|   puts 'Enter interest rate:'
       8|   interest_rate = Integer(gets.chomp)
       9|
      10|   puts 'Enter number of years:'
  =>#0   at computation.rb:1
  (rdbg)

To start the recording, execute the command record on.

# command to start the record
(rdbg) record on
Recorder for #: on (0 records)
(rdbg)

Using next command go to the end of the program and check the values of all the variables using info command. You will be able to see the values of all four variables.

  # command to move the breakpoint to the new line
  (rdbg) next
  [9, 17] in computation.rb
       9|
      10|   puts 'Enter number of years:'
      11|   time = Integer(gets.chomp)
      12|
      13|   interest = (investment * interest_rate * time) / 100
  =>  14|   puts "Interest amount is: #{interest}"
      15| end
      16|

  # command to list the values of the variables
  (rdbg) info
  %self = main
  investment = 12
  interest_rate = 12
  time = 12
  interest = 17
  (rdbg)

Now, to go back to the last state, enter the command step back. Now you will only be able to see values of investment, interest_rate & time.

  # command to move to the previous state
  (rdbg) step back
  [replay] [8, 17] in computation.rb
  [replay]      8|   interest_rate = Integer(gets.chomp)
  [replay]      9|
  [replay]     10|   puts 'Enter number of years:'
  [replay]     11|   time = Integer(gets.chomp)
  [replay]     12|
  [replay] =>  13|   interest = (investment * interest_rate * time) / 100
  [replay]     14|   puts "Interest amount is: #{interest}"
  [replay]     15| end
  [replay]     16|
  [replay]     17| calculate_simple_interest

  # command to list the values of the variables
  (rdbg) info
  [replay] %self = main
  [replay] investment = 12
  [replay] interest_rate = 12
  [replay] time = 12
  [replay] interest = nil
  (rdbg)

You can move back again to the previous state by entering the command step back. Now you will only be able to see the value of investment & interest_rate.

  # command to move to the previous state
  (rdbg) step back
  [replay] [6, 15] in computation.rb
  [replay]      6|
  [replay]      7|   puts 'Enter interest rate:'
  [replay]      8|   interest_rate = Integer(gets.chomp)
  [replay]      9|
  [replay]     10|   puts 'Enter number of years:'
  [replay] =>  11|   time = Integer(gets.chomp)
  [replay]     12|
  [replay]     13|   interest = (investment * interest_rate * time) / 100
  [replay]     14|   puts "Interest amount is: #{interest}"
  [replay]     15| end

  # command to list the values of the variables
  (rdbg) info
  [replay] %self = main
  [replay] investment = 12
  [replay] interest_rate = 12
  [replay] time = nil
  [replay] interest = nil
  (rdbg)

Comparing the performance of the `debug` gem with the other debuggers

We already have existing debuggers like lib/debug.rb, byebug, debase, etc. The reason to use the new debugger is its performance.

Let's have a simple function to find the Fibonacci series.

def fibonacci(n)
  if n < 0
    raise # breakpoint
  elsif n < 2
    n
  else
    fibonacci(n - 1) + fibonacci(n - 2)
  end
end

# running above method in the irb
irb(main):010:0> require 'benchmark'
irb(main):011:0> Benchmark.bm { |x| x.report{ fibonacci(35) } }

Here are the benchmark scores in seconds for various debuggers for the above example.

	Without Breakpoint (sec)	With Breakpoint (sec)
rdbg(debug.gem)	0.92	0.92
Ruby < 3.1	0.93	N/A
RubyMine	0.97	22.6
Byebug	1.23	75.15
old lib/debug.rb	221.88	285.99

Looking at the comparison in the above benchmarks, we can clearly notice that the new debugger is much faster than the rest of the debuggers.

References

Ruby's debug.gem functionality by Koichi Sasada

Rails 7 supports tracking of belongs_to association

Supriya Laxman Medankar — Wed, 06 Oct 2021 07:30:33 GMT

One of the most convenient features of Rails is the ability to track attribute changes. Rails 7 is releasing soon and bringing in a lot of new features as a treat for developers. One of the many features that Rails 7 is introducing, is that we can also track the associated object.

This Rails ActiveRecord specific PR has added two methods for tracking the changes for the belongs_to association. In this article, we will discuss these two new methods with the help of examples.

1. `association_changed?`

The association_changed? method tells if a different associated object has been assigned and the foreign key will be updated in the next save.

Let's take an example of the following Event and Organizer model structure:

class Event
  belongs_to :organizer
end

class Organizer
  has_many :events
end

Before Rails 7, one could only track the change in the target of a belongs_to association by keeping a track of its foreign key. Here's an example:

class Event
  belongs_to :organizer
  before_save :track_change
    
  private
    
  def track_change
    if organizer_id_changed?
      #track something
    end
  end
end

The above example tracks a change in the foreign key (organizer_id), change in the foreign key accordingly means a change in the target of the belongs_to association.

In Rails 7 with the association_changed? method, we can directly check if the associated object is updated or not.

class Event
  belongs_to :organizer
  before_save :track_change
    
  private
    
  def track_change
    if organizer_changed?
      #track something
    end
  end
end

2. `association_previously_changed?`

The association_previously_changed? method tells if the previous save updated the association to reference a different associated object.

So for the above example, we can also check if the organizer association had previously changed using the organizer_previously_changed? method which will return true if the organizer association was changed in the last save.

> event.organizer
=> #

> event.organizer = Organizer.second
=> #

> event.organizer_changed?
=> true

> event.organizer_previously_changed?
=> false

> event.save!
=> true

> event.organizer_changed?
=> false

> event.organizer_previously_changed?
=> true

Similarly, we can track changes on removing the association:

> event.organizer
=> #

> event.organizer = nil
=> true

> event.organizer_changed?
=> true

> event.save!
=> true

> event.organizer_previously_changed?
=> true

Hope these examples help you to understand the new tracking methods Rails 7 is introducing. We can now track the actual association change instead of tracking the id of the associated object.

Thank you for reading! ❤️

References:

1. Rails 7 release notes

2. PR - Add change tracking methods for belongs_to associations

Side effects of Active Record's new feature #invert_where in Rails 7

Manoj Saun — Wed, 22 Sep 2021 07:59:26 GMT

Rails 7 is introducing a new method invert_where that will invert all scope conditions. It allows us to invert an entire where clause instead of manually applying conditions.

We can either chain invert_where to a scope or to a where condition.

class Account
  scope :active, -> { where(active: true) }
end

Account.active.invert_where
=> "SELECT \"accounts\".* FROM \"accounts\" WHERE \"accounts\".\"active\" != 1"

Account.where(active: true).invert_where
=> "SELECT \"accounts\".* FROM \"accounts\" WHERE \"accounts\".\"active\" != 1"

What are the various side effects of using `invert_where`?

1. The invert_where method inverts all the where clauses present in the query

Let's say we create a scope to fetch inactive accounts using invert_where.

class Account
  scope :active, -> { where(active: true) }
  scope :inactive, -> { active.invert_where }
end

Now, somewhere in the controller, we want to fetch all the inactive admin accounts so we may end up with a query as follows:

Account.where(role: 'admin').inactive

We may think that the above query will return all the inactive admin accounts. Let's look at the SQL equivalent of the above code.

=> "SELECT \"accounts\".* FROM \"accounts\" WHERE NOT (\"accounts\".\"role\" = 'admin' AND \"accounts\".\"active\" = 1)"

But instead, we received all the inactive non-admin accounts. The invert_where also inverted the clause where(role: 'admin') internally.

2. The invert_where affects the default scope as well

Continuing the above example, let us add a default scope in the Account model to filter archived accounts (soft-deleted accounts).

class Account
  default_scope { where(archived: false) }
  scope :active, -> { where(active: true) }
end

Now let us write a query to fetch inactive accounts using invert_where.

Account.active.invert_where

The SQL equivalent of the above will be:

=> "SELECT \"accounts\".* FROM \"accounts\" WHERE NOT (\"accounts\".\"archived\" = 0 AND \"accounts\".\"active\" = 1)"

We can see that invert_where has also inverted the default_scope. We wanted to fetch inactive accounts which are not archived but instead received inactive accounts which are archived.

Using default_scope is quite dangerous in the model in the first place, using invert_where will increase the chances of bugs on top of that.

3. Challenges we will face on chaining two or more scopes that contain invert_where

Let's say we have two scopes that contain invert_where as follows:

class Account
  scope :inactive, -> { active.invert_where }
  scope :non_admin, -> { admin.invert_where }
end

Now we want to fetch the inactive non-admin accounts, so we chain the above two scopes together and write the query as follows:

Account.non_admin.inactive

The SQL equivalent of the above is:

=> "SELECT \"accounts\".* FROM \"accounts\" WHERE NOT (\"accounts\".\"role\" != 'admin' AND \"accounts\".\"active\" = 1)"

We can see that the above query will unexpectedly return all the inactive admin accounts.

Substitute for `invert_where`

Rails 7 is still in the alpha phase, invert_where 's behavior might be changed once Rails 7 is officially released. For now, we can make use of the NOT condition to invert the individual WHERE clause.

Let us take the example from the first point and create a scope to fetch inactive accounts using the NOT clause.

class Account
  scope :active, -> { where(active: true) }
  scope :inactive, -> { where.not(active: true) }
end

Now if we want to fetch all the inactive admin accounts, we can write the query as follows:

Account.where(role: 'admin').inactive

The SQL equivalent of the above will be:

=> "SELECT \"accounts\".* FROM \"accounts\" WHERE \"accounts\".\"role\" = 'admin' AND \"accounts\".\"active\" != 1"

The above query will return all the inactive admin accounts. This is the expected behavior we wanted and we achieved it using the NOT clause.

Conclusion

One should cautiously use this proposed method invert_where otherwise we may end up with bugs that may be difficult to debug. What do you think? Do you agree or have any other suggestions?

Thank you for reading.

References

Speeding up Rails 7's Controller Actions using ActiveRecord's #load_async

Krishna Singh — Tue, 22 Jun 2021 11:42:01 GMT

Most of the time in a web application, a single API request consists of multiple database queries. For example:

class DashboardsController < ApplicationController
  def dashboard
    @users = User.some_complex_scope.load_async
    @products = Product.some_complex_scope.load_async
    @library_files = LibraryFile.some_complex_scope.load_async
  end
end

Quoting snippet from load_async PR description from rails repository.

The queries are executed synchronously, which mostly isn’t a huge concern. But, as the database grows larger in size, the response time of requests is getting longer. A significant part of the query time is often just I/O waits. Since these two queries are totally independent, ideally you could execute them in parallel, so that assuming that each take 50ms, the total query time would be 50ms rather than 100ms. In the above scenario, the queries can be executed asynchronously, and the time saved on the I/O waits can be reutilized for the next query execution.

The load_async method will initiate a new thread to execute the query, which will reduce the wait time. If you use the usual lazy loading way in the controller (without the load_async method), the loading will not begin until the view references it. But in the case of load_async, loading will start before it is referenced in the view.

Configuration for `load_async`

We need to set the value for async_query_executor in the environment configuration for load_async to work. By default, Rails initializes async_query_executor to nil. The query runs in the foreground if load_async is called and executor is not defined. async_query_executor comes with two configuration options.

:global_thread_pool is used to create a common thread pool for all database connections.

# config/environments/development.rb

config.active_record.async_query_executor = :global_thread_pool

:multi_thread_pool is used to create a unique thread pool for each database connection.

# config/environments/development.rb

config.active_record.async_query_executor = :multi_thread_pool

References

Everything You Need to know about Serialization in Ruby on Rails - Part III

Athira Kadampatta — Fri, 18 Jun 2021 06:07:26 GMT

We had previously talked about the Serialization formats and How Serialization is implemented for storing objects in the relational database in the first two parts of the blog series. This article focuses on the various Serializers that prepare and construct API transferable data in Ruby on Rails.

Everything You Need to know about Serialization in Rails: Part I

It was the day we were moving. I was observing how the “Packers and Movers”professionals packed our furniture. For example, the King size bed shown belowhad to be accommodated within a space of about 6-7 inches inside a van. While Ikept wondering how they’d manage this, they dismantled the bed. A…

Kiprosh BlogsAthira Kadampatta

Serialization in Ruby on Rails for Storage: Part II

Rails framework allows complex objects to be stored in a DB column via the ActiveRecord::Serialization module.This article explains when and how to do it

Kiprosh BlogsAthira Kadampatta

Serialization in Rails for APIs

What is so special about the data rendered by the APIs? Do we really need to serialize it? The short answer is "YES", we need to send a standardized and formatted response to the API consumers in a way they would be able to parse. Otherwise, it would be difficult for the API consumers to make sense of the transferred data. Since most of the API consumers are JavaScript frameworks, the preferred serialization format for APIs in Rails is JSON.

How to send JSON response from Rails APIs?

For simple use cases, we could use render json: object in the API controller action. Rails internally applies to_json to render the response. Alternatively, we could apply the JSON.dump or to_json methods on our objects and send it in the response.

Let's say we are building a basic e-learning site. So a user has many courses, which in turn has many topics and each topic has many assignments.

To send all courses of the authenticated user, our courses_controller's index action looks somewhat like this:

class Api::V1::CoursesController < ApplicationController
  def index
    render json: current_user.courses
  end
end

[
    {
        "id": 24,
        "title": "Javascript Training",
        "description": "Master JavaScript With The Most Complete Course On The Market.",
        "created_at": "2021-05-22T17:31:59.856Z",
        "updated_at": "2021-05-22T17:31:59.856Z"
    },
    {
        "id": 25,
        "title": "Ruby on Rails",
        "description": " Learn to make innovative web apps with Ruby on Rails and unleash your creativity",
        "created_at": "2021-05-22T17:31:59.886Z",
        "updated_at": "2021-05-22T17:31:59.886Z"
    }
]

Response for courses#index.json

This is so easy, isn't it? Or is it? Just think of it, would the API consumer expect us to simply send the title and description of the course? What about the topics, or assignments?

Let us use the include option for the to_json method to render the nested objects.

class Api::V1::CoursesController < ApplicationController
  def index
    render json: current_user.courses.to_json(include: { topics: { include: :assignments } })
  end
end

[
    {
        "id": 24,
        "title": "Javascript Training",
        "description": "Master JavaScript With The Most Complete Course On The Market.",
        "created_at": "2021-05-22T17:31:59.856Z",
        "updated_at": "2021-05-22T17:31:59.856Z",
        "topics": [
            {
                "id": 159,
                "name": "Variables and flow control",
                "description": "You will learn the fundamentals of JavaScript",
                "course_id": 24,
                "created_at": "2021-05-22T17:31:59.908Z",
                "updated_at": "2021-05-22T17:31:59.908Z",
                "assignments": [
                    {
                        "id": 157,
                        "description": "Declare a variable called myName and initialize it with a value, on the same line.",
                        "topic_id": 159,
                        "created_at": "2021-05-22T17:31:59.962Z",
                        "updated_at": "2021-05-22T17:31:59.962Z"
                    }
                ]
            }
        ]
    },
    {
        "id": 25,
        "title": "Ruby on Rails",
        "description": " Learn to make innovative web apps with Ruby on Rails and unleash your creativity",
        "created_at": "2021-05-22T17:31:59.886Z",
        "updated_at": "2021-05-22T17:31:59.886Z",
        "topics": [
            {
                "id": 160,
                "name": "Serializers in Rails",
                "description": "You will learn all the API Serializers in Ruby on Rails",
                "course_id": 25,
                "created_at": "2021-05-22T17:31:59.920Z",
                "updated_at": "2021-05-22T17:31:59.920Z",
                "assignments": [
                    {
                        "id": 158,
                        "description": "Render a basic json response from an action controller",
                        "topic_id": 160,
                        "created_at": "2021-05-22T17:31:59.975Z",
                        "updated_at": "2021-05-22T17:31:59.975Z"
                    }
                ]
            }
        ]
    }
]

Response for courses#index.json

There it is!! We got all the required data in JSON format.
Observe the response. Don't you think sending the created_at, updated_at, user_id etc. in each array were perhaps not needed? Okay, maybe that's not a big deal - we can limit the response attributes by passing only / except options to the to_json method.

Why do we need JSON serializers?

Well, the real world is not so straightforward as you see. The API consumer would now expect conditional responses, like not sending the assignments data until the corresponding topic has been completed.

We'd then probably create a hash of the course details and render the hash as a JSON object.

class Api::V1::CoursesController < ApplicationController
  def index
    array_of_courses = current_user.courses.map(&:attributes)
    current_user.courses.each_with_index do |course, index|
      array_of_topics = course.topics.map(&:attributes)
      course.topics.each_with_index do |topic, index|
        array_of_topics[index]["is_completed"] = topic.is_completed?(current_user)

        array_of_topics[index]["assignments"] = topic.assignments if topic.is_completed?(current_user)
      end
      array_of_courses[index]["topics"] = array_of_topics
    end
    render json: array_of_courses
  end
end

[
    {
        "id": 24,
        "title": "Javascript Training",
        "description": "Master JavaScript With The Most Complete Course On The Market.",
        "created_at": "2021-05-22T17:31:59.856Z",
        "updated_at": "2021-05-22T17:31:59.856Z",
        "topics": [
            {
                "id": 159,
                "name": "Variables and flow control",
                "description": "You will learn the fundamentals of JavaScript",
                "course_id": 24,
                "created_at": "2021-05-22T17:31:59.908Z",
                "updated_at": "2021-05-22T17:31:59.908Z",
                "is_completed": true,
                "assignments": [
                    {
                        "id": 157,
                        "description": "Declare a variable called myName and initialize it with a value, on the same line.",
                        "topic_id": 159,
                        "created_at": "2021-05-22T17:31:59.962Z",
                        "updated_at": "2021-05-22T17:31:59.962Z"
                    }
                ]
            }
        ]
    },
    {
        "id": 25,
        "title": "Ruby on Rails",
        "description": " Learn to make innovative web apps with Ruby on Rails and unleash your creativity",
        "created_at": "2021-05-22T17:31:59.886Z",
        "updated_at": "2021-05-22T17:31:59.886Z",
        "topics": [
            {
                "id": 160,
                "name": "Serializers in Rails",
                "description": "You will learn all the API Serializers in Ruby on Rails",
                "course_id": 25,
                "created_at": "2021-05-22T17:31:59.920Z",
                "updated_at": "2021-05-22T17:31:59.920Z",
                "is_completed": false
            }
        ]
    }
]

Response for api/v1/courses#index

Doesn't this look very messy? Even with refactoring, the above code is difficult to maintain and has cluttered our controller. For such scenarios instead of manipulating data manually, how about we choose among the various serializers available in the form of gems. These serializers make it easier to build serialized JSON data. They provide wrapper methods that can help us keep the code well-structured and enable customization.

There are different serializer gems available for us to choose from. Out of them we shall explore the four most popular ones and compare them.

Jbuilder

Jbuilder is provided by Rails(>= 5) as the default serializer and is based on the approach that construction of JSON data belongs to the View layer. This gem provides a DSL to customize the responses. The files are placed under the app/views directory with the extension of .json.jbuilder. Jbuilder is quite intuitive and helpful when you need to send complex responses with a lot of data manipulations.

class Api::V1::CoursesController < ApplicationController
  helper_method :current_user
  
  def index
    @courses = current_user.courses
  end
end

# /app/views/api/v1/courses/index.json.jbuilder

json.array! @courses do |course|
  json.extract! course, :id, :title, :description
    json.topics course.topics, partial: 'api/v1/courses/topic', as: :topic
  end
end

# /app/views/api/v1/courses/_topic.json.jbuilder

json.extract! topic, :id, :name, :description
if topic.is_completed?(current_user)
  json.assignments topic.assignments, partial: 'api/v1/courses/assignment', as: :assignment
  end
end

# /app/views/api/v1/courses/_assignment.json.jbuilder

json.extract! assignment, :id, :description

[
    {
        "id": 24,
        "title": "Javascript Training",
        "description": "Master JavaScript With The Most Complete Course On The Market.",
        "topics": [
            {
                "id": 159,
                "name": "Variables and flow control",
                "description": "You will learn the fundamentals of JavaScript",
                "is_completed": true,
                "assignments": [
                    {
                        "id": 157,
                        "description": "Declare a variable called myName and initialize it with a value, on the same line."
                    }
                ]
            }
        ]
    },
    {
        "id": 25,
        "title": "Ruby on Rails",
        "description": " Learn to make innovative web apps with Ruby on Rails and unleash your creativity",
        "topics": [
            {
                "id": 160,
                "name": "Serializers in Rails",
                "description": "You will learn all the API Serializers in Ruby on Rails",
                "is_completed": false
            }
        ]
    }
]

Response for api/v1/courses#index

Ruby Benchmark Report

    user     system      total        real 
0.875491   0.035895   0.911386 	(1.000073)

ActiveModel::Serializers:

If you do not wish to use a DSL and are more comfortable with treating serializers like an ActiveRecord object, then you may go with the Active Model Serializers. It is based on Rails conventions for generating data and extracts the serialization logic into a separate folder /app/serializers.

# /app/serializers/course_serializer.rb

class CourseSerializer < ActiveModel::Serializer
  attributes :id, :title, :description

  has_many :topics
  has_many :user_courses
end

# /app/serializers/topic_serializer.rb

class TopicSerializer < ActiveModel::Serializer
  attributes :id, :name, :description

  belongs_to :course
  has_many :assignments, if: -> { object.is_completed?(current_user) }
  
  attribute :is_completed do
    object.is_completed?(current_user)
  end
end

# /app/serializers/assignment_serializer.rb

class AssignmentSerializer < ActiveModel::Serializer
  attributes :id, :description

  belongs_to :topic
end

Now that we have defined the serializers for our models and their associations, all we have to do in the controller action is this:

class Api::V1::CoursesController < ApplicationController
  def index
    render json: current_user.courses, include: "topics.assignments"
  end
end

Please refer to this documentation to know more about how we can include double nested associations.

The result would be something like this:

[
    {
        "id": 24,
        "title": "Javascript Training",
        "description": "Master JavaScript With The Most Complete Course On The Market.",
        "topics": [
            {
                "id": 159,
                "name": "Variables and flow control",
                "description": "You will learn the fundamentals of JavaScript",
                "is_completed": true,
                "assignments": [
                    {
                        "id": 157,
                        "description": "Declare a variable called myName and initialize it with a value, on the same line."
                    }
                ]
            }
        ]
    },
    {
        "id": 25,
        "title": "Ruby on Rails",
        "description": " Learn to make innovative web apps with Ruby on Rails and unleash your creativity",
        "topics": [
            {
                "id": 160,
                "name": "Serializers in Rails",
                "description": "You will learn all the API Serializers in Ruby on Rails",
                "is_completed": false
            }
        ]
    }
]

Response for courses#index.json

NEAT, isn't it?

Ruby Benchmark Report

    user     system      total        real  
0.340994   0.008847   0.349841 	(0.483862)

Fast JSON API

Fast JSON API is similar to AMS but is much faster. While AMS lets you configure the adapter to be used, Fast JSON API follows the JSON API specifications.
This serializer is not maintained anymore, but we can use its forked version, i.e the jsonapi-serializer as mentioned in their Github page. I have used FastJsonapi::ObjectSerializer in the examples below, but we can replace it with JSONAPI::Serializer too.
The JSON structure may be a bit uncomprehensible specifically for nested relations, but that is how the specifications demand it to be.

# /app/serializers/course_serializer.rb

class CourseSerializer
  include FastJsonapi::ObjectSerializer

  attributes :id, :title, :description
  attribute :topics do |object, params|
    TopicSerializer.new(object.topics, { params: { current_user: params[:current_user] } })
  end
end

# /app/serializers/topic_serializer.rb

class TopicSerializer
  include FastJsonapi::ObjectSerializer
  attributes :id, :name, :description

  has_many :assignments, if: Proc.new { |topic, params| topic.is_completed?(params[:current_user]) }

  attribute :is_completed do |topic, params|
    topic.is_completed?(params[:current_user])
  end
end

# /app/serializers/assignment_serializer.rb

class AssignmentSerializer
  include FastJsonapi::ObjectSerializer

  attributes :id, :description
end

class Api::V1::CoursesController < ApplicationController
  def index
    render json: CourseSerializer.new(current_user.courses, { params: { current_user: current_user }, include: [:'topics.assignments'] })
  end
end

{
    "data": [
        {
            "id": "24",
            "type": "course",
            "attributes": {
                "id": 24,
                "title": "Javascript Training",
                "description": "Master JavaScript With The Most Complete Course On The Market."
            },
            "relationships": {
                "topics": {
                    "data": [
                        {
                            "id": "159",
                            "type": "topic"
                        }
                    ]
                }
            }
        },
        {
            "id": "25",
            "type": "course",
            "attributes": {
                "id": 25,
                "title": "Ruby on Rails",
                "description": " Learn to make innovative web apps with Ruby on Rails and unleash your creativity"
            },
            "relationships": {
                "topics": {
                    "data": [
                        {
                            "id": "160",
                            "type": "topic"
                        }
                    ]
                }
            }
        }
    ],
    "included": [
        {
            "id": "157",
            "type": "assignment",
            "attributes": {
                "id": 157,
                "description": "Declare a variable called myName and initialize it with a value, on the same line."
            }
        },
        {
            "id": "159",
            "type": "topic",
            "attributes": {
                "id": 159,
                "name": "Variables and flow control",
                "description": "You will learn the fundamentals of JavaScript",
                "is_completed": true
            },
            "relationships": {
                "assignments": {
                    "data": [
                        {
                            "id": "157",
                            "type": "assignment"
                        }
                    ]
                }
            }
        },
        {
            "id": "160",
            "type": "topic",
            "attributes": {
                "id": 160,
                "name": "Serializers in Rails",
                "description": "You will learn all the API Serializers in Ruby on Rails",
                "is_completed": false
            },
            "relationships": {}
        }
    ]
}

Response for courses#index.json

Ruby Benchmark Report

    user     system      total        real  
0.468393   0.018134   0.486527 	(0.530162)

JSONAPI-RB

Similar to Fast JSON, JSON API offers a bunch of lightweight modules and wrappers to make Ruby applications abide by the JSON-API Specifications

We will use jsonapi-rails , a gem that provides the required helpers to generate JSON-API conformant responses.

# /app/resources/serializable_course.rb

class SerializableCourse < JSONAPI::Serializable::Resource
  type 'courses'

  attributes :title, :description

  belongs_to :user
  has_many :topics
end

# /app/resources/serializable_topic.rb

class SerializableTopic < JSONAPI::Serializable::Resource
  extend JSONAPI::Serializable::Resource::ConditionalFields
  type 'topics'
  
  attributes :name, :description

  belongs_to :course
  has_many :assignments, if: -> { @object.is_completed?(@user) }
  
  attribute :is_completed do ||
    @object.is_completed?(@user)
  end
end

# /app/resources/serializable_assignment.rb

class SerializableAssignment < JSONAPI::Serializable::Resource
  type 'assignments'

  attributes :description

  belongs_to :topic
end

class Api::V1::CoursesController < ApplicationController
  def index
    render jsonapi: current_user.courses,
           expose: { user: current_user },
           include: ["topics", "topics.assignments"],
           fields: { courses: [:title, :description, :topics],
                     topics: [:name, :description, :assignments],
                     assignments: [:description] }
  end
end

{
    "data": [
        {
            "id": "24",
            "type": "courses",
            "attributes": {
                "title": "Javascript Training",
                "description": "Master JavaScript With The Most Complete Course On The Market."
            },
            "relationships": {
                "topics": {
                    "data": [
                        {
                            "type": "topics",
                            "id": "159"
                        }
                    ]
                }
            }
        },
        {
            "id": "25",
            "type": "courses",
            "attributes": {
                "title": "Ruby on Rails",
                "description": " Learn to make innovative web apps with Ruby on Rails and unleash your creativity"
            },
            "relationships": {
                "topics": {
                    "data": [
                        {
                            "type": "topics",
                            "id": "160"
                        }
                    ]
                }
            }
        }
    ],
    "included": [
        {
            "id": "159",
            "type": "topics",
            "attributes": {
                "name": "Variables and flow control",
                "description": "You will learn the fundamentals of JavaScript",
                "is_completed": true
            },
            "relationships": {
                "course": {
                    "meta": {
                        "included": false
                    }
                },
                "assignments": {
                    "data": [
                        {
                            "type": "assignments",
                            "id": "157"
                        }
                    ]
                }
            }
        },
        {
            "id": "160",
            "type": "topics",
            "attributes": {
                "name": "Serializers in Rails",
                "description": "You will learn all the API Serializers in Ruby on Rails",
                "is_completed": false
            },
            "relationships": {
                "course": {
                    "meta": {
                        "included": false
                    }
                }
            }
        },
        {
            "id": "157",
            "type": "assignments",
            "attributes": {
                "description": "Declare a variable called myName and initialize it with a value, on the same line."
            },
            "relationships": {
                "topic": {
                    "meta": {
                        "included": false
                    }
                }
            }
        },
        {
            "id": "158",
            "type": "assignments",
            "attributes": {
                "description": "Render a basic json response from an action controller"
            },
            "relationships": {
                "topic": {
                    "meta": {
                        "included": false
                    }
                }
            }
        }
    ],
    "jsonapi": {
        "version": "1.0"
    }
}

Response for courses#index.json

Ruby Benchmark Report

    user     system      total        real  
0.418128   0.022805   0.440933 	(0.495260)

Which one to choose?

We have seen a few of the many options out there that can be used for serializing API data in Ruby on Rails applications. Many others are not covered here like RABL, Blueprinter, etc. So now the question arises which serializer should you choose for your application?
Here is a good blog that shares reasons and use-cases where a specific serializer would fit best. To summarize that blog, choose Jbuilder if you need a lot of customized rendering and like the responses to remain at the view level. If you expect your responses to be consistent with the database design go with AMS. If you are a stickler and would like to follow the JSON:API specifications exactly, go with FastJSON-API/JSONAPI-RB

If you wish to see the above examples in action and play around with them, here is a link to the Github repository that I created.

Conclusion

Creating JSON responses in Ruby on Rails might look easy, but with the array of different serializers available, you might be spoilt for choice. The best way to take a decision would be to know who are the potential consumers of your APIs and analyze the expected response. I would also suggest taking a look at their documentation to see which serializer would satisfy all your needs without much overriding.

Everything that we deal with in the inter-connected world today is nothing but "data". Serialization that deals with the translation/storage of this data have become all the more important with the evolution of API-driven applications.

I hope this series of articles dealing with Serialization in Ruby on Rails helps you in choosing the right serialization format and tools for storing and transferring data.

Thank you for reading!

References

Rails Parameters Parsing and the Case of Parameters Corruption

Ayush Billore — Fri, 09 Apr 2021 07:18:09 GMT

Rails is a developer-friendly web application framework that enables developers to do more with less code, but it isn’t always clear exactly what’s going on under the covers. One area where I’ve had a hard time was understanding how Rails parses the Request query parameters and the Form variables.

So what's a query parameter?

Query parameters are an optional set of key-value pairs that appear after the question mark in the URL. For example, name=contra is a query parameter in https://example.com/over/there?name=contra.

And what's a form variable?

When we submit a form on the web, the form sends data to the server as form variables. For example, player[game_attributes][name] and player[game_attributes][release_year] are form variables in the snippet below.


  Task Name: 
  Task Duration:

Parameters Parsing

Several server-side frameworks are designed to handle form data if the inputs are named in a certain way. For e.g., in Rails, if you have an input named user[name], and is submitted as a part of a form, then the server would parse and convert it to a Hash as such: { "user" => { "name" => } }. This allows the server program to access the input value params["user"]["name"], where params are the variable through which the parsed Hash is accessible.

Rails uses Rack for parameters parsing, so it’s the same even for other Ruby frameworks such as Sinatra and Padrino.

What's happening under the covers?

The function in Rack that’s doing all the work in generating the magical params is parse_nested_query. The documentation is not very comprehensive, so we can experiment a bit to figure out what’s going on under the covers.

Here's what the method looks like:

def parse_nested_query(qs, d = '&;')
  params = {}
  (qs || '').split(/[#{d}] */n).each do |p|
    k, v = unescape(p).split('=', 2)
    normalize_params(params, k, v)
  end
  return params
end

Experimental Setup

Since Rack is written in ruby, we can pop open the irb and do the initial setup.

ayush-kiproshs-MacBook ~ $ irb
2.7.0 :001 > require "rack"
 => true
2.7.0 :002 > def parse(query_string)
2.7.0 :003 >   Rack::Utils.parse_nested_query(query_string)
2.7.0 :004 > end
 => :parse

We need to require Rack and create a parse method for our ease of experimentation. Now as we are ready with the initial setup let's begin with the cases.

Case 1: When the fields are named without square brackets

If there are no square brackets, the value directly gets assigned to the parameter.

2.6.6 :013 > query_string = "game_1=contra&game_2=mario"
2.6.6 :014 > parse(query_string)
 => {"game_1"=>"contra", "game_2"=>"mario"}

Case 2: When using square brackets to name attributes

If the parameter is enclosed under square brackets, the value gets assigned to the parameter in a nested structure.

2.6.6 :015 > query_string = "player[game]=contra"
2.6.6 :016 > parse(query_string)
 => {"player"=>{"game"=>"contra"}}

Case 3: When the field name ends with empty square brackets

If the field name is ending with empty square brackets, the parameter is treated as an empty array, and values get appended to it.

2.7.0 :017 > query_string = "player[games][]=contra&
                             player[games][]=mario"
2.7.0 :018 > parse(query_string)
 => {"player"=>{"games"=>["contra", "mario"]}}

Case 4: When parsing objects(Combination of the above three cases)

On analysing the above cases we know that Rails reads the query parameters from left to right in the query-string, and creates a new object each time it sees a repeated attribute.

2.7.0 :021 > query_string = "player[games][][name]=contra&
                             player[games][][release_year]=1987&
                             player[games][][name]=mario&
                             player[games][][release_year]=1983"
2.7.0 :022 > parse(query_string)
 => {"player"=>{"games"=>[{"name"=>"contra", "release_year"=>"1987"}, {"name"=>"mario", "release_year"=>"1983"}]}}

Case 5: Case of Parameters Corruption

We know from the above cases that a new parameter is created each time the parser encounters a repeated attribute. But if the order of query parameters in the query string is not correct, the parser will return a corrupted params object. See examples below for better understanding.

2.7.0 :023 > query_string = "player[games][][name]=contra&
                             player[games][][name]=mario&
                             player[games][][release_year]=1987&
                             player[games][][release_year]=1983"
2.7.0 :024 > parse(query_string)
 => {"player"=>{"games"=>[{"name"=>"contra"}, {"name"=>"mario", "release_year"=>"1987"}, {"release_year"=>"1983"}]}}

Parameters in query parsing are strictly order-dependent. And it is guaranteed that the browsers will send the form parameters in the same order in which they appear in the source.

Refer W3C specification to learn about browser standards.

multipart/form-data The parts are sent to the processing agent in the same order the corresponding controls appear in the document stream. Part boundaries should not occur in any of the data; how this is done lies outside the scope of this specification.

References

Rate limiting using Redis in a Rails app

Prayesh Shah — Thu, 25 Mar 2021 06:17:36 GMT

The web is a weird place. You go to sleep thinking that you have a perfectly functional web application and the next day when you wake up, you might find yourself staring at a sudden huge spike in the number of requests. Either your app got popular overnight or you were just a victim of a DOS attack trying to bring your app server down. Usually, it's the latter.

There are some popular gems like rack-attack and rack-throttle which work quite well and provides a lot of flexibility. But if you're looking to write your custom logic with minimum dependencies, then continue reading.

We will create a middleware that intercepts and blocks any host which tries to overload our servers by firing too many requests within a short timespan. We will be using Redis to store the count of requests from each IP address.

Let's start by writing the most basic middleware.

# app/lib/middlewares/custom_rate_limit.rb
class CustomRateLimit
  def initialize(app)
    @app = app
  end

  def call(env)
    @app.call(env)
  end
end

Add the following line inside your application.rb:

# For Rails version > 5
config.middleware.use CustomRateLimit

If you're using Rails version lesser than 5, add the following:

# For Rails version < 5
config.middleware.use 'CustomRateLimit'

Run the bin/rails middleware command and verify that our custom middleware is present in the middleware stack.
In case you run into an error saying uninitialized constant ::CustomRateLimit, add the below line at the top of your application.rb:

require_relative '../lib/middlewares/custom_rate_limit'

Now, let's install the redis gem by adding it to our Gemfile.

gem 'redis'

Also, ensure that you have the Redis server installed on your local system. If not, install it by following the steps mentioned here.

Note: As Redis is an in-memory database, it's state is not persistent. In other words, if your redis server were to go down due to an outage, you'd lose your data. By default, Redis saves snapshots of the dataset on the disk, in a binary file called dump.rdb. Refer redis persistence for more details.

To initialize redis in your app, add the following file inside the config/initializers:

# config/initializers/redis.rb
require 'redis'

REDIS = Redis.new(url: ENV.fetch('REDIS_URL'))

For development environment, the value of REDIS_URL will be redis://localhost:6379.

Now, let's edit our middleware and add some logic.

def call(env)
  if should_allow?(env)
    @app.call(env)
  else
    request_quota_exceeded
  end
end

The should_allow? function will look something like this:

def should_allow?(env)
  key = "IP:#{env['action_dispatch.remote_ip']}"

  REDIS.set(key, 0, nx: true, ex: TIME_PERIOD)
  REDIS.incr(key) > LIMIT ? false : true
end

We will use the user's IP address as the key and store the request count as the value.
The Redis#set method will set the record in the redis store. We will pass it the following arguments:

key - a unique identifier, which in this case will be the user's IP address
value - sets the value against the given key
ex - sets the expiry time in seconds
nx - sets the key only if it doesn't already exist

On every request, we increment the count using Redis#incr. If the count exceeds the predefined limit, we return false.

Define the constants in the same file and update the values as per your needs.

TIME_PERIOD = 60 # no. of seconds
LIMIT = 20 # no. of allowed requests per IP for unauthenticated user

If you are using Warden based authentication like Devise, and don't want to throttle authenticated requests, add the following guard condition to the should_allow? function.

return true if env['rack.session']['warden.user.user.key'].present?

This will allow all authenticated requests to pass through without any rate limits.

The request_quota_exceeded method will look something like this:

def request_quota_exceeded
  [ 429, {}, ['Too many requests fired. Request quota exceeded!'] ]
end

The HTTP status code 429 indicates Too Many Requests to the server in a given period.

Our middleware will finally look something like this:

# lib/middlewares/custom_rate_limit.rb
class CustomRateLimit
  TIME_PERIOD = 60 # no. of seconds
  LIMIT = 20 # no. of allowed requests per IP for unauthenticated user

  def initialize(app)
    @app = app
  end

  def call(env)
    if should_allow?(env)
      @app.call(env)
    else
      request_quota_exceeded
    end
  end

  private
  def should_allow?(env)
    return true if env['rack.session']['warden.user.user.key'].present?

    key = "IP:#{env['action_dispatch.remote_ip']}"

    REDIS.set(key, 0, nx: true, ex: TIME_PERIOD)
    REDIS.incr(key) > LIMIT ? false : true
  end

  def request_quota_exceeded
    [ 429, {}, ['Too many requests fired. Request quota exceeded!'] ]
  end
end

Hope this blog was helpful.
Thank you!

Useful links:

Redis commands
Basic Rate Limiting

Sync files to a remote location using rsync linux command and Ruby

Supriya Laxman Medankar — Wed, 17 Mar 2021 06:47:02 GMT

Note: The entire approach highlighted in this blog about using the rsync command with Ruby is only applicable on Linux-based systems and won't be available on Windows.

In this article, we will explore how to use rsync in Ruby to sync files from local to a remote location, how to improve its error logging and how rsync is better than scp.

Let's first understand what is `rsync`:

rsync is a Linux command to sync files from one location to another. We can run this command on the terminal and it will copy files from one directory to another, either locally or even on a remote location.

scp is another Linux command which allows file transfer from one location to another similar to rsync. You can refer to this article about scp to learn more about it.

Why we preferred `rsync` over `scp`?

There are some advantages of using rsync over scp, these are:

rsync uses a special delta transfer algorithm along with a few optimizations which makes it faster as compared to scp which performs a plain transfer
rsync provides the option to preserve the partially downloaded files. In case if your download session gets interrupted in the middle of copying a file, you can resume it from where you left off
rsync automatically detects if the file has been transferred correctly

I found this Stack Overflow post helpful while comparing scp to rsync.

How to use `rsync` command?

rsync [options] [source] [destination]

Following are the options which we can use with rsync:

-v : verbose
-r : recursively copies data. It doesn’t preserve timestamps and permission while transferring data
-a : archive mode, it allows copying files recursively and it also preserves symbolic links, file permissions, user & group ownerships and timestamps
-z : compress file data
-h : human-readable, output numbers in a human-readable format
-e : specify type of protocol

The above options (and more) can also be accessed by running rsync -h in the console.
We can also combine multiple options like rsync -avhe [source] [destination] as per our need.

How `rsync` works with `SSH`?

It is important to transfer data on a secure connection to ensure that nobody can read data while it is being transferred. And we can achieve this by using rsync with SSH (Secure Shell) protocol.
As we can see in the above options list, -e option is used to specify the type of protocol we are using to copy/sync the files to/from a remote location.

So with ssh rsync command will be like this:

$ rsync -e ssh /local_dir user@remote-host:/remote_dir/

We can also change the permission of file while copying/syncing files from/to a remote location

$ rsync -avhe ssh --chown=USER:GROUP /local_dir user@remote-host:/remote_dir/`

The above command will sync all the files present in directory /local_dir owned by USER with group GROUP to the /remote_dir directory on the remote-host.

How to use `rsync` in Ruby?

Till now we saw how we can run rsync command in Linux/ Unix terminal. Now let's see how to use this command in Ruby to sync files to a remote location.

There are multiple ways to call shell commands from Ruby, some of which are as follows:

Kernel#` i.e. backticks
It returns the result of shell command

  `echo "Hello World!"`       #=> "Hello World!\n"
  cmd = "echo 'Hello World!'"
  value = `#{cmd}`            #=> "Hello World!\n"

Built-in syntax, %x( cmd )
It also returns the result of shell command similar to backticks. In the syntax following %x is a delimiter, we can use any character as a delimiter. For example (, {, [ or < . For more details refer to this Stack Overflow post.
```
  %x( echo 'Hello World!')  #=> "Hello World!\n"
```
Kernel#exec
It exits the current process and executes the shell command in the new process.
In the following example, exec exits the ruby console and executes the echo command.
```
  exec("echo 'Hello World!'")  #=> It exits ruby console and prints "Hello World" on terminal
```

Kernel#system
Executes the given command in a sub-shell.
On successful execution of command, it returns true otherwise false.

  system('invalid command')                 #=> nil
  system('echo system command executed')    #=> true
  system('echo new command | grep another') #=> false

For more information, refer to this rubyguide.

We will use the system command for error tracking since only this command echoes the STDOUT with true/false/nil according to the result.

Let's create a method using Kernel#system to execute rsync command:

def sync_files
  cmd = 'rsync -e ssh /local_dir user@remote-host:/remote_dir/'

  system cmd
end

In the above method, string cmd is the rsync command which we want to execute on the terminal. We have passed it to the system method. All seems to be good so far, right?

Wait, what happens if the rsync command fails due to unexpected errors, such as the server was out of service?

How to track `system` errors in Ruby?

First, we will understand what the system method returns:
The system returns true if the command gives zero exit status, false for non zero exit status. Returns nil if command execution fails.

We can check the error status using $?. This returned object of class Process::Status with process id and exit code. For example:

#

We can also use $?.success? to check the status which returns true if the last operation was a success.

raise "rsync failed with status code #{status}" unless $?.success?

Let's modify the above sync_files method to track errors

def sync_files
  system 'rsync -e ssh /local_dir user@remote-host:/remote_dir/'

  status = $?.exitstatus

  raise "rsync failed with status code #{status}" unless status.zero?
end

Now, the above method will raise an error whenever rsync command fails to execute due to any issues.

I hope you enjoyed this article and learned how to use rsync in Ruby. Thank you.

ruby - Kiprosh Blogs

Type-checking in Ruby 3 using RBS

Why do we need type-checking?

How to write an RBS file?

How to run an RBS file?

Setup steep gem

Type-checking using Steep

TypeProf

Conclusion

References

All about "Data" Simple Immutable Value Objects in Ruby 3.2

How does it work?

Why not Struct?

Final Thoughts

References

Ruby 3.2.0 enhances Regexp performance and security with ReDoS protections

What is ReDoS?

An example occurrence of a ReDoS

How to prevent ReDoS in Ruby < 3.2.0 ?

Regexp improvements introduced in Ruby 3.2.0 to prevent ReDoS.

Improved Regexp matching algorithm

Preventing ReDoS attacks with regexp timeouts

Regexp.timeout=

timeout keyword of Regexp.new

References

Rails 7.1 supports infinite ranges for Active Record Validators

Infinite Ranges in Ruby

Infinite Ranges in Rails

With Rails < 7.1.0-alpha

With Rails >= 7.1.0-alpha

References

Introduction to Ractors in Ruby 3

What is an actor model?

Creating a Ractor

Messaging in Ractors

1. Push-type messaging

2. Pull-type messaging

Object sharing

Example - Movie seat booking using ractors

Conclusion

References

Selenium 4 CDP Integration With Capybara

Headers

Cookies

Cache

References

Embedding of Scenario Outline with Data Table in Cucumber

References

Advanced features provided by the new debug gem in Ruby

Comparing the performance of the debug gem with the other debuggers

References

Rails 7 supports tracking of belongs_to association

1. association_changed?

2. association_previously_changed?

References:

Side effects of Active Record's new feature #invert_where in Rails 7

What are the various side effects of using invert_where?

Substitute for invert_where

Conclusion

References

Speeding up Rails 7's Controller Actions using ActiveRecord's #load_async

Configuration for load_async

References

Everything You Need to know about Serialization in Ruby on Rails - Part III

Serialization in Rails for APIs

How to send JSON response from Rails APIs?

Why do we need JSON serializers?

Jbuilder

ActiveModel::Serializers:

Fast JSON API

JSONAPI-RB

Which one to choose?

Conclusion

References

Rails Parameters Parsing and the Case of Parameters Corruption

So what's a query parameter?

And what's a form variable?

Parameters Parsing

What's happening under the covers?

Experimental Setup

`Regexp.timeout=`

`timeout` keyword of `Regexp.new`

Comparing the performance of the `debug` gem with the other debuggers

1. `association_changed?`

2. `association_previously_changed?`

What are the various side effects of using `invert_where`?

Substitute for `invert_where`

Configuration for `load_async`

Let's first understand what is `rsync`:

Why we preferred `rsync` over `scp`?

How to use `rsync` command?

How `rsync` works with `SSH`?

How to use `rsync` in Ruby?

How to track `system` errors in Ruby?