As mentioned in our previous article, we have a wdio-based automation project. We utilized Chrome v108 for our end-to-end functional automation testing. However, given the current browser version being > v115 and since Chrome For Testing was available for Chrome version >= v115, we decided it would be beneficial to upgrade to Chrome For Testing (CFT) in our automation project.

Since the introduction of Google's Chrome For Testing, a new Chrome flavor that specifically targets web app testing and automation use cases, in browser automation, it has emerged as the top choice for conducting automated browser tasks.

What is CFT?

Chrome For Testing is a flavor of chrome browser which is specifically created for running automated test cases. It is a browser which is similar to chrome browser but with few key differences. First off, there is no automatic updating for the CFT browser. This makes it possible to execute automated scripts against a specific version of Chrome. The command-line utility can be used to get the binaries for that particular version of CFT.

Note:- Chromedriver prior to version 115 are not integrated with chrome for testing infrastructure and need to be downloaded from Chromedriver Website

Setting up CFT

We explored using "Chrome For Testing" in our WebdriverIO automation project. As mentioned in the article Take a seat, WebdriverIO is driving for you!, Now that CFT is included into WebdriverIO projects, you can run your automation script on a specific version of CFT by specifying browser name and version in the browser capabilities as demonstrated in the example below:

wdio.conf.js

export const config = {
  // ...
  capabilities: [{
    browserName: 'chrome',
    browserVersion: 125.0.6422.141 // Alternatively you can also specify only the initial part of version number e.g. 125. 
  }]
}

With the above config, WDIO will download the designated CFT and chromedriver version and execute tests.

In doing so, we eliminated all browser drivers and explicitly defined the Chrome version in the WDIO capabilities. This allowed WDIO to execute tests on the CFT instead of actual browsers. However, during local testing, we observed unexpected prompts such as "Save password?" appearing on the CFT, which were absent in real Chrome browsers. Additionally, a warning was displayed in the terminal. Please see the screenshot below for reference.

Continue to use real Chrome Browsers

Due to the above behavior, we were uncertain whether employing CFT would be advantageous for our web application's end-to-end regression testing. So we dropped the idea of CFT and opted to utilize real Chrome browsers. However, we still had to address our issue with sudden Chrome browser upgrades, which were causing session creation failures with the message This chromedriver only supports version <version_number> 😞. Through the insights gained from this blog post, we comprehended that starting from WebdriverIO version v8.14.0 and beyond, we could eliminate the need for driver services.

We removed the following two dependencies from our package.json

• chromedriver
• wdio-chromedriver-service

To run the WebdriverIO project with Chrome installed on your computer, you only need to specify the browser name in the browser capabilities.

export const config = {
  // ...
  capabilities: [{
    browserName: 'chrome',
  }]
}

After removing the aforementioned dependencies and updating the browser capabilities, WDIO started searching for the currently installed Chrome browser on our machine and downloaded the appropriate chromedriver.

The issue remains unresolved 😞

After pushing this commit and executing our end-to-end regression tests according to the schedule, we encountered numerous spec failures with the invalid session id error. Our regression runs on CircleCI, as detailed in our previous blog post. Upon investigation, we identified that the latest Chrome browser version v121 (121.0.6167.164), was being utilized. Further investigation revealed that while Chrome v121 had been released, it was not sufficiently stable for automation purposes. Notably, it exhibited high memory usage due to memory leak, leading to frequent browser crashes.

We also encountered this issue on Chromium and found a similar problem here.

Use Specific Stable Chrome Version

In our CircleCI's configuration file, we specified Chrome version 118.0.5993.117

commands:
  install:
    steps:
      - checkout
      - run: sudo apt-get update
      - node/install-packages
      - browser-tools/install-chrome:
          replace-existing: true
          chrome-version: "118.0.5993.117"

With this adjustment, our automation specifications are now executing flawlessly on Chrome version 118.

Disclaimer:- When we looked into CFT, the latest browser version was 121. Since then, newer versions of Chrome have been released, so you may not encounter the same issues. The primary purpose of this blog is to guide you on how to use a specific Chrome version and run tests on real browsers if you prefer not to use CFT.

References:

• Chrome for Testing: reliable downloads for browser automation
• Take a seat, WebdriverIO is driving for you!

We have a wdio-based automation project that does end-to-end functional testing. We use CircleCI as the CI/CD tool to run automation regression. When setting up a project with CircleCI, we need to enter all configuration details in the config.yml file under the .circleci folder. Earlier, we only had staging regression and prod regression jobs and their respective workflows in the config.yml file.

As mentioned in this article, we also started doing screenshot testing for our application. With time, our configuration file expanded and became cluttered, accommodating various workflows like staging sanity, staging regression, prod regression, performance score testing, and screenshot testing jobs.

We explored various approaches to reduce the size of our configuration file. Later, we came across the Dynamic Configuration implementation of CircleCI. In the beginning, it appeared perplexing, but we eventually understood the purpose of using continuation and path-filtering orbs. In this article, we will share how we utilised the continuation orb to fulfil our requirements with CircleCI.

What were the requirements?

Below were our requirements:

1. Weekly trigger staging-regression job on staging branch.
2. Weekly trigger prod-regression job on master branch.
3. Weekly trigger staging-regression-other-features job on staging branch.
4. Weekly trigger prod-regression-other-features job on master branch.
5. Weekly trigger sign-up, registration, and setup jobs on staging branch.
6. As per the need, we should be able to trigger screenshot testing and sanity jobs.

Dynamic Configuration implementation

We created multiple YAML files for our different jobs and workflows. We used the generate_config.js script to choose the workflow file that we wanted to execute. We implemented the below-outlined structure for organising yml files for our different jobs.

Script to generate a new configuration YAML file based on the parameter

We created a script to generate a new YAML configuration based on the job we want to execute. The new configuration file will be a child configuration file.

/** Below script is implemented in generate_config_script.js file*/

const fs = require('fs');
const path = require('path');

function readFile(fileName) {
  const filePath = path.join(__dirname, `${fileName}.yml`);

  return fs.readFileSync(filePath, 'utf8');
}

function generateCircleCIConfig() {
  const workflow = process.env.CONFIG_TXT;
  const workflowConfig = readFile(workflow);

  console.log(workflowConfig);
}

generateCircleCIConfig();

The above script is executed from the parent configuration file config.yml present in the .circleci folder.

version: 2.1

setup: true

orbs:
  node: circleci/node@5.0.2
  continuation: circleci/continuation@0.1.2

parameters:
  config_txt:
    type: string
    default: staging_sanity

jobs:
  setup:
    docker:
      - image: cimg/node:18.16.1-browsers
    executor: continuation/default
    steps:
      - checkout
      - run:
          name: Generate config
          command: |
            node ./circleci-jobs/generate_config_script.js > generated_config.yml
          environment:
            CONFIG_TXT: << pipeline.parameters.config_txt >>
      - continuation/continue:
          configuration_path: generated_config.yml

workflows:
  setup:
    jobs:
      - setup

In the above parent config.yml file,

1. setup: true: This line indicates that the current config.yml file is a parent configuration file.
2. As mentioned in CI Documentation, the continuation orb is used to execute the child configuration file created based on the output of generate_config_script.js.
3. config_txt is a parameter that is passed to generate_config_script.js, based on which it creates a child configuration file. The default value is staging_sanity, so if no explicit parameter value is passed on CircleCI, then it by default triggers the staging-sanity job.

Also, make sure to define the parameters declared in the parent config.yml file in the child generated_config.yml file as well. Otherwise, it will throw an error Unexpected argument. The official documentation says Pipeline parameters declared in the setup configuration must also be declared in the continuation configuration. These parameters can be used at continuation time.

In the parent config.yml file, only one workflow can be defined, which is responsible for generating the child configuration file. For our project, we required defining multiple workflows for our jobs, and to achieve the same with dynamic configuration, we used triggers.

Schedule Triggers on CircleCI

1. The next step was to create a scheduled pipeline to periodically trigger a specific job.

2. Pipelines are scheduled to periodically run a specific job by adding a trigger for each of the jobs that need to be executed.

3. Triggers are added from the Triggers Section present under the Project Settings option for the CircleCI Project.

4. For each of the jobs mentioned above, we created a new Trigger to periodically run a specific job.

Challenges faced in configuring triggers

We filled out the mandatory information needed to create triggers, but we faced challenges in deciding on the value for the field Repeats Per Hour for the trigger. We were unsure whether this field would trigger the job a certain number of times every hour based on the field's name. We intended our trigger to run just once on the designated day. When we got in touch with CircleCI support, they clarified that the field decides how many times a trigger would be executed in each hour chosen in the Start Time (UTC) selection.

For e.g., if the selected "Start Time (UTC)" is 3:00 (UTC) and 7:00 UTC and "Repeats Per Hour" is set to 2, then the trigger will be executed twice between each selected time, i.e., it will be executed twice between 3:00 (UTC) and 4:00 (UTC) and twice between 7:00 UTC and 8:00 (UTC).

As mentioned in the CI Documentation, setting the start time for a trigger does not guarantee the job will be triggered precisely at the "Start Time" selected. This means that for a trigger, when we select "Start Time" as 6:00 AM UTC and "Repeat Per Hour" as 1, the job will be triggered once between 6:00 AM UTC and 7:00 AM UTC and will not always be triggered exactly at 6:00 AM UTC.

Next, we declared the pipeline parameter named config_txt for our trigger and set its value to the <child_job>.yml file name containing the workflow that we wanted to trigger. For instance, in order to trigger the production regression workflow, we set the config_txt parameter value to prod_regression. Similarly, we created triggers for each workflow that we wanted to execute.

We thought the trigger was scheduled for now. However, the trigger did not execute as scheduled. There was an error block-unregistered-user in our workflow. On investigation, we noticed that for all the triggers, we have "Attribution Actor" set as Scheduled Actor (Scheduling System), which is the default. As mentioned in the article, if the option "Prevent unregistered user spend" is turned ON under the "Usage Control" tab present in the "Plan" section of the project, then the workflow won't be triggered for unregistered users and error block-unregistered-user will be displayed.

This was exactly the case with our project, and to resolve the issue, we selected "Attribution Actor" as one of the users registered in the project. With this, we were able to trigger and execute our workflow successfully.

We have created a demo project circleci-dynamic-configuration-demo that you can clone and explore for better understanding.

References:

• Dynamic Configuration in CircleCI
• continuation orb
• How Dynamic config works
• Scheduled pipelines

Sometimes you may want to standardize the data before saving it to the database. For example, downcasing emails, removing any leading and trailing whitespaces, and so on. Rails 7.1 adds ActiveRecord::Base::normalizes API, which allows you to normalize attribute values to a common format before saving them to the database. This can help improve the integrity and consistency of your data and make it simpler to query records based on attribute values.

Before Rails 7.1

• Normalizing it in before_validation or before_save callback

  class User < ActiveRecord::Base
    before_validation do
      self.name = name.downcase.titleize # normalize_name
      self.email = email.strip.downcase # normalize_email
    end
  
    OR
  
    before_save do
      self.name = name.downcase.titleize # normalize_name
      self.email = email.strip.downcase # normalize_email
    end
  end
  

• Normalizing by overriding attribute_writer

  class User < ActiveRecord::Base
    def name=(val)
      self.name = val.downcase.titleize
    end
  
    def email=(val)
      self.name = email.strip.downcase
    end
  end
  

Rails 7.1 onwards

class User < ActiveRecord::Base
  normalizes :name, with: -> name { name.downcase.titleize }
  normalizes :email, with: -> email { email.downcase.strip }
end

3.0.0 :001 > user = User.create(name: 'BOB', email: " BOB@EXAMPLE.COM\n")
  TRANSACTION (0.1ms)  begin transaction
  User Create (2.0ms)  INSERT INTO "users" ("name", "email") VALUES (?, ?)  [["name", "Bob"], ["email", "bob@example.com"]]
  TRANSACTION (1.9ms)  commit transaction
3.0.0 :002 > user.name
 => "Bob"
3.0.0 :003 > user.email
 => "bob@example.com"

By default, the normalization will not be applied to nil values. This behavior can be changed with the apply_to_nil option which defaults to false.

class User < ActiveRecord::Base
  normalizes :name, with: -> name { name&.downcase&.titleize || 'Untitled' }, apply_to_nil: true
end

3.0.0 :004 > user = User.create(name: nil)
  TRANSACTION (0.1ms)  begin transaction
  User Create (1.4ms)  INSERT INTO "users" ("name", "email") VALUES (?, ?)  [["name", "Untitled"], ["email", nil]]
  TRANSACTION (1.2ms)  commit transaction
3.0.0 :005 > user.name
 => "Untitled"

You can also pass multiple attributes with the normalizes method to apply normalization to all of them.

class User < ActiveRecord::Base
  normalizes :first_name, :last_name, :title, with: -> attribute { attribute.strip }
end

The normalization is also applied to the corresponding keyword argument of finder methods. This allows a record to be created and later queried using denormalized values.

3.0.0 :006 > user = User.find_by(email: "\tBOB@EXAMPLE.COM ")
  User Load (0.2ms)  SELECT "users".* FROM "users" WHERE "users"."email" = ? LIMIT ?  [["email", "bob@example.com"], ["LIMIT", 1]]
3.0.0 :007 > user.email
 => "bob@example.com"
3.0.0 :008 > User.exists?(email: "\tBOB@EXAMPLE.COM ")
  User Exists? (0.3ms)  SELECT 1 AS one FROM "users" WHERE "users"."email" = ? LIMIT ?  [["email", "bob@example.com"], ["LIMIT", 1]]
 => true

It will not be applied when we pass it as a raw query.

3.0.0 :009 > User.exists?(["email = ?", "\tBOB@EXAMPLE.COM "])
  User Exists? (0.2ms)  SELECT 1 AS one FROM "users" WHERE (email = ' BOB@EXAMPLE.COM ') LIMIT ?  [["LIMIT", 1]]
 => false

These changes also introduced the normalize_value_for class method which returns the normalized value for the given attribute.

3.0.0 :010 > User.normalize_value_for(:email, "\tBOB@EXAMPLE.COM ")
 => "bob@example.com"

How to apply normalization to existing records?

The normalization is applied when the attribute is assigned or updated. This means that if a record persisted before the normalization was declared, the record's attribute will not be normalized until it is assigned a new value. Thus, the alternative is to explicitly migrate via Normalization#normalize_attribute.

  class User < ActiveRecord::Base
    normalizes :email, with: -> email { email&.downcase&.strip }
  end

  User.find_each do |legacy_user|
    # legacy_user.email # => " BOB@EXAMPLE.COM\n"
    legacy_user.normalize_attribute(:email)
    # legacy_user.email # => "bob@example.com"
    legacy_user.save
  end

Note: The normalization may be applied multiple times, so it should be idempotent. In other words, applying the normalization more than once should have the same result as applying it only once.

References

• Add ActiveRecord::Base::normalizes

In the previous article, we talked about WebdriverIO. A tool that has become popular very fast in the field of test automation due to its powerful advantages. In this article, we will talk about one of its advantages that has advanced our test automation suite. We have an application that helps our clients develop stunning websites. Whenever a new feature is developed that may impact the UI of the site, WebdriverIO can help us test the change in the UI of the sites. WebdriverIO provides a service called as wdio-image-comparison-service, which performs pixel-by-pixel image comparison.

We already have a wdio-based automation project that does end-to-end functional testing. We expanded it to screenshot test all the pages for thousands of sites. We explored various tools for image comparison. We then discovered that WebdriverIO furthermore offers an image comparison tool. Then we began looking into how we could incorporate this "wdio-image-comparison-service" into our already-running automation project.

Set-up

Referring to the official doc for wdio-image-comparison-service, we installed the module:

npm install --save wdio-image-comparison-service

We added the below options to the Test runner services in wdio.conf.js file:

services: [
  'chromedriver',
  [
    'image-comparison',
    // The options
    {
      // Some options, see the docs for more
      baselineFolder: join(process.cwd(), './tests/'),
      formatImageName: '{tag}-{logName}-{width}x{height}',
      screenshotPath: join(process.cwd(), '.tmp/'),
      savePerInstance: true,
      autoSaveBaseline: true,
      blockOutStatusBar: true,
      blockOutToolBar: true,
      disableCSSAnimation: false,
      ignoreColors: true,
      ignoreTransparentPixel: true,
    },
  ],
],

Above is the default setup. With this default setup, we thought that we had completed configuring the setup and the next step was to create a spec file to compare screenshots.

1. browser.saveScreen("homepage") function saves the screenshot as a baseline image with the name homepage.png.
2. browser.checkScreen("homepage") function saves the screenshot of the current page and compares it with the baseline image having the same name i.e. homepage.png.

However, we encountered an issue with the default setup. When we attempted to create a spec to compare screenshots of the two different doodles, it did not detect any mismatches, and the spec passed. We will explain this issue in more detail below:

We created a spec file called image_compare.js as follows:

describe("Image Comparison", () => {
  it("saves the baseline image", async () => {
    await browser.url("https://www.google.com/doodles/teachers-day-2021-september-11");
    await browser.saveScreen("homepage");
  });

  it("compares with baseline image", async () => {
    await browser.url("https://www.google.com/doodles/teachers-day-2022-september-11");

    const result = await browser.checkScreen("homepage");

    expect(result).toEqual(0);
  });
});

In the above spec, we were comparing screenshots of the Teacher's Day - 2021 homepage and Teacher's Day - 2022 homepage. And we were expecting some mismatch percentage. But the spec got passed. When we investigated, we found that wdio-image-comparison-service saved the baseline image under .tmp folder. The .tmp folder has actual as a subfolder. Under the actual folder, it saved the baseline image.

In our case, when we ran the spec, it overrode Teacher's Day - 2021 homepage.png with Teacher's Day - 2022 homepage.png inside .tmp/ folder. Inside ./tests/ folder also, it saved Teacher's Day - 2022 homepage.png. When we investigated more on this, then we got to know that internally this package uses fs-extra plugin and it has limited Windows support. But, running the same script on Mac or Linux did not result in any issues or failures. We also found out that this is an existing issue. We have also added a comment there about what could be a workaround.

Workaround

As we previously said, the baseline picture is saved by 'wdio-image-comparison-service' by default under the '.tmp' folder. In 'wdio.conf.js', we removed the following code from the'services' section:

baselineFolder: join(process.cwd(), './tests/'),
screenshotPath: join(process.cwd(), '.tmp/'),

Then, we specified the testOptions in the it block where screenshots are compared.

const testOptions = {
  actualFolder: join(process.cwd(), './.tmp/checkActual'),
  baselineFolder: join(process.cwd(), './.tmp/actual'),
  diffFolder: join(process.cwd(), './.tmp/testDiff'),
  returnAllCompareData: true,
};

• checkActual: The current image will be saved in this folder. It is called as actualFolder.
• actual: The baseline image will be saved in this folder. It is called as baselineFolder.
• diffFolder: This image gets created with all the mismatches highlighted after the two screenshots are compared.
• returnAllCompareData: By default, check methods of this service returns only mismatch percentage. When this option is passed as true, then it returns the current image filename, folders that contain the path of actual, baseline, and diff, and misMatchPercentage.

After these modifications, the updated code is as follows:

wdio.conf.js

services: [
  'chromedriver',
  [
    'image-comparison',
    // The options
    {
      // Some options, see the docs for more
      savePerInstance: true,
      autoSaveBaseline: true,
      blockOutStatusBar: true,
      blockOutToolBar: true,
      disableCSSAnimation: false,
      ignoreColors: true,
      ignoreTransparentPixel: true,
    },
  ],
],

image_compare.js

import { join } from 'path';

describe("Image Comparison", () => {
  it("saves the baseline image", async () => {
    await browser.url("https://www.google.com/doodles/teachers-day-2021-september-11");
    await browser.saveScreen("homepage");
  });

  it("compares with baseline image", async () => {
    await browser.url("https://www.google.com/doodles/teachers-day-2022-september-11");
    
    const testOptions = {
      actualFolder: join(process.cwd(), './.tmp/checkActual'),
      baselineFolder: join(process.cwd(), './.tmp/actual'),
      diffFolder: join(process.cwd(), './.tmp/testDiff'),
      returnAllCompareData: true,
    };

    const result = await browser.checkScreen("homepage", testOptions);

    expect(result).toEqual(0);
  });
});

When we passed testOptions as a parameter to the checkScreen function, then wdio easily identified where our baseline images are saved and with which image to compare.

Now, if you run the above spec, then it will properly do the comparison, highlight the mismatch, and return the mismatch percentage.

Teacher's Day - 2021	Teacher's Day - 2022
Image Difference

The wdio-image-comparison-service is not just limited to capturing and comparing screens, it can also compare the full page screen, and elements, and also checks if the website supports the keyboard's TAB key. Please refer to wdio-image-comparison-service options.

Mobile Screenshot Testing

If screenshot testing for mobile devices is needed, then just resizing your browser is never a good idea because this module compares visuals of what the end user is going to see. Wdio Devtools can be helpful in such cases. Wdio Devtools is in itself a distinct, extensive subject that offers additional benefits.

We just need to pass service as devtools in wdio.conf.js file. We can emulate a specific device by await browser.emulateDevice('iPhone X'). Please refer to Device emulation.

We have created a small demo project wdio-image-comparison-demo that you can clone for better understanding. You can explore other options available for wdio-image-comparison-service.

References:

• Image Comparison (Visual Regression Testing) Service
• wdio-image-comparison-service options
• wdio-image-comparison-service issue
• Wdio Devtools Device Emulation

Rails 7 has introduced the in_order_of method which allows to sort records in a specific order. There is a detailed explanation of it in the article Rails 7 adds in_order_of for ActiveRecord::QueryMethods and Enumerable. This is a follow-up article to that one, in which we will explore how the in_order_of method has been improved in Rails 7.1.

Rails 7 adds in_order_of for ActiveRecord::QueryMethods and Enumerable

Rails 7 has added in_order_of method for ActiveRecord::QueryMethods and Enumerable to retrieve the data in an explicit order.

Kiprosh BlogsSupriya Laxman Medankar

Allows to order by string column name

In Rails 7, if a column name is passed to the in_order_of method as a string, it raises the following error:

Loading development environment (Rails 7.0.3)
3.0.0 :001 > CustomerOrder.in_order_of('name', ['Shirt', 'Jeans'])
/usr/share/rvm/gems/ruby-3.0.0/gems/activerecord-7.0.3/lib/active_record/relation/query_methods.rb:459:in `in_order_of': undefined method `in' for "name":String (NoMethodError)                                             
Did you mean?  in?           

It has been fixed in PR#45971. From Rails 7.1 onwards, we are allowed to pass the column name as a string value to the in_order_of method.

Loading development environment (Rails 7.1.0.alpha)
3.0.3 :001 > CustomerOrder.in_order_of("name", ['Shirt', 'Jeans'])
  CustomerOrder Load (0.5ms)  SELECT "customer_orders".* FROM "customer_orders" WHERE "customer_orders"."name" IN ('Shirt', 'Jeans') /* loading for pp */ ORDER BY CASE WHEN "customer_orders"."name" = 'Shirt' THEN 1 WHEN "customer_orders"."name" = 'Jeans' THEN 2 END ASC LIMIT ?  [["LIMIT", 11]]                                                            
 =>                                                                   
[#<CustomerOrder:0x000055cb391a3600                                   
  id: 1,
  name: "Shirt",
  order_status: "Completed",
  created_at: Mon, 23 Jan 2023 15:42:08.473523000 UTC +00:00,
  updated_at: Mon, 23 Jan 2023 15:42:08.473523000 UTC +00:00>,
 #<CustomerOrder:0x000055cb391a3538
  id: 2,
  name: "Jeans",
  order_status: nil,
  created_at: Mon, 23 Jan 2023 15:43:03.940857000 UTC +00:00,
  updated_at: Mon, 23 Jan 2023 15:43:03.940857000 UTC +00:00>]    

Works with nil values

In Rails 7, if nil is passed to the in_order_of method, the following SQL is generated:

Loading development environment (Rails 7.0.3)
3.0.0 :001 > CustomerOrder.in_order_of(:order_status, [nil, 'Completed']).to_sql
 => "SELECT \"customer_orders\".* FROM \"customer_orders\" WHERE \"customer_orders\".\"order_status\" IN (NULL, 'Completed') ORDER BY CASE \"customer_orders\".\"order_status\" WHEN NULL THEN 1 WHEN 'Completed' THEN 2 WHEN ELSE 3 END ASC" 

As we can see from the example above, the CASE statement adds a check for WHEN NULL THEN, but NULL != NULL in SQL. As a result, when ordering, records with NULL values are ignored, and those records are excluded from the final result.

In Rails 7.1, The PR#45670 has added a fix to generate SQL WHEN \"customer_orders\".\"order_status\" IS NULL for nil values, which sort records correctly and return records with nil value in the result.

Loading development environment (Rails 7.1.0.alpha)
3.0.3 :001 >  CustomerOrder.in_order_of(:order_status, [nil, 'Completed'])
  CustomerOrder Load (0.3ms)  SELECT "customer_orders".* FROM "customer_orders" WHERE ("customer_orders"."order_status" IN ('Completed') OR "customer_orders"."order_status" IS NULL) /* loading for pp */ ORDER BY CASE WHEN "customer_orders"."order_status" IS NULL THEN 1 WHEN "customer_orders"."order_status" = 'Completed' THEN 2 END ASC LIMIT ?  [["LIMIT", 11]]              
 =>                                                                                          
[#<CustomerOrder:0x000055cb3822b308                                                          
  id: 2,                                                                                     
  name: "Jeans",                                                                              
  order_status: nil,                                                                         
  created_at: Mon, 23 Jan 2023 15:43:03.940857000 UTC +00:00,                                
  updated_at: Mon, 23 Jan 2023 15:43:03.940857000 UTC +00:00>,                               
 #<CustomerOrder:0x000055cb3822b240                                                          
  id: 1,                                                                                     
  name: "Shirt",                                                                             
  order_status: "Completed",                                                                 
  created_at: Mon, 23 Jan 2023 15:42:08.473523000 UTC +00:00,                                
  updated_at: Mon, 23 Jan 2023 15:42:08.473523000 UTC +00:00>]

Using ORDER BY CASE for all

In Rails 7, in_order_of uses the special order field generation, i.e. ORDER BY FIELD for the MySQL adapter. This does not add any performance improvement other than a simplified query, so it has been replaced by ORDER BY CASE.

From Rails 7.1 onwards, all database adapters will now use ORDER BY CASE. The PR#45670 has added this fix, the same PR which has added fix for in_order_of to work with nils.

References:

1. improve "in_order_of" to allow string column name
2. Fix ActiveRecord::QueryMethods#in_order_of to work with nils

In the era of content consumption where streaming platforms like YouTube and Netflix have millions of concurrent users, no one wants to wait more than a minute to consume anything, it must be available instantly!

Streaming becomes a need in this situation. While downloading requires waiting until the entire file has been downloaded on your computer, streaming allows you to view downloaded portions of the material on the fly. This significantly reduces the waiting time.

Let's see with the help of a small example how streaming was handled before Rails 7 and what changed after Rails 7.

With Rails < 7

Suppose we need to write a controller action that streams data chunks to the client in real time as they are getting downloaded.

Before Rails 7, we were required to include the ActionController::Live module for live streaming to work. By mixin the ActionController::Live module in our application's controller, we enable all actions in that controller to stream data to the client as it is written in real-time.

Note: The response headers must be explicitly specified in all actions of the controller. This enables the browser to recognize the type of file it should display.

class VideoController < ApplicationController
  include ActionController::Live

  def show
    response.headers["Content-Type"] = @video.content_type
    response.headers["Content-Disposition"] = "inline; #{@video.filename.parameters}"

    # Streaming downloaded chunks of data to the client as the video downloads
    @video.download do |chunk|
      response.stream.write(chunk)
    end

    response.stream.close
  end
end

Content-Disposition is a response header indicating if the content is expected to be displayed inline in the browser, that is, as a Web page or as part of a Web page, or as an attachment, that is downloaded and saved locally.

Few caveats with Rails < 7:

1. You need to write headers for the actions before a response has been committed (calling write or close on the response stream will cause the response object to be committed). Make sure all headers are set before calling write or close on your stream.
2. You must close the stream when it's finished, otherwise, the socket will remain open forever.

With Rails >= 7

ActiveStorage::Streaming module is introduced in this PR, which supports native file streaming from the controller. Once we include ActiveStorage::Streaming in a controller, we get access to the #send_blob_stream method which takes care of everything, from writing the headers to streaming the downloaded data chunks to the client to closing the stream after it is completed.

Now we don't need to re-write response headers and close the stream before and after every streaming action.

class VideoController < ApplicationController
  include ActiveStorage::Streaming

  def show
    http_cache_forever(public: true) do
      send_blob_stream @video, disposition: params[:disposition]
    end
  end
end

The method http_cache_forever allows us to set response headers to tell the browser that the response has not been modified. It means that on the first hit, we serve the request normally but, then on each subsequent request, the cache is revalidated and a 304 Not Modified response is sent to the browser. This saves us from setting the same response header on every request.

# First request:

Processing by VideoController#show as HTML
  Rendered videos/show.html.erb within layouts/application (1.8ms)
Completed 200 OK in 198ms (Views: 256.7ms | ActiveRecord: 0.0ms)

# Consecutive requests for the same page:

Processing by VideoController#show as HTML
Completed 304 Not Modified in 3ms (ActiveRecord: 0.0ms)

Rails has started to expand its streaming capabilities, and we may see more updates in the coming months. Checkout most recent updates in the pull requests below:

1. Extract ActiveStorage::Streaming so your own controllers can use it
2. Don't stream redirect controller responses
3. Rails ActiveStorage::Streaming module

References:

• Rails PR discussion for adding support to build a controller that streams from active storage - Extract ActiveStorage::Streaming so your own controllers can use it

Rails 7.1 adds the raise_on_assign_to_attr_readonly config to config.active_record that is used to raise ActiveRecord::ReadonlyAttributeError error on assignment to attr_readonly attributes. The previous behavior would allow assignment but silently not persist changes to the database.

Configuring config.active_record.raise_on_assign_to_attr_readonly

In a newly created Rails 7.1 application, config.load_defaults 7.1 is set by default in application.rb. The default value of raise_on_assign_to_attr_readonly for config.load_defaults 7.1 is true and for config.load_defaults < 7.1 it is false. To opt-in:

# config/environments/application.rb

config.active_record.raise_on_assign_to_attr_readonly = true

Let's take an example of the following Post model structure:

# app/models/post.rb

class Post < ActiveRecord::Base
  attr_readonly :title
end

• When config.active_record.raise_on_assign_to_attr_readonly = true

  post = Post.create!(title: "Introducing Ruby on Rails!")
  post.title = "a different title" # raises ActiveRecord::ReadonlyAttributeError
  post.update(title: "a different title") # raises ActiveRecord::ReadonlyAttributeError
  

• When config.active_record.raise_on_assign_to_attr_readonly = false

  post = Post.create!(title: "Introducing Ruby on Rails!")
  post.title = "a different title" # does not raise any error
  post.update(title: "a different title") # change to title will be ignored
  

References

• Rails PR#46105 & Rails PR#46602 - Raise on assignment to readonly attributes

Ruby ranges are very handy and efficient when you want to work with a sequence of numbers or characters bound by start and end values. But we could come across situations where we want ranges that either have no end value or start value. Ruby 2.6 introduced the concept of an endless range and later Ruby 2.7 brought in the beginless range.

Infinite Ranges in Ruby

As mentioned in the official Ruby docs, a beginless range and endless range represents a semi-infinite range.
A beginless range would look like (..3) Or (...3) while endless range looks like (3..) Or (3...) and effectively defines the maximum or minimum boundary of the range.

Infinite Ranges in Rails

Infinite ranges are supported in Rails while fetching records from the database as seen below:

Loading development environment (Rails 7.1.0.alpha)
irb(main):013:0> Event.where(date:  (..Time.now))
  Event Load (0.5ms)  SELECT "events".* FROM "events" WHERE "events"."date" <= ? /* loading for pp */ LIMIT ?  [["date", "2022-12-11 12:56:31.866828"], ["LIMIT", 11]]
=>
[#<Event:0x000055a3f20b22f0
  id: 1,                                                              
  date: Sun, 11 Dec 2022 12:32:57.255815000 UTC +00:00,               
  name: "The December Concert",                                       
  description: "The ultimate concert to end the year",                
  created_at: Sun, 11 Dec 2022 12:32:57.256643000 UTC +00:00,         
  updated_at: Sun, 11 Dec 2022 12:32:57.256643000 UTC +00:00>,
 #<Event:0x000055a3f20b2188
  id: 5,
  date: Mon, 31 Oct 2022 18:30:00.000000000 UTC +00:00,
  name: "The Halloween Festival",
  description: "The eve of All Saints Day",
  created_at: Sun, 11 Dec 2022 12:55:33.831731000 UTC +00:00,
  updated_at: Sun, 11 Dec 2022 12:56:21.981937000 UTC +00:00>]

The above query fetched all the past event records, i.e the ones with the date value less than the current time.

Loading development environment (Rails 7.1.0.alpha)
irb(main):014:0> Event.where(date:  (Time.now..))
  Event Load (0.2ms)  SELECT "events".* FROM "events" WHERE "events"."date" >= ? /* loading for pp */ LIMIT ?  [["date", "2022-12-11 12:56:43.438138"], ["LIMIT", 11]]
=>
[#<Event:0x000055a3f26616b0
  id: 2,
  date: Sat, 31 Dec 2022 18:30:00.000000000 UTC +00:00,
  name: "The New Year Event",
  description: "The first event of the year",
  created_at: Sun, 11 Dec 2022 12:34:23.292070000 UTC +00:00,
  updated_at: Sun, 11 Dec 2022 12:34:23.292070000 UTC +00:00>,
 #<Event:0x000055a3f26615e8
  id: 3,
  date: Thu, 12 Jan 2023 18:30:00.000000000 UTC +00:00,
  name: "The Lohri Festival",
  description: "The celebration of harvest festival",
  created_at: Sun, 11 Dec 2022 12:36:13.234757000 UTC +00:00,
  updated_at: Sun, 11 Dec 2022 12:37:13.423191000 UTC +00:00>]

The above query fetched all the future event records, i.e the ones with the date value greater than the current time.

In addition to using ranges for fetching records, we generally use them in Rails with the Active Record length and inclusion validators.

With Rails < 7.1.0-alpha

If we use infinite range in a validator like:

class Event < ApplicationRecord
  ...
  validates_length_of :name, in: (..30)
  ...
end

it throws an error as follows:

Loading development environment (Rails 7.0.1)
irb(main):001:0> Event
Traceback (most recent call last):
        3: from (irb):1
        2: from app/models/event.rb:4:in `<main>'
        1: from app/models/event.rb:7:in `<class:Event>'
RangeError (cannot get the minimum of beginless range)

The only way to validate is by passing maximum or minimum value to the validates method as follows:

validates_length_of :name, maximum: 30, allow_blank: true

With Rails >= 7.1.0-alpha

With the two PRs that were merged recently - PR#45123 and PR#45138, it is possible to pass beginless/endless ranges to the length and inclusivity validators with the in/within options.

  # Allows date to be either today's or any future date but not a past date
  validates_inclusion_of :date, in: -> { (Date.today..) } 

  # Allows name to be nil or of a max length of 30 characters
  validates_length_of :name, in: (..30)

  # Mandates description to be of minimum 10 characters with no max limit
  validates_length_of :description, in: (10..)

Please note: These changes are available in the alpha phase of Rails 7.1.0 at the time of writing this article. To get the 7.1.0-alpha version, you have to specify the rails main branch in Gemfile -

    # Gemfile
    gem 'rails', github: 'rails/rails', branch: 'main'

References

• Support infinite ranges for LengthValidators :in/:within options
• Add beginless range support to clusivity

Hotwire has been around for a while now. If you got a chance to create an app powered by it yet, you must have realized how easy it is to create apps that feel like single-page applications. In this blog, we will see the lesser-discovered part of the ecosystem - broadcasting.

Turbo streams

Think of turbo streams as a set of changes to perform on your UI. You send these changes back to the UI as a response to an event, form submission, or whenever you feel like it. If you are new to this, I recommend going through the Turbo - The heart of Hotwire first.

Broadcasting?

If you are creating an application where you want changes in the UI to be reflected not just for the instance that requested it but for others (who're viewing the same page) too, then broadcasting is what you need.

Let's take a simple example of a notifications feature.

How would you make sure that if an event occurs, the notification icon updates for everyone? You can create a turbo stream that replaces the icon and send that turbo stream to everyone.

First, everyone has to listen to a WebSocket channel for incoming turbo streams. This is done using

<%= turbo_stream_from(:notifications) %>

where :notifications is just the name of the channel.

Second, you broadcast the update using the special broadcasting methods provided by the Turbo::Broadcastable concern.

broadcast_replace_to(:notifications, target: "icon", html: "<span class='icon'>notification_new</span>")

This method does two things

1. Generates the turbo stream.
2. Broadcasts the turbo stream to all subscribers of the notifications channel.

The broadcasted turbo stream looks something like this:

<turbo-stream action="replace" target="icon">
    <span class='icon'>notification_new</span>
</turbo-stream>

You can call this function in the controller or in the model after a callback like:

# app/models/Post.rb

class Post < ApplicationRecord
  ...
  after_create_commit -> { 
    broadcast_replace_to(
      :notifications,
      target: "icon",
      html: "<span class='icon'>notification_new</span>"
      )
  }
  ...
end

Once a post is created, the icon for all the users will be replaced with a new one indicating a new notification.

The ways of Broadcasting

So far, you have seen one type of broadcasting - replace. There are broadcasting functions for every type of turbo stream actions -  broadcast_update_to, broadcast_append_to ... the entire list is here. Let's see some of the common hurdles that you might face while using them.

Multiple Turbo stream actions at once?

If you want to broadcast multiple turbo stream actions at once, you can put all of your actions in one *.turbo_stream.erb file and broadcast the file!

# app/views/notifications/_new.turbo_stream.erb

<%= turbo_stream.replace "icon" do %>
   <span class='icon'>notification_new</span>
<% end %>
<%= turbo_stream.append "icon" do %>
   <span class="count"><%= count %></span>
<% end %>

# app/models/Post.rb

class Post < ApplicationRecord
  ...
  after_create_commit -> { 
    broadcast_render_to(
      :notifications,
      partial: "notifications/new"
    )
  }
  ...
end

Selective broadcasting?

If you want to multicast a turbo stream instead of broadcasting it, just make sure it reaches the right receivers. You can distinguish between your receivers by giving them their own channels.

For an example, different users are in different groups, and you want only the people inside the group to get the updates, how do you distinguish them from all the other users in the other groups?

<%= turbo_stream_from(@group) %>

This will create a unique channel for each group, and when you broadcast your updates,

# app/models/Post.rb

class Post < ApplicationRecord
  belongs_to :group
  ...
  after_create_commit -> { 
    broadcast_render_to(
      group,
      partial: "notifications/new"
    )
  ...
end

It will be broadcasted to only the required groups/channels.

Testing your broadcasts

Of course, you would want to make sure your broadcasts are going through the correct channels but how do you test that? With ActionCable test helpers.

test "notifications are sent to same group" do
    stream = @group
    target = "icon"

    assert_broadcast_on stream, turbo_stream_action_tag("replace", target: target, template: %(<span class='icon'>notification_new</span>)) do
      Post.create(title: 'A new post')
    end
end

High response time!

If the broadcasting of your turbo stream is an expensive operation, just create a Turbo::Streams::BroadcastJob to broadcast later in a background job. This is done using broadcast_*_later_to methods.

# app/models/Post.rb

class Post < ApplicationRecord
  belongs_to :group
  ...
  after_create_commit -> { 
    broadcast_render_later_to(
      group,
      partial: "notifications/new"
    )
  ...
end

You can try these examples in the demonstration app here. That's all about broadcasting for now. Until next time.

Rails provides the has_secure_password method, which makes it gloriously easy to implement authentication in our application.
But we often need an extra layer of verification before allowing users to update certain fields. For e.g. Users must provide their “old” password when updating their email/password fields.

Before Rails 7.1

To implement this, we must manually add and validate the current_password accessor:

# app/models/user.rb

class User < ActiveRecord::Base
  has_secure_password

  attr_accessor :current_password
end

# app/controllers/passwords_controller.rb

class PasswordsController < ApplicationController
  def update
    password_challenge = password_params.delete(:current_password)
    @password_challenge_succeeded = current_user.authenticate(password_challenge)

    if @password_challenge_succeeded && current_user.update(password_params)
      # ...
    end
  end

  private

  def password_params
    params.require(:user).permit(
      :password, 
      :password_confirmation, 
      :current_password
    )
  end
end

# app/views/users/password_edit.html.erb

<%= form_for current_user do |f| %>
  <div class='mb-4'>
    <%= f.label :current_password, 'Current Password' %>
    <%= f.password_field :current_password %>
    <!-- Render error if @password_challenge_succeeded is false -->
    <% unless @password_challenge_succeeded %>
      <p class='error'>Current password is invalid.</p>
    <% end %>
  </div>

  <div class='mb-4'>
    <%= f.label :password, 'New Password' %>
    <%= f.password_field :password %>
  </div>

  <div class='mb-6'>
    <%= f.label :password_confirmation %>
    <%= f.password_field :password_confirmation %>
  </div>

  <div>
    <%= f.submit 'Update' %>
  </div>
<% end %>

Rails 7.1 onwards

has_secure_password now includes a password_challenge accessor and its corresponding validation. So we no longer need our own accessor and can use password_challenge to verify the existing password.
When password_challenge is set, it will validate against the currently persisted password digest (i.e. password_digest_was).

# app/models/user.rb

class User < ActiveRecord::Base
  has_secure_password
end

# app/controllers/passwords_controller.rb

class PasswordsController < ApplicationController
  def update
    if current_user.update(password_params)
      # ...
    end
  end

  private

  def password_params
    # Note: password_challenge will validate only if it is set to a value other than nil.
    params.require(:user).permit(
      :password, 
      :password_confirmation, 
      :password_challenge
    ).with_defaults(password_challenge: '')
  end
end

And in the views we can replace current_password field with password_challenge.

# app/views/users/password_edit.html.erb

<%= form_for current_user do |f| %>
  <div class='mb-4'>
    <%= f.label :password_challenge, 'Current Password' %>
    <%= f.password_field :password_challenge %>
    <% if f.object.errors[:password_challenge].present? %>
      <p class='error'> <%= f.object.errors[:password_challenge].first %> </p>
    <% end %>
  </div>
    .
    .
    .
<% end %>

This allows password_challenge to be implemented with the same ease as the password_confirmation. If the password_challenge fails, a validation error will be added just like any other attribute.

Check out this pull request for more details.