Merge tag 'v4.2.3' into acc4e_2

Co-authored-by: Claire <claire.github-309c@sitedethib.com>

CHANGELOG.md

@@ -2,6 +2,39 @@
 
 All notable changes to this project will be documented in this file.
 
+## [4.2.3] - 2023-12-05
+
+### Fixed
+
+- Fix dependency on `json-canonicalization` version that has been made unavailable since last release
+
+## [4.2.2] - 2023-12-04
+
+### Changed
+
+- Change dismissed banners to be stored server-side ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27055))
+- Change GIF max matrix size error to explicitly mention GIF files ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27927))
+- Change `Follow` activities delivery to bypass availability check ([ShadowJonathan](https://github.com/mastodon/mastodon/pull/27586))
+- Change single-column navigation notice to be displayed outside of the logo container ([renchap](https://github.com/mastodon/mastodon/pull/27462), [renchap](https://github.com/mastodon/mastodon/pull/27476))
+- Change Content-Security-Policy to be tighter on media paths ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/26889))
+- Change post language code to include country code when relevant ([gunchleoc](https://github.com/mastodon/mastodon/pull/27099), [ClearlyClaire](https://github.com/mastodon/mastodon/pull/27207))
+
+### Fixed
+
+- Fix upper border radius of onboarding columns ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27890))
+- Fix incoming status creation date not being restricted to standard ISO8601 ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27655), [ClearlyClaire](https://github.com/mastodon/mastodon/pull/28081))
+- Fix some posts from threads received out-of-order sometimes not being inserted into timelines ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27653))
+- Fix posts from force-sensitized accounts being able to trend ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27620))
+- Fix error when trying to delete already-deleted file with OpenStack Swift ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27569))
+- Fix batch attachment deletion when using OpenStack Swift ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27554))
+- Fix processing LDSigned activities from actors with unknown public keys ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27474))
+- Fix error and incorrect URLs in `/api/v1/accounts/:id/featured_tags` for remote accounts ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27459))
+- Fix report processing notice not mentioning the report number when performing a custom action ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27442))
+- Fix handling of `inLanguage` attribute in preview card processing ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27423))
+- Fix own posts being removed from home timeline when unfollowing a used hashtag ([kmycode](https://github.com/mastodon/mastodon/pull/27391))
+- Fix some link anchors being recognized as hashtags ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27271), [ClearlyClaire](https://github.com/mastodon/mastodon/pull/27584))
+- Fix format-dependent redirects being cached regardless of requested format ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27634))
+
 ## [4.2.1] - 2023-10-10
 
 ### Added

Gemfile.lock

@@ -148,6 +148,7 @@ GEM
       net-http-persistent (~> 4.0)
       nokogiri (~> 1, >= 1.10.8)
     base64 (0.1.1)
+    bcp47_spec (0.2.1)
     bcrypt (3.1.18)
     better_errors (2.10.1)
       erubi (>= 1.0.0)
@@ -377,19 +378,19 @@ GEM
     ipaddress (0.8.3)
     jmespath (1.6.2)
     json (2.6.3)
-    json-canonicalization (0.3.2)
+    json-canonicalization (1.0.0)
     json-jwt (1.15.3)
       activesupport (>= 4.2)
       aes_key_wrap
       bindata
       httpclient
-    json-ld (3.2.5)
+    json-ld (3.3.1)
       htmlentities (~> 4.3)
-      json-canonicalization (~> 0.3, >= 0.3.2)
+      json-canonicalization (~> 1.0)
       link_header (~> 0.0, >= 0.0.8)
       multi_json (~> 1.15)
       rack (>= 2.2, < 4)
-      rdf (~> 3.2, >= 3.2.10)
+      rdf (~> 3.3)
     json-ld-preloaded (3.2.2)
       json-ld (~> 3.2)
       rdf (~> 3.2)
@@ -593,7 +594,8 @@ GEM
       zeitwerk (~> 2.5)
     rainbow (3.1.1)
     rake (13.0.6)
-    rdf (3.2.11)
+    rdf (3.3.1)
+      bcp47_spec (~> 0.2)
       link_header (~> 0.0, >= 0.0.8)
     rdf-normalize (0.6.1)
       rdf (~> 3.2)

SECURITY.md

@@ -17,6 +17,6 @@ A "vulnerability in Mastodon" is a vulnerability in the code distributed through
 | ------- | ---------------- |
 | 4.2.x   | Yes              |
 | 4.1.x   | Yes              |
-| 4.0.x   | Until 2023-10-31 |
+| 4.0.x   | No               |
 | 3.5.x   | Until 2023-12-31 |
 | < 3.5   | No               |

app/chewy/accounts_index.rb

@@ -1,6 +1,8 @@
 # frozen_string_literal: true
 
 class AccountsIndex < Chewy::Index
+  include DatetimeClampingConcern
+
   settings index: index_preset(refresh_interval: '30s'), analysis: {
     filter: {
       english_stop: {
@@ -60,7 +62,7 @@ class AccountsIndex < Chewy::Index
     field(:following_count, type: 'long')
     field(:followers_count, type: 'long')
     field(:properties, type: 'keyword', value: ->(account) { account.searchable_properties })
-    field(:last_status_at, type: 'date', value: ->(account) { account.last_status_at || account.created_at })
+    field(:last_status_at, type: 'date', value: ->(account) { clamp_date(account.last_status_at || account.created_at) })
     field(:display_name, type: 'text', analyzer: 'verbatim') { field :edge_ngram, type: 'text', analyzer: 'edge_ngram', search_analyzer: 'verbatim' }
     field(:username, type: 'text', analyzer: 'verbatim', value: ->(account) { [account.username, account.domain].compact.join('@') }) { field :edge_ngram, type: 'text', analyzer: 'edge_ngram', search_analyzer: 'verbatim' }
     field(:text, type: 'text', analyzer: 'verbatim', value: ->(account) { account.searchable_text }) { field :stemmed, type: 'text', analyzer: 'natural' }

app/chewy/concerns/datetime_clamping_concern.rb

@@ -0,0 +1,14 @@
+# frozen_string_literal: true
+
+module DatetimeClampingConcern
+  extend ActiveSupport::Concern
+
+  MIN_ISO8601_DATETIME = '0000-01-01T00:00:00Z'.to_datetime.freeze
+  MAX_ISO8601_DATETIME = '9999-12-31T23:59:59Z'.to_datetime.freeze
+
+  class_methods do
+    def clamp_date(datetime)
+      datetime.clamp(MIN_ISO8601_DATETIME, MAX_ISO8601_DATETIME)
+    end
+  end
+end

app/chewy/public_statuses_index.rb

@@ -1,6 +1,8 @@
 # frozen_string_literal: true
 
 class PublicStatusesIndex < Chewy::Index
+  include DatetimeClampingConcern
+
   settings index: index_preset(refresh_interval: '30s', number_of_shards: 5), analysis: {
     filter: {
       english_stop: {
@@ -62,6 +64,6 @@ class PublicStatusesIndex < Chewy::Index
     field(:tags, type: 'text', analyzer: 'hashtag', value: ->(status) { status.tags.map(&:display_name) })
     field(:language, type: 'keyword')
     field(:properties, type: 'keyword', value: ->(status) { status.searchable_properties })
-    field(:created_at, type: 'date')
+    field(:created_at, type: 'date', value: ->(status) { clamp_date(status.created_at) })
   end
 end

app/chewy/statuses_index.rb

@@ -1,6 +1,8 @@
 # frozen_string_literal: true
 
 class StatusesIndex < Chewy::Index
+  include DatetimeClampingConcern
+
   settings index: index_preset(refresh_interval: '30s', number_of_shards: 5), analysis: {
     filter: {
       english_stop: {
@@ -60,6 +62,6 @@ class StatusesIndex < Chewy::Index
     field(:searchable_by, type: 'long', value: ->(status) { status.searchable_by })
     field(:language, type: 'keyword')
     field(:properties, type: 'keyword', value: ->(status) { status.searchable_properties })
-    field(:created_at, type: 'date')
+    field(:created_at, type: 'date', value: ->(status) { clamp_date(status.created_at) })
   end
 end

app/chewy/tags_index.rb

@@ -1,6 +1,8 @@
 # frozen_string_literal: true
 
 class TagsIndex < Chewy::Index
+  include DatetimeClampingConcern
+
   settings index: index_preset(refresh_interval: '30s'), analysis: {
     analyzer: {
       content: {
@@ -42,6 +44,6 @@ class TagsIndex < Chewy::Index
     field(:name, type: 'text', analyzer: 'content', value: :display_name) { field(:edge_ngram, type: 'text', analyzer: 'edge_ngram', search_analyzer: 'content') }
     field(:reviewed, type: 'boolean', value: ->(tag) { tag.reviewed? })
     field(:usage, type: 'long', value: ->(tag, crutches) { tag.history.aggregate(crutches.time_period).accounts })
-    field(:last_status_at, type: 'date', value: ->(tag) { tag.last_status_at || tag.created_at })
+    field(:last_status_at, type: 'date', value: ->(tag) { clamp_date(tag.last_status_at || tag.created_at) })
   end
 end

app/controllers/admin/account_actions_controller.rb

@@ -21,7 +21,7 @@ module Admin
       account_action.save!
 
       if account_action.with_report?
-        redirect_to admin_reports_path, notice: I18n.t('admin.reports.processed_msg', id: params[:report_id])
+        redirect_to admin_reports_path, notice: I18n.t('admin.reports.processed_msg', id: resource_params[:report_id])
       else
         redirect_to admin_account_path(@account.id)
       end

app/helpers/languages_helper.rb

@@ -254,6 +254,7 @@ module LanguagesHelper
 
   def valid_locale_or_nil(str)
     return if str.blank?
+    return str if valid_locale?(str)
 
     code, = str.to_s.split(/[_-]/) # Strip out the region from e.g. en_US or ja-JP
 

app/javascript/mastodon/components/dismissable_banner.tsx

@@ -1,9 +1,16 @@
+/* eslint-disable @typescript-eslint/no-unsafe-call,
+                  @typescript-eslint/no-unsafe-return,
+                  @typescript-eslint/no-unsafe-assignment,
+                  @typescript-eslint/no-unsafe-member-access
+                  -- the settings store is not yet typed */
 import type { PropsWithChildren } from 'react';
-import { useCallback, useState } from 'react';
+import { useCallback, useState, useEffect } from 'react';
 
 import { defineMessages, useIntl } from 'react-intl';
 
+import { changeSetting } from 'mastodon/actions/settings';
 import { bannerSettings } from 'mastodon/settings';
+import { useAppSelector, useAppDispatch } from 'mastodon/store';
 
 import { IconButton } from './icon_button';
 
@@ -19,13 +26,25 @@ export const DismissableBanner: React.FC<PropsWithChildren<Props>> = ({
   id,
   children,
 }) => {
-  const [visible, setVisible] = useState(!bannerSettings.get(id));
+  const dismissed = useAppSelector((state) =>
+    state.settings.getIn(['dismissed_banners', id], false),
+  );
+  const dispatch = useAppDispatch();
+
+  const [visible, setVisible] = useState(!bannerSettings.get(id) && !dismissed);
   const intl = useIntl();
 
   const handleDismiss = useCallback(() => {
     setVisible(false);
     bannerSettings.set(id, true);
-  }, [id]);
+    dispatch(changeSetting(['dismissed_banners', id], true));
+  }, [id, dispatch]);
+
+  useEffect(() => {
+    if (!visible && !dismissed) {
+      dispatch(changeSetting(['dismissed_banners', id], true));
+    }
+  }, [id, dispatch, visible, dismissed]);
 
   if (!visible) {
     return null;

app/javascript/mastodon/features/ui/components/link_footer.jsx

@@ -100,7 +100,7 @@ class LinkFooter extends PureComponent {
           {DividingCircle}
           <a href={source_url} rel='noopener noreferrer' target='_blank'><FormattedMessage id='footer.source_code' defaultMessage='View source code' /></a>
           {DividingCircle}
-          <span class='version'>v{version}</span>
+          <span className='version'>v{version}</span>
         </p>
       </div>
     );

app/javascript/mastodon/features/ui/components/navigation_panel.jsx

@@ -53,24 +53,30 @@ class NavigationPanel extends Component {
     const { intl } = this.props;
     const { signedIn, disabledAccountId } = this.context.identity;
 
+    let banner = undefined;
+
+    if(transientSingleColumn)
+      banner = (<div className='switch-to-advanced'>
+        {intl.formatMessage(messages.openedInClassicInterface)}
+        {" "}
+        <a href={`/deck${location.pathname}`} className='switch-to-advanced__toggle'>
+          {intl.formatMessage(messages.advancedInterface)}
+        </a>
+      </div>);
+
     return (
       <div className='navigation-panel'>
         <div className='navigation-panel__logo'>
           <Link to='/' className='column-link column-link--logo'><WordmarkLogo /></Link>
-
-          {transientSingleColumn ? (
-            <div class='switch-to-advanced'>
-              {intl.formatMessage(messages.openedInClassicInterface)}
-              {" "}
-              <a href={`/deck${location.pathname}`} class='switch-to-advanced__toggle'>
-                {intl.formatMessage(messages.advancedInterface)}
-              </a>
-            </div>
-          ) : (
-            <hr />
-          )}
+          {!banner && <hr />}
         </div>
 
+        {banner &&
+          <div class='navigation-panel__banner'>
+            {banner}
+          </div>
+        }
+
         {signedIn && (
           <>
             <ColumnLink transparent to='/home' icon='home' text={intl.formatMessage(messages.home)} />

app/javascript/mastodon/reducers/settings.js

@@ -100,6 +100,15 @@ const initialState = ImmutableMap({
       body: '',
     }),
   }),
+
+  dismissed_banners: ImmutableMap({
+    'public_timeline': false,
+    'community_timeline': false,
+    'home.explore_prompt': false,
+    'explore/links': false,
+    'explore/statuses': false,
+    'explore/tags': false,
+  }),
 });
 
 const defaultColumns = fromJS([

app/javascript/styles/mastodon/components.scss

@@ -2239,8 +2239,7 @@ $ui-header-height: 55px;
 
   > .scrollable {
     background: $ui-base-color;
-    border-bottom-left-radius: 4px;
-    border-bottom-right-radius: 4px;
+    border-radius: 0 0 4px 4px;
   }
 }
 
@@ -2466,6 +2465,7 @@ $ui-header-height: 55px;
 
     .navigation-panel__sign-in-banner,
     .navigation-panel__logo,
+    .navigation-panel__banner,
     .getting-started__trends {
       display: none;
     }

app/lib/activitypub/case_transform.rb

@@ -14,6 +14,8 @@ module ActivityPub::CaseTransform
       when String
         camel_lower_cache[value] ||= if value.start_with?('_:')
                                        "_:#{value.delete_prefix('_:').underscore.camelize(:lower)}"
+                                     elsif LanguagesHelper::ISO_639_1_REGIONAL.key?(value.to_sym)
+                                       value
                                      else
                                        value.underscore.camelize(:lower)
                                      end

app/lib/activitypub/linked_data_signature.rb

@@ -18,8 +18,8 @@ class ActivityPub::LinkedDataSignature
 
     return unless type == 'RsaSignature2017'
 
-    creator   = ActivityPub::TagManager.instance.uri_to_actor(creator_uri)
-    creator ||= ActivityPub::FetchRemoteKeyService.new.call(creator_uri, id: false)
+    creator = ActivityPub::TagManager.instance.uri_to_actor(creator_uri)
+    creator = ActivityPub::FetchRemoteKeyService.new.call(creator_uri, id: false) if creator&.public_key.blank?
 
     return if creator.nil?
 
@@ -28,6 +28,8 @@ class ActivityPub::LinkedDataSignature
     to_be_verified = options_hash + document_hash
 
     creator if creator.keypair.public_key.verify(OpenSSL::Digest.new('SHA256'), Base64.decode64(signature), to_be_verified)
+  rescue OpenSSL::PKey::RSAError
+    false
   end
 
   def sign!(creator, sign_with: nil)

app/lib/activitypub/parser/status_parser.rb

@@ -53,7 +53,8 @@ class ActivityPub::Parser::StatusParser
   end
 
   def created_at
-    @object['published']&.to_datetime
+    datetime = @object['published']&.to_datetime
+    datetime if datetime.present? && (0..9999).cover?(datetime.year)
   rescue ArgumentError
     nil
   end

app/lib/attachment_batch.rb

@@ -75,7 +75,12 @@ class AttachmentBatch
             end
           when :fog
             logger.debug { "Deleting #{attachment.path(style)}" }
-            attachment.directory.files.new(key: attachment.path(style)).destroy
+
+            begin
+              attachment.send(:directory).files.new(key: attachment.path(style)).destroy
+            rescue Fog::Storage::OpenStack::NotFound
+              # Ignore failure to delete a file that has already been deleted
+            end
           when :azure
             logger.debug { "Deleting #{attachment.path(style)}" }
             attachment.destroy

app/lib/feed_manager.rb

@@ -192,6 +192,7 @@ class FeedManager
     # also tagged with another followed hashtag or from a followed user
     scope = from_tag.statuses
                     .where(id: timeline_status_ids)
+                    .where.not(account: into_account)
                     .where.not(account: into_account.following)
                     .tagged_with_none(TagFollow.where(account: into_account).pluck(:tag_id))
 

app/lib/link_details_extractor.rb

@@ -36,7 +36,8 @@ class LinkDetailsExtractor
     end
 
     def language
-      json['inLanguage']
+      lang = json['inLanguage']
+      lang.is_a?(Hash) ? (lang['alternateName'] || lang['name']) : lang
     end
 
     def type

app/models/concerns/attachmentable.rb

@@ -52,9 +52,13 @@ module Attachmentable
     return if attachment.blank? || !/image.*/.match?(attachment.content_type) || attachment.queued_for_write[:original].blank?
 
     width, height = FastImage.size(attachment.queued_for_write[:original].path)
-    matrix_limit  = attachment.content_type == 'image/gif' ? GIF_MATRIX_LIMIT : MAX_MATRIX_LIMIT
+    return unless width.present? && height.present?
 
-    raise Mastodon::DimensionsValidationError, "#{width}x#{height} images are not supported" if width.present? && height.present? && (width * height > matrix_limit)
+    if attachment.content_type == 'image/gif' && width * height > GIF_MATRIX_LIMIT
+      raise Mastodon::DimensionsValidationError, "#{width}x#{height} GIF files are not supported"
+    elsif width * height > MAX_MATRIX_LIMIT
+      raise Mastodon::DimensionsValidationError, "#{width}x#{height} images are not supported"
+    end
   end
 
   def appropriate_extension(attachment)

app/models/tag.rb

@@ -35,7 +35,7 @@ class Tag < ApplicationRecord
   HASHTAG_LAST_SEQUENCE = '([[:word:]_]*[[:alpha:]][[:word:]_]*)'
   HASHTAG_NAME_PAT = "#{HASHTAG_FIRST_SEQUENCE}|#{HASHTAG_LAST_SEQUENCE}"
 
-  HASHTAG_RE = %r{(?:^|[^/)\w])#(#{HASHTAG_NAME_PAT})}i
+  HASHTAG_RE = %r{(?<![=/)\w])#(#{HASHTAG_NAME_PAT})}i
   HASHTAG_NAME_RE = /\A(#{HASHTAG_NAME_PAT})\z/i
   HASHTAG_INVALID_CHARS_RE = /[^[:alnum:]#{HASHTAG_SEPARATORS}]/
 

app/models/trends/statuses.rb

@@ -106,7 +106,7 @@ class Trends::Statuses < Trends::Base
   private
 
   def eligible?(status)
-    status.public_visibility? && status.account.discoverable? && !status.account.silenced? && status.spoiler_text.blank? && !status.sensitive? && !status.reply? && valid_locale?(status.language)
+    status.public_visibility? && status.account.discoverable? && !status.account.silenced? && !status.account.sensitized? && status.spoiler_text.blank? && !status.sensitive? && !status.reply? && valid_locale?(status.language)
   end
 
   def calculate_scores(statuses, at_time)

app/serializers/rest/featured_tag_serializer.rb

@@ -10,7 +10,9 @@ class REST::FeaturedTagSerializer < ActiveModel::Serializer
   end
 
   def url
-    short_account_tag_url(object.account, object.tag)
+    # The path is hardcoded because we have to deal with both local and
+    # remote users, which are different routes
+    account_with_domain_url(object.account, "tagged/#{object.tag.to_param}")
   end
 
   def name

app/services/fan_out_on_write_service.rb

@@ -8,6 +8,7 @@ class FanOutOnWriteService < BaseService
   # @param [Hash] options
   # @option options [Boolean] update
   # @option options [Array<Integer>] silenced_account_ids
+  # @option options [Boolean] skip_notifications
   def call(status, options = {})
     @status    = status
     @account   = status.account
@@ -37,8 +38,11 @@ class FanOutOnWriteService < BaseService
 
   def fan_out_to_local_recipients!
     deliver_to_self!
-    notify_mentioned_accounts!
-    notify_about_update! if update?
+
+    unless @options[:skip_notifications]
+      notify_mentioned_accounts!
+      notify_about_update! if update?
+    end
 
     case @status.visibility.to_sym
     when :public, :unlisted, :private

app/services/follow_service.rb

@@ -71,7 +71,7 @@ class FollowService < BaseService
     if @target_account.local?
       LocalNotificationWorker.perform_async(@target_account.id, follow_request.id, follow_request.class.name, 'follow_request')
     elsif @target_account.activitypub?
-      ActivityPub::DeliveryWorker.perform_async(build_json(follow_request), @source_account.id, @target_account.inbox_url)
+      ActivityPub::DeliveryWorker.perform_async(build_json(follow_request), @source_account.id, @target_account.inbox_url, { 'bypass_availability' => true })
     end
 
     follow_request

app/workers/activitypub/delivery_worker.rb

@@ -23,9 +23,10 @@ class ActivityPub::DeliveryWorker
   HEADERS = { 'Content-Type' => 'application/activity+json' }.freeze
 
   def perform(json, source_account_id, inbox_url, options = {})
-    return unless DeliveryFailureTracker.available?(inbox_url)
-
     @options        = options.with_indifferent_access
+
+    return unless @options[:bypass_availability] || DeliveryFailureTracker.available?(inbox_url)
+
     @json           = json
     @source_account = Account.find(source_account_id)
     @inbox_url      = inbox_url

app/workers/thread_resolve_worker.rb

@@ -7,13 +7,18 @@ class ThreadResolveWorker
   sidekiq_options queue: 'pull', retry: 3
 
   def perform(child_status_id, parent_url, options = {})
-    child_status  = Status.find(child_status_id)
-    parent_status = FetchRemoteStatusService.new.call(parent_url, **options.deep_symbolize_keys)
+    child_status = Status.find(child_status_id)
+    return if child_status.in_reply_to_id.present?
+
+    parent_status = ActivityPub::TagManager.instance.uri_to_resource(parent_url, Status)
+    parent_status ||= FetchRemoteStatusService.new.call(parent_url, **options.deep_symbolize_keys)
 
     return if parent_status.nil?
 
     child_status.thread = parent_status
     child_status.save!
+
+    DistributionWorker.perform_async(child_status_id, { 'skip_notifications' => true }) if child_status.within_realtime_window?
   rescue ActiveRecord::RecordNotFound
     true
   end

config/initializers/content_security_policy.rb

@@ -5,7 +5,11 @@
 # https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy
 
 def host_to_url(str)
-  "http#{Rails.configuration.x.use_https ? 's' : ''}://#{str.split('/').first}" if str.present?
+  return if str.blank?
+
+  uri = Addressable::URI.parse("http#{Rails.configuration.x.use_https ? 's' : ''}://#{str}")
+  uri.path += '/' unless uri.path.blank? || uri.path.end_with?('/')
+  uri.to_s
 end
 
 base_host = Rails.configuration.x.web_domain

config/routes.rb

@@ -3,6 +3,18 @@
 require 'sidekiq_unique_jobs/web'
 require 'sidekiq-scheduler/web'
 
+class RedirectWithVary < ActionDispatch::Routing::PathRedirect
+  def serve(...)
+    super.tap do |_, headers, _|
+      headers['Vary'] = 'Origin, Accept'
+    end
+  end
+end
+
+def redirect_with_vary(path)
+  RedirectWithVary.new(301, path)
+end
+
 Rails.application.routes.draw do
   # Paths of routes on the web app that to not require to be indexed or
   # have alternative format representations requiring separate controllers
@@ -90,10 +102,13 @@ Rails.application.routes.draw do
     confirmations:      'auth/confirmations',
   }
 
-  get '/users/:username', to: redirect('/@%{username}'), constraints: lambda { |req| req.format.nil? || req.format.html? }
-  get '/users/:username/following', to: redirect('/@%{username}/following'), constraints: lambda { |req| req.format.nil? || req.format.html? }
-  get '/users/:username/followers', to: redirect('/@%{username}/followers'), constraints: lambda { |req| req.format.nil? || req.format.html? }
-  get '/users/:username/statuses/:id', to: redirect('/@%{username}/%{id}'), constraints: lambda { |req| req.format.nil? || req.format.html? }
+  # rubocop:disable Style/FormatStringToken - those do not go through the usual formatting functions and are not safe to correct
+  get '/users/:username', to: redirect_with_vary('/@%{username}'), constraints: lambda { |req| req.format.nil? || req.format.html? }
+  get '/users/:username/following', to: redirect_with_vary('/@%{username}/following'), constraints: lambda { |req| req.format.nil? || req.format.html? }
+  get '/users/:username/followers', to: redirect_with_vary('/@%{username}/followers'), constraints: lambda { |req| req.format.nil? || req.format.html? }
+  get '/users/:username/statuses/:id', to: redirect_with_vary('/@%{username}/%{id}'), constraints: lambda { |req| req.format.nil? || req.format.html? }
+  # rubocop:enable Style/FormatStringToken
+
   get '/authorize_follow', to: redirect { |_, request| "/authorize_interaction?#{request.params.to_query}" }
 
   resources :accounts, path: 'users', only: [:show], param: :username do
@@ -134,7 +149,7 @@ Rails.application.routes.draw do
     get '/@:account_username/:id/embed', to: 'statuses#embed', as: :embed_short_account_status
   end
 
-  get '/@:username_with_domain/(*any)', to: 'home#index', constraints: { username_with_domain: %r{([^/])+?} }, format: false
+  get '/@:username_with_domain/(*any)', to: 'home#index', constraints: { username_with_domain: %r{([^/])+?} }, as: :account_with_domain, format: false
   get '/settings', to: redirect('/settings/profile')
 
   draw(:settings)

docker-compose.yml

@@ -56,7 +56,7 @@ services:
 
   web:
     build: .
-    image: ghcr.io/mastodon/mastodon:v4.2.1
+    image: ghcr.io/mastodon/mastodon:v4.2.3
     restart: always
     env_file: .env.production
     command: bash -c "rm -f /mastodon/tmp/pids/server.pid; bundle exec rails s -p 3000"
@@ -77,7 +77,7 @@ services:
 
   streaming:
     build: .
-    image: ghcr.io/mastodon/mastodon:v4.2.1
+    image: ghcr.io/mastodon/mastodon:v4.2.3
     restart: always
     env_file: .env.production
     command: node ./streaming
@@ -95,7 +95,7 @@ services:
 
   sidekiq:
     build: .
-    image: ghcr.io/mastodon/mastodon:v4.2.1
+    image: ghcr.io/mastodon/mastodon:v4.2.3
     restart: always
     env_file: .env.production
     command: bundle exec sidekiq

lib/mastodon/version.rb

@@ -13,7 +13,7 @@ module Mastodon
     end
 
     def patch
-      1
+      3
     end
 
     def default_prerelease

spec/controllers/api/v1/accounts/featured_tags_controller_spec.rb

@@ -1,23 +0,0 @@
-# frozen_string_literal: true
-
-require 'rails_helper'
-
-describe Api::V1::Accounts::FeaturedTagsController do
-  render_views
-
-  let(:user)    { Fabricate(:user) }
-  let(:token)   { Fabricate(:accessible_access_token, resource_owner_id: user.id, scopes: 'read:accounts') }
-  let(:account) { Fabricate(:account) }
-
-  before do
-    allow(controller).to receive(:doorkeeper_token) { token }
-  end
-
-  describe 'GET #index' do
-    it 'returns http success' do
-      get :index, params: { account_id: account.id, limit: 2 }
-
-      expect(response).to have_http_status(200)
-    end
-  end
-end

spec/lib/activitypub/activity/create_spec.rb

@@ -23,6 +23,109 @@ RSpec.describe ActivityPub::Activity::Create do
     stub_request(:get, 'http://example.com/emojib.png').to_return(body: attachment_fixture('emojo.png'), headers: { 'Content-Type' => 'application/octet-stream' })
   end
 
+  describe 'processing posts received out of order' do
+    let(:follower) { Fabricate(:account, username: 'bob') }
+
+    let(:object_json) do
+      {
+        id: [ActivityPub::TagManager.instance.uri_for(sender), 'post1'].join('/'),
+        type: 'Note',
+        to: [
+          'https://www.w3.org/ns/activitystreams#Public',
+          ActivityPub::TagManager.instance.uri_for(follower),
+        ],
+        content: '@bob lorem ipsum',
+        published: 1.hour.ago.utc.iso8601,
+        updated: 1.hour.ago.utc.iso8601,
+        tag: {
+          type: 'Mention',
+          href: ActivityPub::TagManager.instance.uri_for(follower),
+        },
+      }
+    end
+
+    let(:reply_json) do
+      {
+        id: [ActivityPub::TagManager.instance.uri_for(sender), 'reply'].join('/'),
+        type: 'Note',
+        inReplyTo: object_json[:id],
+        to: [
+          'https://www.w3.org/ns/activitystreams#Public',
+          ActivityPub::TagManager.instance.uri_for(follower),
+        ],
+        content: '@bob lorem ipsum',
+        published: Time.now.utc.iso8601,
+        updated: Time.now.utc.iso8601,
+        tag: {
+          type: 'Mention',
+          href: ActivityPub::TagManager.instance.uri_for(follower),
+        },
+      }
+    end
+
+    def activity_for_object(json)
+      {
+        '@context': 'https://www.w3.org/ns/activitystreams',
+        id: [json[:id], 'activity'].join('/'),
+        type: 'Create',
+        actor: ActivityPub::TagManager.instance.uri_for(sender),
+        object: json,
+      }.with_indifferent_access
+    end
+
+    before do
+      follower.follow!(sender)
+    end
+
+    around do |example|
+      Sidekiq::Testing.fake! do
+        example.run
+        Sidekiq::Worker.clear_all
+      end
+    end
+
+    it 'correctly processes posts and inserts them in timelines', :aggregate_failures do
+      # Simulate a temporary failure preventing from fetching the parent post
+      stub_request(:get, object_json[:id]).to_return(status: 500)
+
+      # When receiving the reply…
+      described_class.new(activity_for_object(reply_json), sender, delivery: true).perform
+
+      # NOTE: Refering explicitly to the workers is a bit awkward
+      DistributionWorker.drain
+      FeedInsertWorker.drain
+
+      # …it creates a status with an unknown parent
+      reply = Status.find_by(uri: reply_json[:id])
+      expect(reply.reply?).to be true
+      expect(reply.in_reply_to_id).to be_nil
+
+      # …and creates a notification
+      expect(LocalNotificationWorker.jobs.size).to eq 1
+
+      # …but does not insert it into timelines
+      expect(redis.zscore(FeedManager.instance.key(:home, follower.id), reply.id)).to be_nil
+
+      # When receiving the parent…
+      described_class.new(activity_for_object(object_json), sender, delivery: true).perform
+
+      Sidekiq::Worker.drain_all
+
+      # …it creates a status and insert it into timelines
+      parent = Status.find_by(uri: object_json[:id])
+      expect(parent.reply?).to be false
+      expect(parent.in_reply_to_id).to be_nil
+      expect(reply.reload.in_reply_to_id).to eq parent.id
+
+      # Check that the both statuses have been inserted into the home feed
+      expect(redis.zscore(FeedManager.instance.key(:home, follower.id), parent.id)).to be_within(0.1).of(parent.id.to_f)
+      expect(redis.zscore(FeedManager.instance.key(:home, follower.id), reply.id)).to be_within(0.1).of(reply.id.to_f)
+
+      # Creates two notifications
+      expect(Notification.count).to eq 2
+    end
+  end
+
   describe '#perform' do
     context 'when fetching' do
       subject { described_class.new(json, sender) }
@@ -31,6 +134,46 @@ RSpec.describe ActivityPub::Activity::Create do
         subject.perform
       end
 
+      context 'when object publication date is below ISO8601 range' do
+        let(:object_json) do
+          {
+            id: [ActivityPub::TagManager.instance.uri_for(sender), '#bar'].join,
+            type: 'Note',
+            content: 'Lorem ipsum',
+            published: '-0977-11-03T08:31:22Z',
+          }
+        end
+
+        it 'creates status with a valid creation date', :aggregate_failures do
+          status = sender.statuses.first
+
+          expect(status).to_not be_nil
+          expect(status.text).to eq 'Lorem ipsum'
+
+          expect(status.created_at).to be_within(30).of(Time.now.utc)
+        end
+      end
+
+      context 'when object publication date is above ISO8601 range' do
+        let(:object_json) do
+          {
+            id: [ActivityPub::TagManager.instance.uri_for(sender), '#bar'].join,
+            type: 'Note',
+            content: 'Lorem ipsum',
+            published: '10000-11-03T08:31:22Z',
+          }
+        end
+
+        it 'creates status with a valid creation date', :aggregate_failures do
+          status = sender.statuses.first
+
+          expect(status).to_not be_nil
+          expect(status.text).to eq 'Lorem ipsum'
+
+          expect(status.created_at).to be_within(30).of(Time.now.utc)
+        end
+      end
+
       context 'when object has been edited' do
         let(:object_json) do
           {
@@ -42,18 +185,16 @@ RSpec.describe ActivityPub::Activity::Create do
           }
         end
 
-        it 'creates status' do
+        it 'creates status with appropriate creation and edition dates', :aggregate_failures do
           status = sender.statuses.first
 
           expect(status).to_not be_nil
           expect(status.text).to eq 'Lorem ipsum'
-        end
 
-        it 'marks status as edited' do
-          status = sender.statuses.first
+          expect(status.created_at).to eq '2022-01-22T15:00:00Z'.to_datetime
 
-          expect(status).to_not be_nil
           expect(status.edited?).to be true
+          expect(status.edited_at).to eq '2022-01-22T16:00:00Z'.to_datetime
         end
       end
 

spec/lib/activitypub/linked_data_signature_spec.rb

@@ -38,6 +38,40 @@ RSpec.describe ActivityPub::LinkedDataSignature do
       end
     end
 
+    context 'when local account record is missing a public key' do
+      let(:raw_signature) do
+        {
+          'creator' => 'http://example.com/alice',
+          'created' => '2017-09-23T20:21:34Z',
+        }
+      end
+
+      let(:signature) { raw_signature.merge('type' => 'RsaSignature2017', 'signatureValue' => sign(sender, raw_signature, raw_json)) }
+
+      let(:service_stub) { instance_double(ActivityPub::FetchRemoteKeyService) }
+
+      before do
+        # Ensure signature is computed with the old key
+        signature
+
+        # Unset key
+        old_key = sender.public_key
+        sender.update!(private_key: '', public_key: '')
+
+        allow(ActivityPub::FetchRemoteKeyService).to receive(:new).and_return(service_stub)
+
+        allow(service_stub).to receive(:call).with('http://example.com/alice', id: false) do
+          sender.update!(public_key: old_key)
+          sender
+        end
+      end
+
+      it 'fetches key and returns creator' do
+        expect(subject.verify_actor!).to eq sender
+        expect(service_stub).to have_received(:call).with('http://example.com/alice', id: false).once
+      end
+    end
+
     context 'when signature is missing' do
       let(:signature) { nil }
 

spec/lib/feed_manager_spec.rb

@@ -525,6 +525,44 @@ RSpec.describe FeedManager do
     end
   end
 
+  describe '#unmerge_tag_from_home' do
+    let(:receiver) { Fabricate(:account) }
+    let(:tag) { Fabricate(:tag) }
+
+    it 'leaves a tagged status' do
+      status = Fabricate(:status)
+      status.tags << tag
+      described_class.instance.push_to_home(receiver, status)
+
+      described_class.instance.unmerge_tag_from_home(tag, receiver)
+
+      expect(redis.zrange("feed:home:#{receiver.id}", 0, -1)).to_not include(status.id.to_s)
+    end
+
+    it 'remains a tagged status written by receiver\'s followee' do
+      followee = Fabricate(:account)
+      receiver.follow!(followee)
+
+      status = Fabricate(:status, account: followee)
+      status.tags << tag
+      described_class.instance.push_to_home(receiver, status)
+
+      described_class.instance.unmerge_tag_from_home(tag, receiver)
+
+      expect(redis.zrange("feed:home:#{receiver.id}", 0, -1)).to include(status.id.to_s)
+    end
+
+    it 'remains a tagged status written by receiver' do
+      status = Fabricate(:status, account: receiver)
+      status.tags << tag
+      described_class.instance.push_to_home(receiver, status)
+
+      described_class.instance.unmerge_tag_from_home(tag, receiver)
+
+      expect(redis.zrange("feed:home:#{receiver.id}", 0, -1)).to include(status.id.to_s)
+    end
+  end
+
   describe '#clear_from_home' do
     let(:account) { Fabricate(:account) }
     let(:followed_account) { Fabricate(:account) }

spec/lib/link_details_extractor_spec.rb

@@ -82,6 +82,10 @@ RSpec.describe LinkDetailsExtractor do
           'name' => 'Pet News',
           'url' => 'https://example.com',
         },
+        'inLanguage' => {
+          name: 'English',
+          alternateName: 'en',
+        },
       }.to_json
     end
 
@@ -115,6 +119,12 @@ RSpec.describe LinkDetailsExtractor do
           expect(subject.provider_name).to eq 'Pet News'
         end
       end
+
+      describe '#language' do
+        it 'returns the language from structured data' do
+          expect(subject.language).to eq 'en'
+        end
+      end
     end
 
     context 'when is wrapped in CDATA tags' do

spec/models/tag_spec.rb

@@ -32,44 +32,52 @@ RSpec.describe Tag do
       expect(subject.match('https://en.wikipedia.org/wiki/Ghostbusters_(song)#Lawsuit')).to be_nil
     end
 
+    it 'does not match URLs with hashtag-like anchors after a numeral' do
+      expect(subject.match('https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111895#c4')).to be_nil
+    end
+
+    it 'does not match URLs with hashtag-like anchors after an empty query parameter' do
+      expect(subject.match('https://en.wikipedia.org/wiki/Ghostbusters_(song)?foo=#Lawsuit')).to be_nil
+    end
+
     it 'matches #ａｅｓｔｈｅｔｉｃ' do
-      expect(subject.match('this is #ａｅｓｔｈｅｔｉｃ').to_s).to eq ' #ａｅｓｔｈｅｔｉｃ'
+      expect(subject.match('this is #ａｅｓｔｈｅｔｉｃ').to_s).to eq '#ａｅｓｔｈｅｔｉｃ'
     end
 
     it 'matches digits at the start' do
-      expect(subject.match('hello #3d').to_s).to eq ' #3d'
+      expect(subject.match('hello #3d').to_s).to eq '#3d'
     end
 
     it 'matches digits in the middle' do
-      expect(subject.match('hello #l33ts35k').to_s).to eq ' #l33ts35k'
+      expect(subject.match('hello #l33ts35k').to_s).to eq '#l33ts35k'
     end
 
     it 'matches digits at the end' do
-      expect(subject.match('hello #world2016').to_s).to eq ' #world2016'
+      expect(subject.match('hello #world2016').to_s).to eq '#world2016'
     end
 
     it 'matches underscores at the beginning' do
-      expect(subject.match('hello #_test').to_s).to eq ' #_test'
+      expect(subject.match('hello #_test').to_s).to eq '#_test'
     end
 
     it 'matches underscores at the end' do
-      expect(subject.match('hello #test_').to_s).to eq ' #test_'
+      expect(subject.match('hello #test_').to_s).to eq '#test_'
     end
 
     it 'matches underscores in the middle' do
-      expect(subject.match('hello #one_two_three').to_s).to eq ' #one_two_three'
+      expect(subject.match('hello #one_two_three').to_s).to eq '#one_two_three'
     end
 
     it 'matches middle dots' do
-      expect(subject.match('hello #one·two·three').to_s).to eq ' #one·two·three'
+      expect(subject.match('hello #one·two·three').to_s).to eq '#one·two·three'
     end
 
     it 'matches ・unicode in ぼっち・ざ・ろっく correctly' do
-      expect(subject.match('testing #ぼっち・ざ・ろっく').to_s).to eq ' #ぼっち・ざ・ろっく'
+      expect(subject.match('testing #ぼっち・ざ・ろっく').to_s).to eq '#ぼっち・ざ・ろっく'
     end
 
     it 'matches ZWNJ' do
-      expect(subject.match('just add #نرم‌افزار and').to_s).to eq ' #نرم‌افزار'
+      expect(subject.match('just add #نرم‌افزار and').to_s).to eq '#نرم‌افزار'
     end
 
     it 'does not match middle dots at the start' do
@@ -77,7 +85,7 @@ RSpec.describe Tag do
     end
 
     it 'does not match middle dots at the end' do
-      expect(subject.match('hello #one·two·three·').to_s).to eq ' #one·two·three'
+      expect(subject.match('hello #one·two·three·').to_s).to eq '#one·two·three'
     end
 
     it 'does not match purely-numeric hashtags' do

spec/requests/api/v1/accounts/featured_tags_spec.rb

@@ -0,0 +1,50 @@
+# frozen_string_literal: true
+
+require 'rails_helper'
+
+RSpec.describe 'account featured tags API' do
+  let(:user)     { Fabricate(:user) }
+  let(:token)    { Fabricate(:accessible_access_token, resource_owner_id: user.id, scopes: scopes) }
+  let(:scopes)   { 'read:accounts' }
+  let(:headers)  { { 'Authorization' => "Bearer #{token.token}" } }
+  let(:account)  { Fabricate(:account) }
+
+  describe 'GET /api/v1/accounts/:id/featured_tags' do
+    subject do
+      get "/api/v1/accounts/#{account.id}/featured_tags", headers: headers
+    end
+
+    before do
+      account.featured_tags.create!(name: 'foo')
+      account.featured_tags.create!(name: 'bar')
+    end
+
+    it 'returns the expected tags', :aggregate_failures do
+      subject
+
+      expect(response).to have_http_status(200)
+      expect(body_as_json).to contain_exactly(a_hash_including({
+        name: 'bar',
+        url: "https://cb6e6126.ngrok.io/@#{account.username}/tagged/bar",
+      }), a_hash_including({
+        name: 'foo',
+        url: "https://cb6e6126.ngrok.io/@#{account.username}/tagged/foo",
+      }))
+    end
+
+    context 'when the account is remote' do
+      it 'returns the expected tags', :aggregate_failures do
+        subject
+
+        expect(response).to have_http_status(200)
+        expect(body_as_json).to contain_exactly(a_hash_including({
+          name: 'bar',
+          url: "https://cb6e6126.ngrok.io/@#{account.pretty_acct}/tagged/bar",
+        }), a_hash_including({
+          name: 'foo',
+          url: "https://cb6e6126.ngrok.io/@#{account.pretty_acct}/tagged/foo",
+        }))
+      end
+    end
+  end
+end

spec/requests/cache_spec.rb

@@ -124,7 +124,7 @@ describe 'Caching behavior' do
       expect(response.cookies).to be_empty
     end
 
-    it 'sets public cache control' do
+    it 'sets public cache control', :aggregate_failures do
       # expect(response.cache_control[:max_age]&.to_i).to be_positive
       expect(response.cache_control[:public]).to be_truthy
       expect(response.cache_control[:private]).to be_falsy
@@ -141,11 +141,8 @@ describe 'Caching behavior' do
   end
 
   shared_examples 'non-cacheable error' do
-    it 'does not return HTTP success' do
+    it 'does not return HTTP success and does not have cache headers', :aggregate_failures do
       expect(response).to_not have_http_status(200)
-    end
-
-    it 'does not have cache headers' do
       expect(response.cache_control[:public]).to be_falsy
     end
   end
@@ -182,6 +179,15 @@ describe 'Caching behavior' do
   end
 
   context 'when anonymously accessed' do
+    describe '/users/alice' do
+      it 'redirects with proper cache header', :aggregate_failures do
+        get '/users/alice'
+
+        expect(response).to redirect_to('/@alice')
+        expect(response.headers['Vary']&.split(',')&.map { |x| x.strip.downcase }).to include('accept')
+      end
+    end
+
     TestEndpoints::ALWAYS_CACHED.each do |endpoint|
       describe endpoint do
         before { get endpoint }

spec/serializers/activitypub/note_serializer_spec.rb

@@ -7,7 +7,7 @@ describe ActivityPub::NoteSerializer do
 
   let!(:account) { Fabricate(:account) }
   let!(:other) { Fabricate(:account) }
-  let!(:parent) { Fabricate(:status, account: account, visibility: :public) }
+  let!(:parent) { Fabricate(:status, account: account, visibility: :public, language: 'zh-TW') }
   let!(:reply_by_account_first) { Fabricate(:status, account: account, thread: parent, visibility: :public) }
   let!(:reply_by_account_next) { Fabricate(:status, account: account, thread: parent, visibility: :public) }
   let!(:reply_by_other_first) { Fabricate(:status, account: other, thread: parent, visibility: :public) }
@@ -18,8 +18,15 @@ describe ActivityPub::NoteSerializer do
     @serialization = ActiveModelSerializers::SerializableResource.new(parent, serializer: described_class, adapter: ActivityPub::Adapter)
   end
 
-  it 'has a Note type' do
-    expect(subject['type']).to eql('Note')
+  it 'has the expected shape' do
+    expect(subject).to include({
+      '@context' => include('https://www.w3.org/ns/activitystreams'),
+      'type' => 'Note',
+      'attributedTo' => ActivityPub::TagManager.instance.uri_for(account),
+      'contentMap' => include({
+        'zh-TW' => a_kind_of(String),
+      }),
+    })
   end
 
   it 'has a replies collection' do