kikobar/mastodon
1
0
Fork 0
mirror of https://github.com/kikobar/mastodon.git synced 2024-11-16 12:12:14 +00:00
Code Issues Packages Projects Releases Wiki Activity

Revamp post filtering system (#18058)

Browse source 
* Add model for custom filter keywords

* Use CustomFilterKeyword internally

Does not change the API

* Fix /filters/edit and /filters/new

* Add migration tests

* Remove whole_word column from custom_filters (covered by custom_filter_keywords)

* Redesign /filters

Instead of a list, present a card that displays more information and handles
multiple keywords per filter.

* Redesign /filters/new and /filters/edit to add and remove keywords

This adds a new gem dependency: cocoon, as well as a npm dependency:
cocoon-js-vanilla. Those are used to easily populate and remove form fields
from the user interface when manipulating multiple keyword filters at once.

* Add /api/v2/filters to edit filter with multiple keywords

Entities:
- `Filter`: `id`, `title`, `filter_action` (either `hide` or `warn`), `context`
  `keywords`
- `FilterKeyword`: `id`, `keyword`, `whole_word`

API endpoits:
- `GET /api/v2/filters` to list filters (including keywords)
- `POST /api/v2/filters` to create a new filter
  `keywords_attributes` can also be passed to create keywords in one request
- `GET /api/v2/filters/:id` to read a particular filter
- `PUT /api/v2/filters/:id` to update a new filter
  `keywords_attributes` can also be passed to edit, delete or add keywords in
   one request
- `DELETE /api/v2/filters/:id` to delete a particular filter
- `GET /api/v2/filters/:id/keywords` to list keywords for a filter
- `POST /api/v2/filters/:filter_id/keywords/:id` to add a new keyword to a
   filter
- `GET /api/v2/filter_keywords/:id` to read a particular keyword
- `PUT /api/v2/filter_keywords/:id` to edit a particular keyword
- `DELETE /api/v2/filter_keywords/:id` to delete a particular keyword

* Change from `irreversible` boolean to `action` enum

* Remove irrelevent `irreversible_must_be_within_context` check

* Fix /filters/new and /filters/edit with update for filter_action

* Fix Rubocop/Codeclimate complaining about task names

* Refactor FeedManager#phrase_filtered?

This moves regexp building and filter caching to the `CustomFilter` class.

This does not change the functional behavior yet, but this changes how the
cache is built, doing per-custom_filter regexps so that filters can be matched
independently, while still offering caching.

* Perform server-side filtering and output result in REST API

* Fix numerous filters_changed events being sent when editing multiple keywords at once

* Add some tests

* Use the new API in the WebUI

- use client-side logic for filters we have fetched rules for.
  This is so that filter changes can be retroactively applied without
  reloading the UI.
- use server-side logic for filters we haven't fetched rules for yet
  (e.g. network error, or initial timeline loading)

* Minor optimizations and refactoring

* Perform server-side filtering on the streaming server

* Change the wording of filter action labels

* Fix issues pointed out by linter

* Change design of “Show anyway” link in accordence to review comments

* Drop “irreversible” filtering behavior

* Move /api/v2/filter_keywords to /api/v1/filters/keywords

* Rename `filter_results` attribute to `filtered`

* Rename REST::LegacyFilterSerializer to REST::V1::FilterSerializer

* Fix systemChannelId value in streaming server

* Simplify code by removing client-side filtering code

The simplifcation comes at a cost though: filters aren't retroactively
applied anymore.
This commit is contained in:
Claire 2022-06-28 09:42:13 +02:00 committed by GitHub
parent 5823ae70c4
commit 02851848e9
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
60 changed files with 1292 additions and 250 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

18
.circleci/config.yml
View file

@ -133,6 +133,12 @@ jobs:
- run:
command: ./bin/rails tests:migrations:populate_v2_4
name: Populate database with test data
- run:
command: ./bin/rails db:migrate VERSION=20180707154237
name: Run migrations up to v2.4.3
- run:
command: ./bin/rails tests:migrations:populate_v2_4_3
name: Populate database with test data
- run:
command: ./bin/rails db:migrate
name: Run all remaining migrations
@ -167,14 +173,22 @@ jobs:
- run:
command: ./bin/rails tests:migrations:populate_v2_4
name: Populate database with test data
- run:
command: ./bin/rails db:migrate VERSION=20180707154237
name: Run migrations up to v2.4.3
environment:
SKIP_POST_DEPLOYMENT_MIGRATIONS: true
- run:
command: ./bin/rails tests:migrations:populate_v2_4_3
name: Populate database with test data
- run:
command: ./bin/rails db:migrate
name: Run all pre-deployment migrations
name: Run all remaining pre-deployment migrations
environment:
SKIP_POST_DEPLOYMENT_MIGRATIONS: true
- run:
command: ./bin/rails db:migrate
name: Run all post-deployment remaining migrations
name: Run all post-deployment migrations
- run:
command: ./bin/rails tests:migrations:check_database
name: Check migration result

2
Gemfile
View file

@ -153,3 +153,5 @@ gem 'concurrent-ruby', require: false
gem 'connection_pool', require: false
gem 'xorcist', '~> 1.1'
gem 'cocoon', '~> 1.2'

2
Gemfile.lock
View file

@ -163,6 +163,7 @@ GEM
elasticsearch-dsl
chunky_png (1.4.0)
climate_control (0.2.0)
cocoon (1.2.15)
coderay (1.1.3)
color_diff (0.1)
concurrent-ruby (1.1.10)
@ -746,6 +747,7 @@ DEPENDENCIES
charlock_holmes (~> 0.7.7)
chewy (~> 7.2)
climate_control (~> 0.2)
cocoon (~> 1.2)
color_diff (~> 0.1)
concurrent-ruby
connection_pool

50
app/controllers/api/v1/filters/keywords_controller.rb Normal file
View file

@ -0,0 +1,50 @@
# frozen_string_literal: true
class Api::V1::Filters::KeywordsController < Api::BaseController
before_action -> { doorkeeper_authorize! :read, :'read:filters' }, only: [:index, :show]
before_action -> { doorkeeper_authorize! :write, :'write:filters' }, except: [:index, :show]
before_action :require_user!
before_action :set_keywords, only: :index
before_action :set_keyword, only: [:show, :update, :destroy]
def index
render json: @keywords, each_serializer: REST::FilterKeywordSerializer
end
def create
@keyword = current_account.custom_filters.find(params[:filter_id]).keywords.create!(resource_params)
render json: @keyword, serializer: REST::FilterKeywordSerializer
end
def show
render json: @keyword, serializer: REST::FilterKeywordSerializer
end
def update
@keyword.update!(resource_params)
render json: @keyword, serializer: REST::FilterKeywordSerializer
end
def destroy
@keyword.destroy!
render_empty
end
private
def set_keywords
filter = current_account.custom_filters.includes(:keywords).find(params[:filter_id])
@keywords = filter.keywords
end
def set_keyword
@keyword = CustomFilterKeyword.includes(:custom_filter).where(custom_filter: { account: current_account }).find(params[:id])
end
def resource_params
params.permit(:keyword, :whole_word)
end
end

35
app/controllers/api/v1/filters_controller.rb
View file

@ -8,21 +8,32 @@ class Api::V1::FiltersController < Api::BaseController
before_action :set_filter, only: [:show, :update, :destroy]
def index
render json: @filters, each_serializer: REST::FilterSerializer
render json: @filters, each_serializer: REST::V1::FilterSerializer
end
def create
@filter = current_account.custom_filters.create!(resource_params)
render json: @filter, serializer: REST::FilterSerializer
ApplicationRecord.transaction do
filter_category = current_account.custom_filters.create!(resource_params)
@filter = filter_category.keywords.create!(keyword_params)
end
render json: @filter, serializer: REST::V1::FilterSerializer
end
def show
render json: @filter, serializer: REST::FilterSerializer
render json: @filter, serializer: REST::V1::FilterSerializer
end
def update
@filter.update!(resource_params)
render json: @filter, serializer: REST::FilterSerializer
ApplicationRecord.transaction do
@filter.update!(keyword_params)
@filter.custom_filter.assign_attributes(filter_params)
raise Mastodon::ValidationError, I18n.t('filters.errors.deprecated_api_multiple_keywords') if @filter.custom_filter.changed? && @filter.custom_filter.keywords.count > 1
@filter.custom_filter.save!
end
render json: @filter, serializer: REST::V1::FilterSerializer
end
def destroy
@ -33,14 +44,22 @@ class Api::V1::FiltersController < Api::BaseController
private
def set_filters
@filters = current_account.custom_filters
@filters = CustomFilterKeyword.includes(:custom_filter).where(custom_filter: { account: current_account })
end
def set_filter
@filter = current_account.custom_filters.find(params[:id])
@filter = CustomFilterKeyword.includes(:custom_filter).where(custom_filter: { account: current_account }).find(params[:id])
end
def resource_params
params.permit(:phrase, :expires_in, :irreversible, :whole_word, context: [])
end
def filter_params
resource_params.slice(:expires_in, :irreversible, :context)
end
def keyword_params
resource_params.slice(:phrase, :whole_word)
end
end

48
app/controllers/api/v2/filters_controller.rb Normal file
View file

@ -0,0 +1,48 @@
# frozen_string_literal: true
class Api::V2::FiltersController < Api::BaseController
before_action -> { doorkeeper_authorize! :read, :'read:filters' }, only: [:index, :show]
before_action -> { doorkeeper_authorize! :write, :'write:filters' }, except: [:index, :show]
before_action :require_user!
before_action :set_filters, only: :index
before_action :set_filter, only: [:show, :update, :destroy]
def index
render json: @filters, each_serializer: REST::FilterSerializer, rules_requested: true
end
def create
@filter = current_account.custom_filters.create!(resource_params)
render json: @filter, serializer: REST::FilterSerializer, rules_requested: true
end
def show
render json: @filter, serializer: REST::FilterSerializer, rules_requested: true
end
def update
@filter.update!(resource_params)
render json: @filter, serializer: REST::FilterSerializer, rules_requested: true
end
def destroy
@filter.destroy!
render_empty
end
private
def set_filters
@filters = current_account.custom_filters.includes(:keywords)
end
def set_filter
@filter = current_account.custom_filters.find(params[:id])
end
def resource_params
params.permit(:title, :expires_in, :filter_action, context: [], keywords_attributes: [:id, :keyword, :whole_word, :_destroy])
end
end

12
app/controllers/filters_controller.rb
View file

@ -4,16 +4,16 @@ class FiltersController < ApplicationController
layout 'admin'
before_action :authenticate_user!
before_action :set_filters, only: :index
before_action :set_filter, only: [:edit, :update, :destroy]
before_action :set_body_classes
def index
@filters = current_account.custom_filters.order(:phrase)
@filters = current_account.custom_filters.includes(:keywords).order(:phrase)
end
def new
@filter = current_account.custom_filters.build
@filter = current_account.custom_filters.build(action: :warn)
@filter.keywords.build
end
def create
@ -43,16 +43,12 @@ class FiltersController < ApplicationController
private
def set_filters
@filters = current_account.custom_filters
end
def set_filter
@filter = current_account.custom_filters.find(params[:id])
end
def resource_params
params.require(:custom_filter).permit(:phrase, :expires_in, :irreversible, :whole_word, context: [])
params.require(:custom_filter).permit(:title, :expires_in, :filter_action, context: [], keywords_attributes: [:id, :keyword, :whole_word, :_destroy])
end
def set_body_classes

26
app/javascript/mastodon/actions/filters.js
View file

@ -1,26 +0,0 @@
import api from '../api';
export const FILTERS_FETCH_REQUEST = 'FILTERS_FETCH_REQUEST';
export const FILTERS_FETCH_SUCCESS = 'FILTERS_FETCH_SUCCESS';
export const FILTERS_FETCH_FAIL = 'FILTERS_FETCH_FAIL';
export const fetchFilters = () => (dispatch, getState) => {
dispatch({
type: FILTERS_FETCH_REQUEST,
skipLoading: true,
});
api(getState)
.get('/api/v1/filters')
.then(({ data }) => dispatch({
type: FILTERS_FETCH_SUCCESS,
filters: data,
skipLoading: true,
}))
.catch(err => dispatch({
type: FILTERS_FETCH_FAIL,
err,
skipLoading: true,
skipAlert: true,
}));
};

11
app/javascript/mastodon/actions/importer/index.js
View file

@ -5,6 +5,7 @@ export const ACCOUNTS_IMPORT = 'ACCOUNTS_IMPORT';
export const STATUS_IMPORT = 'STATUS_IMPORT';
export const STATUSES_IMPORT = 'STATUSES_IMPORT';
export const POLLS_IMPORT = 'POLLS_IMPORT';
export const FILTERS_IMPORT = 'FILTERS_IMPORT';
function pushUnique(array, object) {
if (array.every(element => element.id !== object.id)) {
@ -28,6 +29,10 @@ export function importStatuses(statuses) {
return { type: STATUSES_IMPORT, statuses };
}
export function importFilters(filters) {
return { type: FILTERS_IMPORT, filters };
}
export function importPolls(polls) {
return { type: POLLS_IMPORT, polls };
}
@ -61,11 +66,16 @@ export function importFetchedStatuses(statuses) {
const accounts = [];
const normalStatuses = [];
const polls = [];
const filters = [];
function processStatus(status) {
pushUnique(normalStatuses, normalizeStatus(status, getState().getIn(['statuses', status.id])));
pushUnique(accounts, status.account);
if (status.filtered) {
status.filtered.forEach(result => pushUnique(filters, result.filter));
}
if (status.reblog && status.reblog.id) {
processStatus(status.reblog);
}
@ -80,6 +90,7 @@ export function importFetchedStatuses(statuses) {
dispatch(importPolls(polls));
dispatch(importFetchedAccounts(accounts));
dispatch(importStatuses(normalStatuses));
dispatch(importFilters(filters));
};
}

12
app/javascript/mastodon/actions/importer/normalizer.js
View file

@ -42,6 +42,14 @@ export function normalizeAccount(account) {
return account;
}
export function normalizeFilterResult(result) {
const normalResult = { ...result };
normalResult.filter = normalResult.filter.id;
return normalResult;
}
export function normalizeStatus(status, normalOldStatus) {
const normalStatus = { ...status };
normalStatus.account = status.account.id;
@ -54,6 +62,10 @@ export function normalizeStatus(status, normalOldStatus) {
normalStatus.poll = status.poll.id;
}
if (status.filtered) {
normalStatus.filtered = status.filtered.map(normalizeFilterResult);
}
// Only calculate these values when status first encountered and
// when the underlying values change. Otherwise keep the ones
// already in the reducer

13
app/javascript/mastodon/actions/notifications.js
View file

@ -12,10 +12,8 @@ import { saveSettings } from './settings';
import { defineMessages } from 'react-intl';
import { List as ImmutableList } from 'immutable';
import { unescapeHTML } from '../utils/html';
import { getFiltersRegex } from '../selectors';
import { usePendingItems as preferPendingItems } from 'mastodon/initial_state';
import compareId from 'mastodon/compare_id';
import { searchTextFromRawStatus } from 'mastodon/actions/importer/normalizer';
import { requestNotificationPermission } from '../utils/notifications';
export const NOTIFICATIONS_UPDATE = 'NOTIFICATIONS_UPDATE';
@ -62,20 +60,17 @@ export function updateNotifications(notification, intlMessages, intlLocale) {
const showInColumn = activeFilter === 'all' ? getState().getIn(['settings', 'notifications', 'shows', notification.type], true) : activeFilter === notification.type;
const showAlert = getState().getIn(['settings', 'notifications', 'alerts', notification.type], true);
const playSound = getState().getIn(['settings', 'notifications', 'sounds', notification.type], true);
const filters = getFiltersRegex(getState(), { contextType: 'notifications' });
let filtered = false;
if (['mention', 'status'].includes(notification.type)) {
const dropRegex = filters[0];
const regex = filters[1];
const searchIndex = searchTextFromRawStatus(notification.status);
if (['mention', 'status'].includes(notification.type) && notification.status.filtered) {
const filters = notification.status.filtered.filter(result => result.filter.context.includes('notifications'));
if (dropRegex && dropRegex.test(searchIndex)) {
if (filters.some(result => result.filter.filter_action === 'hide')) {
return;
}
filtered = regex && regex.test(searchIndex);
filtered = filters.length > 0;
}
if (['follow_request'].includes(notification.type)) {

4
app/javascript/mastodon/actions/streaming.js
View file

@ -21,7 +21,6 @@ import {
updateReaction as updateAnnouncementsReaction,
deleteAnnouncement,
} from './announcements';
import { fetchFilters } from './filters';
import { getLocale } from '../locales';
const { messages } = getLocale();
@ -97,9 +96,6 @@ export const connectTimelineStream = (timelineId, channelName, params = {}, opti
case 'conversation':
dispatch(updateConversations(JSON.parse(data.payload)));
break;
case 'filters_changed':
dispatch(fetchFilters());
break;
case 'announcement':
dispatch(updateAnnouncements(JSON.parse(data.payload)));
break;

21
app/javascript/mastodon/components/status.js
View file

@ -116,6 +116,7 @@ class Status extends ImmutablePureComponent {
state = {
showMedia: defaultMediaVisibility(this.props.status),
statusId: undefined,
forceFilter: undefined,
};
static getDerivedStateFromProps(nextProps, prevState) {
@ -277,6 +278,15 @@ class Status extends ImmutablePureComponent {
this.handleToggleMediaVisibility();
}
handleUnfilterClick = e => {
this.setState({ forceFilter: false });
e.preventDefault();
}
handleFilterClick = () => {
this.setState({ forceFilter: true });
}
_properStatus () {
const { status } = this.props;
@ -328,7 +338,8 @@ class Status extends ImmutablePureComponent {
);
}
if (status.get('filtered') || status.getIn(['reblog', 'filtered'])) {
const matchedFilters = status.get('filtered') || status.getIn(['reblog', 'filtered']);
if (this.state.forceFilter === undefined ? matchedFilters : this.state.forceFilter) {
const minHandlers = this.props.muted ? {} : {
moveUp: this.handleHotkeyMoveUp,
moveDown: this.handleHotkeyMoveDown,
@ -337,7 +348,11 @@ class Status extends ImmutablePureComponent {
return (
<HotKeys handlers={minHandlers}>
<div className='status__wrapper status__wrapper--filtered focusable' tabIndex='0' ref={this.handleRef}>
<FormattedMessage id='status.filtered' defaultMessage='Filtered' />
<FormattedMessage id='status.filtered' defaultMessage='Filtered' />: {matchedFilters.join(', ')}.
{' '}
<button className='status__wrapper--filtered__button' onClick={this.handleUnfilterClick}>
<FormattedMessage id='status.show_filter_reason' defaultMessage='Show anyway' />
</button>
</div>
</HotKeys>
);
@ -496,7 +511,7 @@ class Status extends ImmutablePureComponent {
{media}
<StatusActionBar scrollKey={scrollKey} status={status} account={account} {...other} />
<StatusActionBar scrollKey={scrollKey} status={status} account={account} onFilter={matchedFilters && this.handleFilterClick} {...other} />
</div>
</div>
</HotKeys>

17
app/javascript/mastodon/components/status_action_bar.js
View file

@ -38,6 +38,7 @@ const messages = defineMessages({
admin_account: { id: 'status.admin_account', defaultMessage: 'Open moderation interface for @{name}' },
admin_status: { id: 'status.admin_status', defaultMessage: 'Open this status in the moderation interface' },
copy: { id: 'status.copy', defaultMessage: 'Copy link to status' },
hide: { id: 'status.hide', defaultMessage: 'Hide toot' },
blockDomain: { id: 'account.block_domain', defaultMessage: 'Block domain {domain}' },
unblockDomain: { id: 'account.unblock_domain', defaultMessage: 'Unblock domain {domain}' },
unmute: { id: 'account.unmute', defaultMessage: 'Unmute @{name}' },
@ -76,6 +77,7 @@ class StatusActionBar extends ImmutablePureComponent {
onMuteConversation: PropTypes.func,
onPin: PropTypes.func,
onBookmark: PropTypes.func,
onFilter: PropTypes.func,
withDismiss: PropTypes.bool,
withCounters: PropTypes.bool,
scrollKey: PropTypes.string,
@ -207,6 +209,10 @@ class StatusActionBar extends ImmutablePureComponent {
this.props.onMuteConversation(this.props.status);
}
handleFilter = () => {
this.props.onFilter();
}
handleCopy = () => {
const url = this.props.status.get('url');
const textarea = document.createElement('textarea');
@ -226,6 +232,11 @@ class StatusActionBar extends ImmutablePureComponent {
}
}
handleFilterClick = () => {
this.props.onFilter();
}
render () {
const { status, relationship, intl, withDismiss, withCounters, scrollKey } = this.props;
@ -329,6 +340,10 @@ class StatusActionBar extends ImmutablePureComponent {
<IconButton className='status__action-bar-button' title={intl.formatMessage(messages.share)} icon='share-alt' onClick={this.handleShareClick} />
);
const filterButton = this.props.onFilter && (
<IconButton className='status__action-bar-button' title={intl.formatMessage(messages.hide)} icon='eye' onClick={this.handleFilterClick} />
);
return (
<div className='status__action-bar'>
<IconButton className='status__action-bar-button' title={replyTitle} icon={status.get('in_reply_to_account_id') === status.getIn(['account', 'id']) ? 'reply' : replyIcon} onClick={this.handleReplyClick} counter={status.get('replies_count')} obfuscateCount />
@ -337,6 +352,8 @@ class StatusActionBar extends ImmutablePureComponent {
{shareButton}
{filterButton}
<div className='status__action-bar-dropdown'>
<DropdownMenuContainer
scrollKey={scrollKey}

3
app/javascript/mastodon/features/ui/index.js
View file

@ -13,7 +13,6 @@ import { debounce } from 'lodash';
import { uploadCompose, resetCompose, changeComposeSpoilerness } from '../../actions/compose';
import { expandHomeTimeline } from '../../actions/timelines';
import { expandNotifications } from '../../actions/notifications';
import { fetchFilters } from '../../actions/filters';
import { fetchRules } from '../../actions/rules';
import { clearHeight } from '../../actions/height_cache';
import { focusApp, unfocusApp, changeLayout } from 'mastodon/actions/app';
@ -368,7 +367,7 @@ class UI extends React.PureComponent {
this.props.dispatch(fetchMarkers());
this.props.dispatch(expandHomeTimeline());
this.props.dispatch(expandNotifications());
setTimeout(() => this.props.dispatch(fetchFilters()), 500);
setTimeout(() => this.props.dispatch(fetchRules()), 3000);
this.hotkeys.__mousetrap__.stopCallback = (e, element) => {

34
app/javascript/mastodon/reducers/filters.js
View file

@ -1,10 +1,34 @@
import { FILTERS_FETCH_SUCCESS } from '../actions/filters';
import { List as ImmutableList, fromJS } from 'immutable';
import { FILTERS_IMPORT } from '../actions/importer';
import { Map as ImmutableMap, is, fromJS } from 'immutable';
export default function filters(state = ImmutableList(), action) {
const normalizeFilter = (state, filter) => {
const normalizedFilter = fromJS({
id: filter.id,
title: filter.title,
context: filter.context,
filter_action: filter.filter_action,
expires_at: filter.expires_at ? Date.parse(filter.expires_at) : null,
});
if (is(state.get(filter.id), normalizedFilter)) {
return state;
} else {
return state.set(filter.id, normalizedFilter);
}
};
const normalizeFilters = (state, filters) => {
filters.forEach(filter => {
state = normalizeFilter(state, filter);
});
return state;
};
export default function filters(state = ImmutableMap(), action) {
switch(action.type) {
case FILTERS_FETCH_SUCCESS:
return fromJS(action.filters);
case FILTERS_IMPORT:
return normalizeFilters(state, action.filters);
default:
return state;
}

47
app/javascript/mastodon/selectors/index.js
View file

@ -40,15 +40,15 @@ const toServerSideType = columnType => {
const escapeRegExp = string =>
string.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); // $& means the whole matched string
const regexFromFilters = filters => {
if (filters.size === 0) {
const regexFromKeywords = keywords => {
if (keywords.size === 0) {
return null;
}
return new RegExp(filters.map(filter => {
let expr = escapeRegExp(filter.get('phrase'));
return new RegExp(keywords.map(keyword_filter => {
let expr = escapeRegExp(keyword_filter.get('keyword'));
if (filter.get('whole_word')) {
if (keyword_filter.get('whole_word')) {
if (/^[\w]/.test(expr)) {
expr = `\\b${expr}`;
}
@ -62,27 +62,15 @@ const regexFromFilters = filters => {
}).join('|'), 'i');
};
// Memoize the filter regexps for each valid server contextType
const makeGetFiltersRegex = () => {
let memo = {};
return (state, { contextType }) => {
if (!contextType) return ImmutableList();
const getFilters = (state, { contextType }) => {
if (!contextType) return null;
const serverSideType = toServerSideType(contextType);
const filters = state.get('filters', ImmutableList()).filter(filter => filter.get('context').includes(serverSideType) && (filter.get('expires_at') === null || Date.parse(filter.get('expires_at')) > (new Date())));
const now = new Date();
if (!memo[serverSideType] || !is(memo[serverSideType].filters, filters)) {
const dropRegex = regexFromFilters(filters.filter(filter => filter.get('irreversible')));
const regex = regexFromFilters(filters);
memo[serverSideType] = { filters: filters, results: [dropRegex, regex] };
}
return memo[serverSideType].results;
};
return state.get('filters').filter((filter) => filter.get('context').includes(serverSideType) && (filter.get('expires_at') === null || filter.get('expires_at') > now));
};
export const getFiltersRegex = makeGetFiltersRegex();
export const makeGetStatus = () => {
return createSelector(
[
@ -90,10 +78,10 @@ export const makeGetStatus = () => {
(state, { id }) => state.getIn(['statuses', state.getIn(['statuses', id, 'reblog'])]),
(state, { id }) => state.getIn(['accounts', state.getIn(['statuses', id, 'account'])]),
(state, { id }) => state.getIn(['accounts', state.getIn(['statuses', state.getIn(['statuses', id, 'reblog']), 'account'])]),
getFiltersRegex,
getFilters,
],
(statusBase, statusReblog, accountBase, accountReblog, filtersRegex) => {
(statusBase, statusReblog, accountBase, accountReblog, filters) => {
if (!statusBase) {
return null;
}
@ -104,13 +92,16 @@ export const makeGetStatus = () => {
statusReblog = null;
}
const dropRegex = (accountReblog || accountBase).get('id') !== me && filtersRegex[0];
if (dropRegex && dropRegex.test(statusBase.get('reblog') ? statusReblog.get('search_index') : statusBase.get('search_index'))) {
let filtered = false;
if ((accountReblog || accountBase).get('id') !== me && filters) {
let filterResults = statusReblog?.get('filtered') || statusBase.get('filtered') || ImmutableList();
if (filterResults.some((result) => filters.getIn([result.get('filter'), 'filter_action']) === 'hide')) {
return null;
}
const regex = (accountReblog || accountBase).get('id') !== me && filtersRegex[1];
const filtered = regex && regex.test(statusBase.get('reblog') ? statusReblog.get('search_index') : statusBase.get('search_index'));
if (!filterResults.isEmpty()) {
filtered = filterResults.map(result => filters.getIn([result.get('filter'), 'title']));
}
}
return statusBase.withMutations(map => {
map.set('reblog', statusReblog);

1
app/javascript/packs/public.js
View file

@ -4,6 +4,7 @@ import loadPolyfills from '../mastodon/load_polyfills';
import ready from '../mastodon/ready';
import { start } from '../mastodon/common';
import loadKeyboardExtensions from '../mastodon/load_keyboard_extensions';
import 'cocoon-js-vanilla';
start();

33
app/javascript/styles/mastodon/admin.scss
View file

@ -915,7 +915,8 @@ a.name-tag,
text-align: center;
}
.applications-list__item {
.applications-list__item,
.filters-list__item {
padding: 15px 0;
background: $ui-base-color;
border: 1px solid lighten($ui-base-color, 4%);
@ -923,7 +924,8 @@ a.name-tag,
margin-top: 15px;
}
.announcements-list {
.announcements-list,
.filters-list {
border: 1px solid lighten($ui-base-color, 4%);
border-radius: 4px;
@ -976,6 +978,33 @@ a.name-tag,
}
}
.filters-list__item {
&__title {
display: flex;
justify-content: space-between;
margin-bottom: 0;
}
&__permissions {
margin-top: 0;
margin-bottom: 10px;
}
.expiration {
font-size: 13px;
}
&.expired {
.expiration {
color: lighten($error-red, 12%);
}
.permissions-list__item__icon {
color: $dark-text-color;
}
}
}
.dashboard__counters.admin-account-counters {
margin-top: 10px;
}

15
app/javascript/styles/mastodon/components.scss
View file

@ -959,6 +959,21 @@
width: 100%;
clear: both;
border-bottom: 1px solid lighten($ui-base-color, 8%);
&__button {
display: inline;
color: lighten($ui-highlight-color, 8%);
border: 0;
background: transparent;
padding: 0;
font-size: inherit;
line-height: inherit;
&:hover,
&:active {
text-decoration: underline;
}
}
}
.status__prepend-icon-wrapper {

31
app/javascript/styles/mastodon/forms.scss
View file

@ -1070,3 +1070,34 @@ code {
}
}
}
.keywords-table {
thead {
th {
white-space: nowrap;
}
th:first-child {
width: 100%;
}
}
tfoot {
td {
border: 0;
}
}
.input.string {
margin-bottom: 0;
}
.label_input__wrapper {
margin-top: 10px;
}
.table-action-link {
margin-top: 10px;
white-space: nowrap;
}
}

30
app/lib/feed_manager.rb
View file

@ -352,7 +352,6 @@ class FeedManager
def filter_from_home?(status, receiver_id, crutches)
return false if receiver_id == status.account_id
return true if status.reply? && (status.in_reply_to_id.nil? || status.in_reply_to_account_id.nil?)
return true if phrase_filtered?(status, receiver_id, :home)
check_for_blocks = crutches[:active_mentions][status.id] || []
check_for_blocks.concat([status.account_id])
@ -388,7 +387,6 @@ class FeedManager
# @return [Boolean]
def filter_from_mentions?(status, receiver_id)
return true if receiver_id == status.account_id
return true if phrase_filtered?(status, receiver_id, :notifications)
# This filter is called from NotifyService, but already after the sender of
# the notification has been checked for mute/block. Therefore, it's not
@ -418,34 +416,6 @@ class FeedManager
false
end
# Check if the status hits a phrase filter
# @param [Status] status
# @param [Integer] receiver_id
# @param [Symbol] context
# @return [Boolean]
def phrase_filtered?(status, receiver_id, context)
active_filters = Rails.cache.fetch("filters:#{receiver_id}") { CustomFilter.where(account_id: receiver_id).active_irreversible.to_a }.to_a
active_filters.select! { |filter| filter.context.include?(context.to_s) && !filter.expired? }
active_filters.map! do |filter|
if filter.whole_word
sb = /\A[[:word:]]/.match?(filter.phrase) ? '\b' : ''
eb = /[[:word:]]\z/.match?(filter.phrase) ? '\b' : ''
/(?mix:#{sb}#{Regexp.escape(filter.phrase)}#{eb})/
else
/#{Regexp.escape(filter.phrase)}/i
end
end
return false if active_filters.empty?
combined_regex = Regexp.union(active_filters)
combined_regex.match?(status.proper.searchable_text)
end
# Adds a status to an account's feed, returning true if a status was
# added, and false if it was not added to the feed. Note that this is
# an internal helper: callers must call trim or push updates if

13
app/models/concerns/account_interactions.rb
View file

@ -247,6 +247,19 @@ module AccountInteractions
account_pins.where(target_account: account).exists?
end
def status_matches_filters(status)
active_filters = CustomFilter.cached_filters_for(id)
filter_matches = active_filters.filter_map do |filter, rules|
next if rules[:keywords].blank?
match = rules[:keywords].match(status.proper.searchable_text)
FilterResultPresenter.new(filter: filter, keyword_matches: [match.to_s]) unless match.nil?
end
filter_matches
end
def followers_for_local_distribution
followers.local
.joins(:user)

77
app/models/custom_filter.rb
View file

@ -3,18 +3,22 @@
#
# Table name: custom_filters
#
# id :bigint(8) not null, primary key
# account_id :bigint(8)
# id :bigint not null, primary key
# account_id :bigint
# expires_at :datetime
# phrase :text default(""), not null
# context :string default([]), not null, is an Array
# whole_word :boolean default(TRUE), not null
# irreversible :boolean default(FALSE), not null
# created_at :datetime not null
# updated_at :datetime not null
# action :integer default(0), not null
#
class CustomFilter < ApplicationRecord
self.ignored_columns = %w(whole_word irreversible)
alias_attribute :title, :phrase
alias_attribute :filter_action, :action
VALID_CONTEXTS = %w(
home
notifications
@ -26,16 +30,20 @@ class CustomFilter < ApplicationRecord
include Expireable
include Redisable
enum action: [:warn, :hide], _suffix: :action
belongs_to :account
has_many :keywords, class_name: 'CustomFilterKeyword', foreign_key: :custom_filter_id, inverse_of: :custom_filter, dependent: :destroy
accepts_nested_attributes_for :keywords, reject_if: :all_blank, allow_destroy: true
validates :phrase, :context, presence: true
validates :title, :context, presence: true
validate :context_must_be_valid
validate :irreversible_must_be_within_context
scope :active_irreversible, -> { where(irreversible: true).where(Arel.sql('expires_at IS NULL OR expires_at > NOW()')) }
before_validation :clean_up_contexts
after_commit :remove_cache
before_save :prepare_cache_invalidation!
before_destroy :prepare_cache_invalidation!
after_commit :invalidate_cache!
def expires_in
return @expires_in if defined?(@expires_in)
@ -44,22 +52,55 @@ class CustomFilter < ApplicationRecord
[30.minutes, 1.hour, 6.hours, 12.hours, 1.day, 1.week].find { |expires_in| expires_in.from_now >= expires_at }
end
def irreversible=(value)
self.action = value ? :hide : :warn
end
def irreversible?
hide_action?
end
def self.cached_filters_for(account_id)
active_filters = Rails.cache.fetch("filters:v3:#{account_id}") do
scope = CustomFilterKeyword.includes(:custom_filter).where(custom_filter: { account_id: account_id }).where(Arel.sql('expires_at IS NULL OR expires_at > NOW()'))
scope.to_a.group_by(&:custom_filter).map do |filter, keywords|
keywords.map! do |keyword|
if keyword.whole_word
sb = /\A[[:word:]]/.match?(keyword.keyword) ? '\b' : ''
eb = /[[:word:]]\z/.match?(keyword.keyword) ? '\b' : ''
/(?mix:#{sb}#{Regexp.escape(keyword.keyword)}#{eb})/
else
/#{Regexp.escape(keyword.keyword)}/i
end
end
[filter, { keywords: Regexp.union(keywords) }]
end
end.to_a
active_filters.select { |custom_filter, _| !custom_filter.expired? }
end
def prepare_cache_invalidation!
@should_invalidate_cache = true
end
def invalidate_cache!
return unless @should_invalidate_cache
@should_invalidate_cache = false
Rails.cache.delete("filters:v3:#{account_id}")
redis.publish("timeline:#{account_id}", Oj.dump(event: :filters_changed))
redis.publish("timeline:system:#{account_id}", Oj.dump(event: :filters_changed))
end
private
def clean_up_contexts
self.context = Array(context).map(&:strip).filter_map(&:presence)
end
def remove_cache
Rails.cache.delete("filters:#{account_id}")
redis.publish("timeline:#{account_id}", Oj.dump(event: :filters_changed))
end
def context_must_be_valid
errors.add(:context, I18n.t('filters.errors.invalid_context')) if context.empty? || context.any? { |c| !VALID_CONTEXTS.include?(c) }
end
def irreversible_must_be_within_context
errors.add(:irreversible, I18n.t('filters.errors.invalid_irreversible')) if irreversible? && !context.include?('home') && !context.include?('notifications')
end
end

34
app/models/custom_filter_keyword.rb Normal file
View file

@ -0,0 +1,34 @@
# frozen_string_literal: true
# == Schema Information
#
# Table name: custom_filter_keywords
#
# id :bigint not null, primary key
# custom_filter_id :bigint not null
# keyword :text default(""), not null
# whole_word :boolean default(TRUE), not null
# created_at :datetime not null
# updated_at :datetime not null
#
class CustomFilterKeyword < ApplicationRecord
belongs_to :custom_filter
validates :keyword, presence: true
alias_attribute :phrase, :keyword
before_save :prepare_cache_invalidation!
before_destroy :prepare_cache_invalidation!
after_commit :invalidate_cache!
private
def prepare_cache_invalidation!
custom_filter.prepare_cache_invalidation!
end
def invalidate_cache!
custom_filter.invalidate_cache!
end
end

5
app/presenters/filter_result_presenter.rb Normal file
View file

@ -0,0 +1,5 @@
# frozen_string_literal: true
class FilterResultPresenter < ActiveModelSerializers::Model
attributes :filter, :keyword_matches
end

24
app/presenters/status_relationships_presenter.rb
View file

@ -2,7 +2,7 @@
class StatusRelationshipsPresenter
attr_reader :reblogs_map, :favourites_map, :mutes_map, :pins_map,
:bookmarks_map
:bookmarks_map, :filters_map
def initialize(statuses, current_account_id = nil, **options)
if current_account_id.nil?
@ -11,12 +11,14 @@ class StatusRelationshipsPresenter
@bookmarks_map = {}
@mutes_map = {}
@pins_map = {}
@filters_map = {}
else
statuses = statuses.compact
status_ids = statuses.flat_map { |s| [s.id, s.reblog_of_id] }.uniq.compact
conversation_ids = statuses.filter_map(&:conversation_id).uniq
pinnable_status_ids = statuses.map(&:proper).filter_map { |s| s.id if s.account_id == current_account_id && %w(public unlisted private).include?(s.visibility) }
@filters_map = build_filters_map(statuses, current_account_id).merge(options[:filters_map] || {})
@reblogs_map = Status.reblogs_map(status_ids, current_account_id).merge(options[:reblogs_map] || {})
@favourites_map = Status.favourites_map(status_ids, current_account_id).merge(options[:favourites_map] || {})
@bookmarks_map = Status.bookmarks_map(status_ids, current_account_id).merge(options[:bookmarks_map] || {})
@ -24,4 +26,24 @@ class StatusRelationshipsPresenter
@pins_map = Status.pins_map(pinnable_status_ids, current_account_id).merge(options[:pins_map] || {})
end
end
private
def build_filters_map(statuses, current_account_id)
active_filters = CustomFilter.cached_filters_for(current_account_id)
@filters_map = statuses.each_with_object({}) do |status, h|
filter_matches = active_filters.filter_map do |filter, rules|
next if rules[:keywords].blank?
match = rules[:keywords].match(status.proper.searchable_text)
FilterResultPresenter.new(filter: filter, keyword_matches: [match.to_s]) unless match.nil?
end
unless filter_matches.empty?
h[status.id] = filter_matches
h[status.reblog_of_id] = filter_matches if status.reblog?
end
end
end
end

9
app/serializers/rest/filter_keyword_serializer.rb Normal file
View file

@ -0,0 +1,9 @@
# frozen_string_literal: true
class REST::FilterKeywordSerializer < ActiveModel::Serializer
attributes :id, :keyword, :whole_word
def id
object.id.to_s
end
end

6
app/serializers/rest/filter_result_serializer.rb Normal file
View file

@ -0,0 +1,6 @@
# frozen_string_literal: true
class REST::FilterResultSerializer < ActiveModel::Serializer
belongs_to :filter, serializer: REST::FilterSerializer
has_many :keyword_matches
end

8
app/serializers/rest/filter_serializer.rb
View file

@ -1,10 +1,14 @@
# frozen_string_literal: true
class REST::FilterSerializer < ActiveModel::Serializer
attributes :id, :phrase, :context, :whole_word, :expires_at,
:irreversible
attributes :id, :title, :context, :expires_at, :filter_action
has_many :keywords, serializer: REST::FilterKeywordSerializer, if: :rules_requested?
def id
object.id.to_s
end
def rules_requested?
instance_options[:rules_requested]
end
end

9
app/serializers/rest/status_serializer.rb
View file

@ -13,6 +13,7 @@ class REST::StatusSerializer < ActiveModel::Serializer
attribute :muted, if: :current_user?
attribute :bookmarked, if: :current_user?
attribute :pinned, if: :pinnable?
has_many :filtered, serializer: REST::FilterResultSerializer, if: :current_user?
attribute :content, unless: :source_requested?
attribute :text, if: :source_requested?
@ -120,6 +121,14 @@ class REST::StatusSerializer < ActiveModel::Serializer
end
end
def filtered
if instance_options && instance_options[:relationships]
instance_options[:relationships].filters_map[object.id] || []
else
current_user.account.status_matches_filters(object)
end
end
def pinnable?
current_user? &&
current_user.account_id == object.account_id &&

26
app/serializers/rest/v1/filter_serializer.rb Normal file
View file

@ -0,0 +1,26 @@
# frozen_string_literal: true
class REST::V1::FilterSerializer < ActiveModel::Serializer
attributes :id, :phrase, :context, :whole_word, :expires_at,
:irreversible
delegate :context, :expires_at, to: :custom_filter
def id
object.id.to_s
end
def phrase
object.keyword
end
def irreversible
custom_filter.irreversible?
end
private
def custom_filter
object.custom_filter
end
end

16
app/views/filters/_fields.html.haml
View file

@ -1,16 +0,0 @@
.fields-row
.fields-row__column.fields-row__column-6.fields-group
= f.input :phrase, as: :string, wrapper: :with_label, hint: false
.fields-row__column.fields-row__column-6.fields-group
= f.input :expires_in, wrapper: :with_label, collection: [30.minutes, 1.hour, 6.hours, 12.hours, 1.day, 1.week].map(&:to_i), label_method: lambda { |i| I18n.t("invites.expires_in.#{i}") }, include_blank: I18n.t('invites.expires_in_prompt')
.fields-group
= f.input :context, wrapper: :with_block_label, collection: CustomFilter::VALID_CONTEXTS, as: :check_boxes, collection_wrapper_tag: 'ul', item_wrapper_tag: 'li', label_method: lambda { |context| I18n.t("filters.contexts.#{context}") }, include_blank: false
%hr.spacer/
.fields-group
= f.input :irreversible, wrapper: :with_label
.fields-group
= f.input :whole_word, wrapper: :with_label

32
app/views/filters/_filter.html.haml Normal file
View file

@ -0,0 +1,32 @@
.filters-list__item{ class: [filter.expired? && 'expired'] }
= link_to edit_filter_path(filter), class: 'filters-list__item__title' do
= filter.title
- if filter.expires?
.expiration{ title: t('filters.index.expires_on', date: l(filter.expires_at)) }
- if filter.expired?
= t('invites.expired')
- else
= t('filters.index.expires_in', distance: distance_of_time_in_words_to_now(filter.expires_at))
.filters-list__item__permissions
%ul.permissions-list
- unless filter.keywords.empty?
%li.permissions-list__item
.permissions-list__item__icon
= fa_icon('paragraph')
.permissions-list__item__text
.permissions-list__item__text__title
= t('filters.index.keywords', count: filter.keywords.size)
.permissions-list__item__text__type
- keywords = filter.keywords.map(&:keyword)
- keywords = keywords.take(5) + ['…'] if keywords.size > 5 # TODO
= keywords.join(', ')
.announcements-list__item__action-bar
.announcements-list__item__meta
= t('filters.index.contexts', contexts: filter.context.map { |context| I18n.t("filters.contexts.#{context}") }.join(', '))
%div
= table_link_to 'pencil', t('filters.edit.title'), edit_filter_path(filter)
= table_link_to 'times', t('filters.index.delete'), filter_path(filter), method: :delete, data: { confirm: t('admin.accounts.are_you_sure') }

33
app/views/filters/_filter_fields.html.haml Normal file
View file

@ -0,0 +1,33 @@
.fields-row
.fields-row__column.fields-row__column-6.fields-group
= f.input :title, as: :string, wrapper: :with_label, hint: false
.fields-row__column.fields-row__column-6.fields-group
= f.input :expires_in, wrapper: :with_label, collection: [30.minutes, 1.hour, 6.hours, 12.hours, 1.day, 1.week].map(&:to_i), label_method: lambda { |i| I18n.t("invites.expires_in.#{i}") }, include_blank: I18n.t('invites.expires_in_prompt')
.fields-group
= f.input :context, wrapper: :with_block_label, collection: CustomFilter::VALID_CONTEXTS, as: :check_boxes, collection_wrapper_tag: 'ul', item_wrapper_tag: 'li', label_method: lambda { |context| I18n.t("filters.contexts.#{context}") }, include_blank: false
%hr.spacer/
.fields-group
= f.input :filter_action, as: :radio_buttons, collection: %i(warn hide), include_blank: false, wrapper: :with_block_label, label_method: ->(action) { safe_join([t("simple_form.labels.filters.actions.#{action}"), content_tag(:span, t("simple_form.hints.filters.actions.#{action}"), class: 'hint')]) }, hint: t('simple_form.hints.filters.action'), required: true
%hr.spacer/
%h4= t('filters.edit.keywords')
.table-wrapper
%table.table.keywords-table
%thead
%tr
%th= t('simple_form.labels.defaults.phrase')
%th= t('simple_form.labels.defaults.whole_word')
%th
%tbody
= f.simple_fields_for :keywords do |keyword|
= render 'keyword_fields', f: keyword
%tfoot
%tr
%td{ colspan: 3}
= link_to_add_association f, :keywords, class: 'table-action-link', partial: 'keyword_fields', 'data-association-insertion-node': '.keywords-table tbody', 'data-association-insertion-method': 'append' do
= safe_join([fa_icon('plus'), t('filters.edit.add_keyword')])

8
app/views/filters/_keyword_fields.html.haml Normal file
View file

@ -0,0 +1,8 @@
%tr.nested-fields
%td= f.input :keyword, as: :string
%td
.label_input__wrapper= f.input_field :whole_word
%td
= f.hidden_field :id if f.object&.persisted? # Required so Rails doesn't put the field outside of the <tr/>
= link_to_remove_association(f, class: 'table-action-link') do
= safe_join([fa_icon('times'), t('filters.index.delete')])

2
app/views/filters/edit.html.haml
View file

@ -2,7 +2,7 @@
= t('filters.edit.title')
= simple_form_for @filter, url: filter_path(@filter), method: :put do |f|
= render 'fields', f: f
= render 'filter_fields', f: f
.actions
= f.button :button, t('generic.save_changes'), type: :submit

17
app/views/filters/index.html.haml
View file

@ -7,18 +7,5 @@
- if @filters.empty?
%div.muted-hint.center-text= t 'filters.index.empty'
- else
.table-wrapper
%table.table
%thead
%tr
%th= t('simple_form.labels.defaults.phrase')
%th= t('simple_form.labels.defaults.context')
%th
%tbody
- @filters.each do |filter|
%tr
%td= filter.phrase
%td= filter.context.map { |context| I18n.t("filters.contexts.#{context}") }.join(', ')
%td
= table_link_to 'pencil', t('filters.edit.title'), edit_filter_path(filter)
= table_link_to 'times', t('filters.index.delete'), filter_path(filter), method: :delete
.applications-list
= render partial: 'filter', collection: @filters

4
app/views/filters/new.html.haml
View file

@ -2,7 +2,7 @@
= t('filters.new.title')
= simple_form_for @filter, url: filters_path do |f|
= render 'fields', f: f
= render 'filter_fields', f: f
.actions
= f.button :button, t('filters.new.title'), type: :submit
= f.button :button, t('filters.new.save'), type: :submit

11
config/locales/en.yml
View file

@ -1124,15 +1124,24 @@ en:
public: Public timelines
thread: Conversations
edit:
add_keyword: Add keyword
keywords: Keywords
title: Edit filter
errors:
deprecated_api_multiple_keywords: These parameters cannot be changed from this application because they apply to more than one filter keyword. Use a more recent application or the web interface.
invalid_context: None or invalid context supplied
invalid_irreversible: Irreversible filtering only works with home or notifications context
index:
contexts: Filters in %{contexts}
delete: Delete
empty: You have no filters.
expires_in: Expires in %{distance}
expires_on: Expires on %{date}
keywords:
one: "%{count} keyword"
other: "%{count} keywords"
title: Filters
new:
save: Save new filter
title: Add new filter
footer:
developers: Developers

10
config/locales/simple_form.en.yml
View file

@ -68,6 +68,11 @@ en:
with_dns_records: An attempt to resolve the given domain's DNS records will be made and the results will also be blocked
featured_tag:
name: 'You might want to use one of these:'
filters:
action: Chose which action to perform when a post matches the filter
actions:
hide: Completely hide the filtered content, behaving as if it did not exist
warn: Hide the filtered content behind a warning mentioning the filter's title
form_challenge:
current_password: You are entering a secure area
imports:
@ -181,6 +186,7 @@ en:
setting_use_pending_items: Slow mode
severity: Severity
sign_in_token_attempt: Security code
title: Title
type: Import type
username: Username
username_or_email: Username or Email
@ -189,6 +195,10 @@ en:
with_dns_records: Include MX records and IPs of the domain
featured_tag:
name: Hashtag
filters:
actions:
hide: Hide completely
warn: Hide with a warning
interactions:
must_be_follower: Block notifications from non-followers
must_be_following: Block notifications from people you don't follow

9
config/routes.rb
View file

@ -451,10 +451,16 @@ Rails.application.routes.draw do
resources :bookmarks, only: [:index]
resources :reports, only: [:create]
resources :trends, only: [:index], controller: 'trends/tags'
resources :filters, only: [:index, :create, :show, :update, :destroy]
resources :filters, only: [:index, :create, :show, :update, :destroy] do
resources :keywords, only: [:index, :create], controller: 'filters/keywords'
end
resources :endorsements, only: [:index]
resources :markers, only: [:index, :create]
namespace :filters do
resources :keywords, only: [:show, :update, :destroy]
end
namespace :apps do
get :verify_credentials, to: 'credentials#show'
end
@ -589,6 +595,7 @@ Rails.application.routes.draw do
resources :media, only: [:create]
get '/search', to: 'search#index', as: :search
resources :suggestions, only: [:index]
resources :filters, only: [:index, :create, :show, :update, :destroy]
namespace :admin do
resources :accounts, only: [:index]

13
db/migrate/20220613110628_create_custom_filter_keywords.rb Normal file
View file

@ -0,0 +1,13 @@
# frozen_string_literal: true
class CreateCustomFilterKeywords < ActiveRecord::Migration[6.1]
def change
create_table :custom_filter_keywords do |t|
t.belongs_to :custom_filter, foreign_key: { on_delete: :cascade }, null: false
t.text :keyword, null: false, default: ''
t.boolean :whole_word, null: false, default: true
t.timestamps
end
end
end

34
db/migrate/20220613110711_migrate_custom_filters.rb Normal file
View file

@ -0,0 +1,34 @@
# frozen_string_literal: true
class MigrateCustomFilters < ActiveRecord::Migration[6.1]
def up
# Preserve IDs as much as possible to not confuse existing clients.
# As long as this migration is irreversible, we do not have to deal with conflicts.
safety_assured do
execute <<-SQL.squish
INSERT INTO custom_filter_keywords (id, custom_filter_id, keyword, whole_word, created_at, updated_at)
SELECT id, id, phrase, whole_word, created_at, updated_at
FROM custom_filters
SQL
end
end
def down
# Copy back changes from custom filters guaranteed to be from the old API
safety_assured do
execute <<-SQL.squish
UPDATE custom_filters
SET phrase = custom_filter_keywords.keyword, whole_word = custom_filter_keywords.whole_word
FROM custom_filter_keywords
WHERE custom_filters.id = custom_filter_keywords.id AND custom_filters.id = custom_filter_keywords.custom_filter_id
SQL
end
# Drop every keyword as we can't safely provide a 1:1 mapping
safety_assured do
execute <<-SQL.squish
TRUNCATE custom_filter_keywords RESTART IDENTITY
SQL
end
end
end

20
db/migrate/20220613110834_add_action_to_custom_filters.rb Normal file
View file

@ -0,0 +1,20 @@
# frozen_string_literal: true
require Rails.root.join('lib', 'mastodon', 'migration_helpers')
class AddActionToCustomFilters < ActiveRecord::Migration[6.1]
include Mastodon::MigrationHelpers
disable_ddl_transaction!
def up
safety_assured do
add_column_with_default :custom_filters, :action, :integer, allow_null: false, default: 0
execute 'UPDATE custom_filters SET action = 1 WHERE irreversible IS TRUE'
end
end
def down
execute 'UPDATE custom_filters SET irreversible = (action = 1)'
remove_column :custom_filters, :action
end
end

20
db/post_migrate/20220613110802_remove_whole_word_from_custom_filters.rb Normal file
View file

@ -0,0 +1,20 @@
# frozen_string_literal: true
require Rails.root.join('lib', 'mastodon', 'migration_helpers')
class RemoveWholeWordFromCustomFilters < ActiveRecord::Migration[6.1]
include Mastodon::MigrationHelpers
disable_ddl_transaction!
def up
safety_assured do
remove_column :custom_filters, :whole_word
end
end
def down
safety_assured do
add_column_with_default :custom_filters, :whole_word, :boolean, default: true, allow_null: false
end
end
end

20
db/post_migrate/20220613110903_remove_irreversible_from_custom_filters.rb Normal file
View file

@ -0,0 +1,20 @@
# frozen_string_literal: true
require Rails.root.join('lib', 'mastodon', 'migration_helpers')
class RemoveIrreversibleFromCustomFilters < ActiveRecord::Migration[6.1]
include Mastodon::MigrationHelpers
disable_ddl_transaction!
def up
safety_assured do
remove_column :custom_filters, :irreversible
end
end
def down
safety_assured do
add_column_with_default :custom_filters, :irreversible, :boolean, allow_null: false, default: false
end
end
end

15
db/schema.rb
View file

@ -10,7 +10,7 @@
#
# It's strongly recommended that you check this file into your version control system.
ActiveRecord::Schema.define(version: 2022_06_06_044941) do
ActiveRecord::Schema.define(version: 2022_06_13_110903) do
# These are extensions that must be enabled in order to support this database
enable_extension "plpgsql"
@ -339,15 +339,23 @@ ActiveRecord::Schema.define(version: 2022_06_06_044941) do
t.index ["shortcode", "domain"], name: "index_custom_emojis_on_shortcode_and_domain", unique: true
end
create_table "custom_filter_keywords", force: :cascade do |t|
t.bigint "custom_filter_id", null: false
t.text "keyword", default: "", null: false
t.boolean "whole_word", default: true, null: false
t.datetime "created_at", precision: 6, null: false
t.datetime "updated_at", precision: 6, null: false
t.index ["custom_filter_id"], name: "index_custom_filter_keywords_on_custom_filter_id"
end
create_table "custom_filters", force: :cascade do |t|
t.bigint "account_id"
t.datetime "expires_at"
t.text "phrase", default: "", null: false
t.string "context", default: [], null: false, array: true
t.boolean "irreversible", default: false, null: false
t.datetime "created_at", null: false
t.datetime "updated_at", null: false
t.boolean "whole_word", default: true, null: false
t.integer "action", default: 0, null: false
t.index ["account_id"], name: "index_custom_filters_on_account_id"
end
@ -1082,6 +1090,7 @@ ActiveRecord::Schema.define(version: 2022_06_06_044941) do
add_foreign_key "canonical_email_blocks", "accounts", column: "reference_account_id", on_delete: :cascade
add_foreign_key "conversation_mutes", "accounts", name: "fk_225b4212bb", on_delete: :cascade
add_foreign_key "conversation_mutes", "conversations", on_delete: :cascade
add_foreign_key "custom_filter_keywords", "custom_filters", on_delete: :cascade
add_foreign_key "custom_filters", "accounts", on_delete: :cascade
add_foreign_key "devices", "accounts", on_delete: :cascade
add_foreign_key "devices", "oauth_access_tokens", column: "access_token_id", on_delete: :cascade

18
lib/tasks/tests.rake
View file

@ -38,10 +38,26 @@ namespace :tests do
puts 'Instance actor does not have a private key'
exit(1)
end
unless Account.find_by(username: 'user', domain: nil).custom_filters.map { |filter| filter.keywords.pluck(:keyword) } == [['test'], ['take']]
puts 'CustomFilterKeyword records not created as expected'
exit(1)
end
end
desc 'Populate the database with test data for 2.4.3'
task populate_v2_4_3: :environment do # rubocop:disable Naming/VariableNumber
ActiveRecord::Base.connection.execute(<<~SQL)
INSERT INTO "custom_filters"
(id, account_id, phrase, context, whole_word, irreversible, created_at, updated_at)
VALUES
(1, 2, 'test', '{ "home", "public" }', true, true, now(), now()),
(2, 2, 'take', '{ "home" }', false, false, now(), now());
SQL
end
desc 'Populate the database with test data for 2.4.0'
task populate_v2_4: :environment do
task populate_v2_4: :environment do # rubocop:disable Naming/VariableNumber
ActiveRecord::Base.connection.execute(<<~SQL)
INSERT INTO "settings"
(id, thing_type, thing_id, var, value, created_at, updated_at)

2
package.json
View file

@ -45,6 +45,7 @@
"babel-plugin-transform-react-remove-prop-types": "^0.4.24",
"blurhash": "^1.1.5",
"classnames": "^2.3.1",
"cocoon-js-vanilla": "^1.2.0",
"color-blend": "^3.0.1",
"compression-webpack-plugin": "^6.1.1",
"cross-env": "^7.0.3",
@ -71,6 +72,7 @@
"intl-relativeformat": "^6.4.3",
"is-nan": "^1.3.2",
"js-yaml": "^4.1.0",
"jsdom": "^20.0.0",
"lodash": "^4.17.21",
"mark-loader": "^0.1.6",
"marky": "^1.2.4",

142
spec/controllers/api/v1/filters/keywords_controller_spec.rb Normal file
View file

@ -0,0 +1,142 @@
require 'rails_helper'
RSpec.describe Api::V1::Filters::KeywordsController, type: :controller do
render_views
let(:user) { Fabricate(:user) }
let(:token) { Fabricate(:accessible_access_token, resource_owner_id: user.id, scopes: scopes) }
let(:filter) { Fabricate(:custom_filter, account: user.account) }
let(:other_user) { Fabricate(:user) }
let(:other_filter) { Fabricate(:custom_filter, account: other_user.account) }
before do
allow(controller).to receive(:doorkeeper_token) { token }
end
describe 'GET #index' do
let(:scopes) { 'read:filters' }
let!(:keyword) { Fabricate(:custom_filter_keyword, custom_filter: filter) }
it 'returns http success' do
get :index, params: { filter_id: filter.id }
expect(response).to have_http_status(200)
end
context "when trying to access another's user filters" do
it 'returns http not found' do
get :index, params: { filter_id: other_filter.id }
expect(response).to have_http_status(404)
end
end
end
describe 'POST #create' do
let(:scopes) { 'write:filters' }
let(:filter_id) { filter.id }
before do
post :create, params: { filter_id: filter_id, keyword: 'magic', whole_word: false }
end
it 'returns http success' do
expect(response).to have_http_status(200)
end
it 'returns a keyword' do
json = body_as_json
expect(json[:keyword]).to eq 'magic'
expect(json[:whole_word]).to eq false
end
it 'creates a keyword' do
filter = user.account.custom_filters.first
expect(filter).to_not be_nil
expect(filter.keywords.pluck(:keyword)).to eq ['magic']
end
context "when trying to add to another another's user filters" do
let(:filter_id) { other_filter.id }
it 'returns http not found' do
expect(response).to have_http_status(404)
end
end
end
describe 'GET #show' do
let(:scopes) { 'read:filters' }
let(:keyword) { Fabricate(:custom_filter_keyword, keyword: 'foo', whole_word: false, custom_filter: filter) }
before do
get :show, params: { id: keyword.id }
end
it 'returns http success' do
expect(response).to have_http_status(200)
end
it 'returns expected data' do
json = body_as_json
expect(json[:keyword]).to eq 'foo'
expect(json[:whole_word]).to eq false
end
context "when trying to access another user's filter keyword" do
let(:keyword) { Fabricate(:custom_filter_keyword, custom_filter: other_filter) }
it 'returns http not found' do
expect(response).to have_http_status(404)
end
end
end
describe 'PUT #update' do
let(:scopes) { 'write:filters' }
let(:keyword) { Fabricate(:custom_filter_keyword, custom_filter: filter) }
before do
get :update, params: { id: keyword.id, keyword: 'updated' }
end
it 'returns http success' do
expect(response).to have_http_status(200)
end
it 'updates the keyword' do
expect(keyword.reload.keyword).to eq 'updated'
end
context "when trying to update another user's filter keyword" do
let(:keyword) { Fabricate(:custom_filter_keyword, custom_filter: other_filter) }
it 'returns http not found' do
expect(response).to have_http_status(404)
end
end
end
describe 'DELETE #destroy' do
let(:scopes) { 'write:filters' }
let(:keyword) { Fabricate(:custom_filter_keyword, custom_filter: filter) }
before do
delete :destroy, params: { id: keyword.id }
end
it 'returns http success' do
expect(response).to have_http_status(200)
end
it 'removes the filter' do
expect { keyword.reload }.to raise_error ActiveRecord::RecordNotFound
end
context "when trying to update another user's filter keyword" do
let(:keyword) { Fabricate(:custom_filter_keyword, custom_filter: other_filter) }
it 'returns http not found' do
expect(response).to have_http_status(404)
end
end
end
end

15
spec/controllers/api/v1/filters_controller_spec.rb
View file

@ -34,7 +34,7 @@ RSpec.describe Api::V1::FiltersController, type: :controller do
it 'creates a filter' do
filter = user.account.custom_filters.first
expect(filter).to_not be_nil
expect(filter.phrase).to eq 'magic'
expect(filter.keywords.pluck(:keyword)).to eq ['magic']
expect(filter.context).to eq %w(home)
expect(filter.irreversible?).to be true
expect(filter.expires_at).to be_nil
@ -44,9 +44,10 @@ RSpec.describe Api::V1::FiltersController, type: :controller do
describe 'GET #show' do
let(:scopes) { 'read:filters' }
let(:filter) { Fabricate(:custom_filter, account: user.account) }
let(:keyword) { Fabricate(:custom_filter_keyword, custom_filter: filter) }
it 'returns http success' do
get :show, params: { id: filter.id }
get :show, params: { id: keyword.id }
expect(response).to have_http_status(200)
end
end
@ -54,9 +55,10 @@ RSpec.describe Api::V1::FiltersController, type: :controller do
describe 'PUT #update' do
let(:scopes) { 'write:filters' }
let(:filter) { Fabricate(:custom_filter, account: user.account) }
let(:keyword) { Fabricate(:custom_filter_keyword, custom_filter: filter) }
before do
put :update, params: { id: filter.id, phrase: 'updated' }
put :update, params: { id: keyword.id, phrase: 'updated' }
end
it 'returns http success' do
@ -64,16 +66,17 @@ RSpec.describe Api::V1::FiltersController, type: :controller do
end
it 'updates the filter' do
expect(filter.reload.phrase).to eq 'updated'
expect(keyword.reload.phrase).to eq 'updated'
end
end
describe 'DELETE #destroy' do
let(:scopes) { 'write:filters' }
let(:filter) { Fabricate(:custom_filter, account: user.account) }
let(:keyword) { Fabricate(:custom_filter_keyword, custom_filter: filter) }
before do
delete :destroy, params: { id: filter.id }
delete :destroy, params: { id: keyword.id }
end
it 'returns http success' do
@ -81,7 +84,7 @@ RSpec.describe Api::V1::FiltersController, type: :controller do
end
it 'removes the filter' do
expect { filter.reload }.to raise_error ActiveRecord::RecordNotFound
expect { keyword.reload }.to raise_error ActiveRecord::RecordNotFound
end
end
end

52
spec/controllers/api/v1/statuses_controller_spec.rb
View file

@ -20,6 +20,58 @@ RSpec.describe Api::V1::StatusesController, type: :controller do
get :show, params: { id: status.id }
expect(response).to have_http_status(200)
end
context 'when post includes filtered terms' do
let(:status) { Fabricate(:status, text: 'this toot is about that banned word') }
before do
user.account.custom_filters.create!(phrase: 'filter1', context: %w(home), action: :hide, keywords_attributes: [{ keyword: 'banned' }, { keyword: 'irrelevant' }])
end
it 'returns http success' do
get :show, params: { id: status.id }
expect(response).to have_http_status(200)
end
it 'returns filter information' do
get :show, params: { id: status.id }
json = body_as_json
expect(json[:filtered][0]).to include({
filter: a_hash_including({
id: user.account.custom_filters.first.id.to_s,
title: 'filter1',
filter_action: 'hide',
}),
keyword_matches: ['banned'],
})
end
end
context 'when reblog includes filtered terms' do
let(:status) { Fabricate(:status, reblog: Fabricate(:status, text: 'this toot is about that banned word')) }
before do
user.account.custom_filters.create!(phrase: 'filter1', context: %w(home), action: :hide, keywords_attributes: [{ keyword: 'banned' }, { keyword: 'irrelevant' }])
end
it 'returns http success' do
get :show, params: { id: status.id }
expect(response).to have_http_status(200)
end
it 'returns filter information' do
get :show, params: { id: status.id }
json = body_as_json
expect(json[:reblog][:filtered][0]).to include({
filter: a_hash_including({
id: user.account.custom_filters.first.id.to_s,
title: 'filter1',
filter_action: 'hide',
}),
keyword_matches: ['banned'],
})
end
end
end
describe 'GET #context' do

121
spec/controllers/api/v2/filters_controller_spec.rb Normal file
View file

@ -0,0 +1,121 @@
require 'rails_helper'
RSpec.describe Api::V2::FiltersController, type: :controller do
render_views
let(:user) { Fabricate(:user) }
let(:token) { Fabricate(:accessible_access_token, resource_owner_id: user.id, scopes: scopes) }
before do
allow(controller).to receive(:doorkeeper_token) { token }
end
describe 'GET #index' do
let(:scopes) { 'read:filters' }
let!(:filter) { Fabricate(:custom_filter, account: user.account) }
it 'returns http success' do
get :index
expect(response).to have_http_status(200)
end
end
describe 'POST #create' do
let(:scopes) { 'write:filters' }
before do
post :create, params: { title: 'magic', context: %w(home), filter_action: 'hide', keywords_attributes: [keyword: 'magic'] }
end
it 'returns http success' do
expect(response).to have_http_status(200)
end
it 'returns a filter with keywords' do
json = body_as_json
expect(json[:title]).to eq 'magic'
expect(json[:filter_action]).to eq 'hide'
expect(json[:context]).to eq ['home']
expect(json[:keywords].map { |keyword| keyword.slice(:keyword, :whole_word) }).to eq [{ keyword: 'magic', whole_word: true }]
end
it 'creates a filter' do
filter = user.account.custom_filters.first
expect(filter).to_not be_nil
expect(filter.keywords.pluck(:keyword)).to eq ['magic']
expect(filter.context).to eq %w(home)
expect(filter.irreversible?).to be true
expect(filter.expires_at).to be_nil
end
end
describe 'GET #show' do
let(:scopes) { 'read:filters' }
let(:filter) { Fabricate(:custom_filter, account: user.account) }
it 'returns http success' do
get :show, params: { id: filter.id }
expect(response).to have_http_status(200)
end
end
describe 'PUT #update' do
let(:scopes) { 'write:filters' }
let!(:filter) { Fabricate(:custom_filter, account: user.account) }
let!(:keyword) { Fabricate(:custom_filter_keyword, custom_filter: filter) }
context 'updating filter parameters' do
before do
put :update, params: { id: filter.id, title: 'updated', context: %w(home public) }
end
it 'returns http success' do
expect(response).to have_http_status(200)
end
it 'updates the filter title' do
expect(filter.reload.title).to eq 'updated'
end
it 'updates the filter context' do
expect(filter.reload.context).to eq %w(home public)
end
end
context 'updating keywords in bulk' do
before do
allow(redis).to receive_messages(publish: nil)
put :update, params: { id: filter.id, keywords_attributes: [{ id: keyword.id, keyword: 'updated' }] }
end
it 'returns http success' do
expect(response).to have_http_status(200)
end
it 'updates the keyword' do
expect(keyword.reload.keyword).to eq 'updated'
end
it 'sends exactly one filters_changed event' do
expect(redis).to have_received(:publish).with("timeline:#{user.account.id}", Oj.dump(event: :filters_changed)).once
end
end
end
describe 'DELETE #destroy' do
let(:scopes) { 'write:filters' }
let(:filter) { Fabricate(:custom_filter, account: user.account) }
before do
delete :destroy, params: { id: filter.id }
end
it 'returns http success' do
expect(response).to have_http_status(200)
end
it 'removes the filter' do
expect { filter.reload }.to raise_error ActiveRecord::RecordNotFound
end
end
end

4
spec/fabricators/custom_filter_keyword_fabricator.rb Normal file
View file

@ -0,0 +1,4 @@
Fabricator(:custom_filter_keyword) do
custom_filter
keyword 'discourse'
end

32
spec/lib/feed_manager_spec.rb
View file

@ -127,38 +127,6 @@ RSpec.describe FeedManager do
reblog = Fabricate(:status, reblog: status, account: jeff)
expect(FeedManager.instance.filter?(:home, reblog, alice)).to be true
end
context 'for irreversibly muted phrases' do
it 'considers word boundaries when matching' do
alice.custom_filters.create!(phrase: 'bob', context: %w(home), irreversible: true)
alice.follow!(jeff)
status = Fabricate(:status, text: 'bobcats', account: jeff)
expect(FeedManager.instance.filter?(:home, status, alice)).to be_falsy
end
it 'returns true if phrase is contained' do
alice.custom_filters.create!(phrase: 'farts', context: %w(home public), irreversible: true)
alice.custom_filters.create!(phrase: 'pop tarts', context: %w(home), irreversible: true)
alice.follow!(jeff)
status = Fabricate(:status, text: 'i sure like POP TARts', account: jeff)
expect(FeedManager.instance.filter?(:home, status, alice)).to be true
end
it 'matches substrings if whole_word is false' do
alice.custom_filters.create!(phrase: 'take', context: %w(home), whole_word: false, irreversible: true)
alice.follow!(jeff)
status = Fabricate(:status, text: 'shiitake', account: jeff)
expect(FeedManager.instance.filter?(:home, status, alice)).to be true
end
it 'returns true if phrase is contained in a poll option' do
alice.custom_filters.create!(phrase: 'farts', context: %w(home public), irreversible: true)
alice.custom_filters.create!(phrase: 'pop tarts', context: %w(home), irreversible: true)
alice.follow!(jeff)
status = Fabricate(:status, text: 'what do you prefer', poll: Fabricate(:poll, options: %w(farts POP TARts)), account: jeff)
expect(FeedManager.instance.filter?(:home, status, alice)).to be true
end
end
end
context 'for mentions feed' do

4
spec/models/custom_filter_keyword_spec.rb Normal file
View file

@ -0,0 +1,4 @@
require 'rails_helper'
RSpec.describe CustomFilterKeyword, type: :model do
end

29
spec/presenters/status_relationships_presenter_spec.rb
View file

@ -5,7 +5,7 @@ require 'rails_helper'
RSpec.describe StatusRelationshipsPresenter do
describe '.initialize' do
before do
allow(Status).to receive(:reblogs_map).with(status_ids, current_account_id).and_return(default_map)
allow(Status).to receive(:reblogs_map).with(match_array(status_ids), current_account_id).and_return(default_map)
allow(Status).to receive(:favourites_map).with(status_ids, current_account_id).and_return(default_map)
allow(Status).to receive(:bookmarks_map).with(status_ids, current_account_id).and_return(default_map)
allow(Status).to receive(:mutes_map).with(anything, current_account_id).and_return(default_map)
@ -15,7 +15,7 @@ RSpec.describe StatusRelationshipsPresenter do
let(:presenter) { StatusRelationshipsPresenter.new(statuses, current_account_id, **options) }
let(:current_account_id) { Fabricate(:account).id }
let(:statuses) { [Fabricate(:status)] }
let(:status_ids) { statuses.map(&:id) }
let(:status_ids) { statuses.map(&:id) + statuses.map(&:reblog_of_id).compact }
let(:default_map) { { 1 => true } }
context 'options are not set' do
@ -69,5 +69,30 @@ RSpec.describe StatusRelationshipsPresenter do
expect(presenter.pins_map).to eq default_map.merge(options[:pins_map])
end
end
context 'when post includes filtered terms' do
let(:statuses) { [Fabricate(:status, text: 'this toot is about that banned word'), Fabricate(:status, reblog: Fabricate(:status, text: 'this toot is about an irrelevant word'))] }
let(:options) { {} }
before do
Account.find(current_account_id).custom_filters.create!(phrase: 'filter1', context: %w(home), action: :hide, keywords_attributes: [{ keyword: 'banned' }, { keyword: 'irrelevant' }])
end
it 'sets @filters_map to filter top-level status' do
matched_filters = presenter.filters_map[statuses[0].id]
expect(matched_filters.size).to eq 1
expect(matched_filters[0].filter.title).to eq 'filter1'
expect(matched_filters[0].keyword_matches).to eq ['banned']
end
it 'sets @filters_map to filter reblogged status' do
matched_filters = presenter.filters_map[statuses[1].reblog_of_id]
expect(matched_filters.size).to eq 1
expect(matched_filters[0].filter.title).to eq 'filter1'
expect(matched_filters[0].keyword_matches).to eq ['irrelevant']
end
end
end
end

90
streaming/index.js
View file

@ -12,6 +12,7 @@ const url = require('url');
const uuid = require('uuid');
const fs = require('fs');
const WebSocket = require('ws');
const { JSDOM } = require('jsdom');
const env = process.env.NODE_ENV || 'development';
const alwaysRequireAuth = process.env.LIMITED_FEDERATION_MODE === 'true' || process.env.WHITELIST_MODE === 'true' || process.env.AUTHORIZED_FETCH === 'true';
@ -503,6 +504,9 @@ const startWorker = async (workerId) => {
if (event === 'kill') {
log.verbose(req.requestId, `Closing connection for ${req.accountId} due to expired access token`);
eventHandlers.onKill();
} else if (event === 'filters_changed') {
log.verbose(req.requestId, `Invalidating filters cache for ${req.accountId}`);
req.cachedFilters = null;
}
};
};
@ -512,7 +516,8 @@ const startWorker = async (workerId) => {
* @param {any} res
*/
const subscribeHttpToSystemChannel = (req, res) => {
const systemChannelId = `timeline:access_token:${req.accessTokenId}`;
const accessTokenChannelId = `timeline:access_token:${req.accessTokenId}`;
const systemChannelId = `timeline:system:${req.accountId}`;
const listener = createSystemMessageListener(req, {
@ -523,9 +528,11 @@ const startWorker = async (workerId) => {
});
res.on('close', () => {
unsubscribe(`${redisPrefix}${accessTokenChannelId}`, listener);
unsubscribe(`${redisPrefix}${systemChannelId}`, listener);
});
subscribe(`${redisPrefix}${accessTokenChannelId}`, listener);
subscribe(`${redisPrefix}${systemChannelId}`, listener);
};
@ -674,17 +681,84 @@ const startWorker = async (workerId) => {
queries.push(client.query('SELECT 1 FROM account_domain_blocks WHERE account_id = $1 AND domain = $2', [req.accountId, accountDomain]));
}
if (!unpackedPayload.filter_results && !req.cachedFilters) {
queries.push(client.query('SELECT filter.id AS id, filter.phrase AS title, filter.context AS context, filter.expires_at AS expires_at, filter.action AS filter_action, keyword.keyword AS keyword, keyword.whole_word AS whole_word FROM custom_filter_keywords keyword JOIN custom_filters filter ON keyword.custom_filter_id = filter.id WHERE filter.account_id = $1 AND filter.expires_at IS NULL OR filter.expires_at > NOW()', [req.accountId]));
}
Promise.all(queries).then(values => {
done();
if (values[0].rows.length > 0 || (values.length > 1 && values[1].rows.length > 0)) {
if (values[0].rows.length > 0 || (accountDomain && values[1].rows.length > 0)) {
return;
}
if (!unpackedPayload.filter_results && !req.cachedFilters) {
const filterRows = values[accountDomain ? 2 : 1].rows;
req.cachedFilters = filterRows.reduce((cache, row) => {
if (cache[row.id]) {
cache[row.id].keywords.push([row.keyword, row.whole_word]);
} else {
cache[row.id] = {
keywords: [[row.keyword, row.whole_word]],
expires_at: row.expires_at,
repr: {
id: row.id,
title: row.title,
context: row.context,
expires_at: row.expires_at,
filter_action: row.filter_action,
},
};
}
return cache;
}, {});
Object.keys(req.cachedFilters).forEach((key) => {
req.cachedFilters[key].regexp = new RegExp(req.cachedFilters[key].keywords.map(([keyword, whole_word]) => {
let expr = keyword.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');;
if (whole_word) {
if (/^[\w]/.test(expr)) {
expr = `\\b${expr}`;
}
if (/[\w]$/.test(expr)) {
expr = `${expr}\\b`;
}
}
return expr;
}).join('|'), 'i');
});
}
// Check filters
if (req.cachedFilters && !unpackedPayload.filter_results) {
const status = unpackedPayload;
const searchContent = ([status.spoiler_text || '', status.content].concat((status.poll && status.poll.options) ? status.poll.options.map(option => option.title) : [])).concat(status.media_attachments.map(att => att.description)).join('\n\n').replace(/<br\s*\/?>/g, '\n').replace(/<\/p><p>/g, '\n\n');
const searchIndex = JSDOM.fragment(searchContent).textContent;
const now = new Date();
payload.filter_results = [];
Object.values(req.cachedFilters).forEach((cachedFilter) => {
if ((cachedFilter.expires_at === null || cachedFilter.expires_at > now)) {
const keyword_matches = searchIndex.match(cachedFilter.regexp);
if (keyword_matches) {
payload.filter_results.push({
filter: cachedFilter.repr,
keyword_matches,
});
}
}
});
}
transmit();
}).catch(err => {
done();
log.error(err);
done();
});
});
};
@ -1009,7 +1083,8 @@ const startWorker = async (workerId) => {
* @param {WebSocketSession} session
*/
const subscribeWebsocketToSystemChannel = ({ socket, request, subscriptions }) => {
const systemChannelId = `timeline:access_token:${request.accessTokenId}`;
const accessTokenChannelId = `timeline:access_token:${request.accessTokenId}`;
const systemChannelId = `timeline:system:${request.accountId}`;
const listener = createSystemMessageListener(request, {
@ -1019,8 +1094,15 @@ const startWorker = async (workerId) => {
});
subscribe(`${redisPrefix}${accessTokenChannelId}`, listener);
subscribe(`${redisPrefix}${systemChannelId}`, listener);
subscriptions[accessTokenChannelId] = {
listener,
stopHeartbeat: () => {
},
};
subscriptions[systemChannelId] = {
listener,
stopHeartbeat: () => {

67
yarn.lock
View file

@ -1486,7 +1486,7 @@
"@types/yargs" "^16.0.0"
chalk "^4.0.0"
"@jest/types@^28.1.0", "@jest/types@^28.1.1":
"@jest/types@^28.1.1":
version "28.1.1"
resolved "https://registry.yarnpkg.com/@jest/types/-/types-28.1.1.tgz#d059bbc80e6da6eda9f081f293299348bd78ee0b"
integrity sha512-vRXVqSg1VhDnB8bWcmvLzmg0Bt9CRKVgHPXqYwvWMX3TvAjeO+nRuK6+VdTKCtWOvYlmkF/HqNAL/z+N3B53Kw==
@ -2153,7 +2153,7 @@ acorn@^8.0.4:
resolved "https://registry.yarnpkg.com/acorn/-/acorn-8.3.0.tgz#1193f9b96c4e8232f00b11a9edff81b2c8b98b88"
integrity sha512-tqPKHZ5CaBJw0Xmy0ZZvLs1qTV+BNFSyvn77ASXkpBNfIRk8ev26fKrD9iLGwGA9zedPao52GSHzq8lyZG0NUw==
acorn@^8.5.0:
acorn@^8.5.0, acorn@^8.7.1:
version "8.7.1"
resolved "https://registry.yarnpkg.com/acorn/-/acorn-8.7.1.tgz#0197122c843d1bf6d0a5e83220a788f278f63c30"
integrity sha512-Xx54uLJQZ19lKygFXOWsscKUbsBZW0CPykPhVQdhIeIwrbPmJzqeASDInc8nKBnp/JT6igTs82qPXz069H8I/A==
@ -3304,6 +3304,11 @@ coa@^2.0.2:
chalk "^2.4.1"
q "^1.1.2"
cocoon-js-vanilla@^1.2.0:
version "1.2.0"
resolved "https://registry.yarnpkg.com/cocoon-js-vanilla/-/cocoon-js-vanilla-1.2.0.tgz#595348499d315d3b5828dd77a20974756cf59321"
integrity sha512-qLomIVL0Krfc983WLgaYPPktMjMtBN+F/CV15NPVDc9U9BCe2OL5WyAIYkPrVhDRphoYBmHCdIlZkq+vSBI4xg==
collect-v8-coverage@^1.0.0:
version "1.0.1"
resolved "https://registry.yarnpkg.com/collect-v8-coverage/-/collect-v8-coverage-1.0.1.tgz#cc2c8e94fc18bbdffe64d6534570c8a673b27f59"
@ -3892,7 +3897,7 @@ damerau-levenshtein@^1.0.7:
resolved "https://registry.yarnpkg.com/damerau-levenshtein/-/damerau-levenshtein-1.0.7.tgz#64368003512a1a6992593741a09a9d31a836f55d"
integrity sha512-VvdQIPGdWP0SqFXghj79Wf/5LArmreyMsGLa6FG6iC4t3j7j5s71TrwWmT/4akbDQIqjfACkLZmjXhA7g2oUZw==
data-urls@^3.0.1:
data-urls@^3.0.1, data-urls@^3.0.2:
version "3.0.2"
resolved "https://registry.yarnpkg.com/data-urls/-/data-urls-3.0.2.tgz#9cf24a477ae22bcef5cd5f6f0bfbc1d2d3be9143"
integrity sha512-Jy/tj3ldjZJo63sVAvg6LHt2mHvl4V6AgRAmNDtLdm7faqtsx+aJG42rsyCo9JCoRVKwPFzKlIPx3DIibwSIaQ==
@ -4338,6 +4343,11 @@ entities@^2.0.0:
resolved "https://registry.yarnpkg.com/entities/-/entities-2.0.3.tgz#5c487e5742ab93c15abb5da22759b8590ec03b7f"
integrity sha512-MyoZ0jgnLvB2X3Lg5HqpFmn1kybDiIfEQmKzTb5apr51Rb+T3KdmMiqa70T+bhGnyv7bQ6WMj2QMHpGMmlrUYQ==
entities@^4.3.0:
version "4.3.0"
resolved "https://registry.yarnpkg.com/entities/-/entities-4.3.0.tgz#62915f08d67353bb4eb67e3d62641a4059aec656"
integrity sha512-/iP1rZrSEJ0DTlPiX+jbzlA3eVkY/e8L8SozroF395fIqE3TYF/Nz7YOMAawta+vLmyJ/hkGNNPcSbMADCCXbg==
errno@^0.1.3, errno@~0.1.7:
version "0.1.7"
resolved "https://registry.yarnpkg.com/errno/-/errno-0.1.7.tgz#4684d71779ad39af177e3f007996f7c67c852618"
@ -5700,7 +5710,7 @@ https-browserify@^1.0.0:
resolved "https://registry.yarnpkg.com/https-browserify/-/https-browserify-1.0.0.tgz#ec06c10e0a34c0f2faf199f7fd7fc78fffd03c73"
integrity sha1-7AbBDgo0wPL68Zn3/X/Hj//QPHM=
https-proxy-agent@^5.0.0:
https-proxy-agent@^5.0.0, https-proxy-agent@^5.0.1:
version "5.0.1"
resolved "https://registry.yarnpkg.com/https-proxy-agent/-/https-proxy-agent-5.0.1.tgz#c59ef224a04fe8b754f3db0063a25ea30d0005d6"
integrity sha512-dFcAjpTQFgoLMzC2VwU+C/CbS7uRL0lWmxDITmqm7C+7F0Odmj6s9l6alZc6AELXhrnggM2CeWSXHGOdX2YtwA==
@ -6728,7 +6738,7 @@ jest-snapshot@^28.1.1:
pretty-format "^28.1.1"
semver "^7.3.5"
jest-util@^28.1.0, jest-util@^28.1.1:
jest-util@^28.1.1:
version "28.1.1"
resolved "https://registry.yarnpkg.com/jest-util/-/jest-util-28.1.1.tgz#ff39e436a1aca397c0ab998db5a51ae2b7080d05"
integrity sha512-FktOu7ca1DZSyhPAxgxB6hfh2+9zMoJ7aEQA759Z6p45NuO8mWcqujH+UdHlCm/V6JTWwDztM2ITCzU1ijJAfw==
@ -6852,6 +6862,39 @@ jsdom@^19.0.0:
ws "^8.2.3"
xml-name-validator "^4.0.0"
jsdom@^20.0.0:
version "20.0.0"
resolved "https://registry.yarnpkg.com/jsdom/-/jsdom-20.0.0.tgz#882825ac9cc5e5bbee704ba16143e1fa78361ebf"
integrity sha512-x4a6CKCgx00uCmP+QakBDFXwjAJ69IkkIWHmtmjd3wvXPcdOS44hfX2vqkOQrVrq8l9DhNNADZRXaCEWvgXtVA==
dependencies:
abab "^2.0.6"
acorn "^8.7.1"
acorn-globals "^6.0.0"
cssom "^0.5.0"
cssstyle "^2.3.0"
data-urls "^3.0.2"
decimal.js "^10.3.1"
domexception "^4.0.0"
escodegen "^2.0.0"
form-data "^4.0.0"
html-encoding-sniffer "^3.0.0"
http-proxy-agent "^5.0.0"
https-proxy-agent "^5.0.1"
is-potential-custom-element-name "^1.0.1"
nwsapi "^2.2.0"
parse5 "^7.0.0"
saxes "^6.0.0"
symbol-tree "^3.2.4"
tough-cookie "^4.0.0"
w3c-hr-time "^1.0.2"
w3c-xmlserializer "^3.0.0"
webidl-conversions "^7.0.0"
whatwg-encoding "^2.0.0"
whatwg-mimetype "^3.0.0"
whatwg-url "^11.0.0"
ws "^8.8.0"
xml-name-validator "^4.0.0"
jsesc@^2.5.1:
version "2.5.2"
resolved "https://registry.yarnpkg.com/jsesc/-/jsesc-2.5.2.tgz#80564d2e483dacf6e8ef209650a67df3f0c283a4"
@ -8141,6 +8184,13 @@ parse5@6.0.1:
resolved "https://registry.yarnpkg.com/parse5/-/parse5-6.0.1.tgz#e1a1c085c569b3dc08321184f19a39cc27f7c30b"
integrity sha512-Ofn/CTFzRGTTxwpNEs9PP93gXShHcTq255nzRYSKe8AkVpZY7e1fpmTfOyoIvjP5HG7Z2ZM7VS9PPhQGW2pOpw==
parse5@^7.0.0:
version "7.0.0"
resolved "https://registry.yarnpkg.com/parse5/-/parse5-7.0.0.tgz#51f74a5257f5fcc536389e8c2d0b3802e1bfa91a"
integrity sha512-y/t8IXSPWTuRZqXc0ajH/UwDj4mnqLEbSttNbThcFhGrZuOyoyvNBO85PBp2jQa55wY9d07PBNjsK8ZP3K5U6g==
dependencies:
entities "^4.3.0"
parseurl@~1.3.2, parseurl@~1.3.3:
version "1.3.3"
resolved "https://registry.yarnpkg.com/parseurl/-/parseurl-1.3.3.tgz#9da19e7bee8d12dff0513ed5b76957793bc2e8d4"
@ -9771,6 +9821,13 @@ saxes@^5.0.1:
dependencies:
xmlchars "^2.2.0"
saxes@^6.0.0:
version "6.0.0"
resolved "https://registry.yarnpkg.com/saxes/-/saxes-6.0.0.tgz#fe5b4a4768df4f14a201b1ba6a65c1f3d9988cc5"
integrity sha512-xAg7SOnEhrm5zI3puOOKyy1OMcMlIJZYNJY7xLBwSze0UjhPLnWfj2GF2EpT0jmzaJKIWKHLsaSSajf35bcYnA==
dependencies:
xmlchars "^2.2.0"
scheduler@^0.19.1:
version "0.19.1"
resolved "https://registry.yarnpkg.com/scheduler/-/scheduler-0.19.1.tgz#4f3e2ed2c1a7d65681f4c854fa8c5a1ccb40f196"