Typification of SearXNG
=======================
This patch introduces the typing of the results. The why and how is described
in the documentation, please generate the documentation ..
$ make docs.clean docs.live
and read the following articles in the "Developer documentation":
- result types --> http://0.0.0.0:8000/dev/result_types/index.html
The result types are available from the `searx.result_types` module. The
following have been implemented so far:
- base result type: `searx.result_type.Result`
--> http://0.0.0.0:8000/dev/result_types/base_result.html
- answer results
--> http://0.0.0.0:8000/dev/result_types/answer.html
including the type for translations (inspired by #3925). For all other
types (which still need to be set up in subsequent PRs), template documentation
has been created for the transition period.
Doc of the fields used in Templates
===================================
The template documentation is the basis for the typing and is the first complete
documentation of the results (needed for engine development). It is the
"working paper" (the plan) with which further typifications can be implemented
in subsequent PRs.
- https://github.com/searxng/searxng/issues/357
Answer Templates
================
With the new (sub) types for `Answer`, the templates for the answers have also
been revised, `Translation` are now displayed with collapsible entries (inspired
by #3925).
!en-de dog
Plugins & Answerer
==================
The implementation for `Plugin` and `Answer` has been revised, see
documentation:
- Plugin: http://0.0.0.0:8000/dev/plugins/index.html
- Answerer: http://0.0.0.0:8000/dev/answerers/index.html
With `AnswerStorage` and `AnswerStorage` to manage those items (in follow up
PRs, `ArticleStorage`, `InfoStorage` and .. will be implemented)
Autocomplete
============
The autocompletion had a bug where the results from `Answer` had not been shown
in the past. To test activate autocompletion and try search terms for which we
have answerers
- statistics: type `min 1 2 3` .. in the completion list you should find an
entry like `[de] min(1, 2, 3) = 1`
- random: type `random uuid` .. in the completion list, the first item is a
random UUID
Extended Types
==============
SearXNG extends e.g. the request and response types of flask and httpx, a module
has been set up for type extensions:
- Extended Types
--> http://0.0.0.0:8000/dev/extended_types.html
Unit-Tests
==========
The unit tests have been completely revised. In the previous implementation,
the runtime (the global variables such as `searx.settings`) was not initialized
before each test, so the runtime environment with which a test ran was always
determined by the tests that ran before it. This was also the reason why we
sometimes had to observe non-deterministic errors in the tests in the past:
- https://github.com/searxng/searxng/issues/2988 is one example for the Runtime
issues, with non-deterministic behavior ..
- https://github.com/searxng/searxng/pull/3650
- https://github.com/searxng/searxng/pull/3654
- https://github.com/searxng/searxng/pull/3642#issuecomment-2226884469
- https://github.com/searxng/searxng/pull/3746#issuecomment-2300965005
Why msgspec.Struct
==================
We have already discussed typing based on e.g. `TypeDict` or `dataclass` in the past:
- https://github.com/searxng/searxng/pull/1562/files
- https://gist.github.com/dalf/972eb05e7a9bee161487132a7de244d2
- https://github.com/searxng/searxng/pull/1412/files
- https://github.com/searxng/searxng/pull/1356
In my opinion, TypeDict is unsuitable because the objects are still dictionaries
and not instances of classes / the `dataclass` are classes but ...
The `msgspec.Struct` combine the advantages of typing, runtime behaviour and
also offer the option of (fast) serializing (incl. type check) the objects.
Currently not possible but conceivable with `msgspec`: Outsourcing the engines
into separate processes, what possibilities this opens up in the future is left
to the imagination!
Internally, we have already defined that it is desirable to decouple the
development of the engines from the development of the SearXNG core / The
serialization of the `Result` objects is a prerequisite for this.
HINT: The threads listed above were the template for this PR, even though the
implementation here is based on msgspec. They should also be an inspiration for
the following PRs of typification, as the models and implementations can provide
a good direction.
Why just one commit?
====================
I tried to create several (thematically separated) commits, but gave up at some
point ... there are too many things to tackle at once / The comprehensibility of
the commits would not be improved by a thematic separation. On the contrary, we
would have to make multiple changes at the same places and the goal of a change
would be vaguely recognizable in the fog of the commits.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
To avoid unnecessary changes to the file, the list should be sorted before it is
written to the file.
You can test it by calling multiple times::
make data.locales
and searx/data/locales.json should be unchanged.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
All the environments defined in ./utils/brand.env are generated on the fly, so
there is no longer a need to define the brand environment in this file and all
the workflows to handle this file.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
BTW force modularization of the ./mange script into sub modules:
- utils/lib_sxng_data.sh
- utils/lib_sxng_node.sh
- utils/lib_sxng_static.sh
- utils/lib_sxng_test.sh
- utils/lib_sxng_themes.sh
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Since ./utils/searxng.sh is implemented, the old installation procedures from
filtron, morty and searx can be removed.
For users who want to upgrade, the procedures for removing old installations
have still been retained.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Since we currently have many type checking errors,
we for now only test with typeCheckingMode: off
which makes pyright only check files that contain a comment:
# pyright: basic
to enable basic type checking, or
# pyright: strict
to enable strict type checking.
This patch implements the command and Makefile target::
./manage test.pyright
make test.pyright
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Environment variable to get live builds while modifying CSS & JS of a theme::
LIVE_THEME=simple make run
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Add the 'SearXNG JavaScript Style Guide' as one quality gate in the build chain
of the simple theme::
make themes.simple
make themes.simple.test
In the github CI the `themes.all` target enforce a `themes.simple`.
BTW: Remove 'jshint' left overs from 0ee316f3d
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Functions implemented to run *Weblate* workflows should use prefix 'weblate.':
- babel.setup.translations.worktree --> weblate.translations.worktree
Create git worktree ${TRANSLATIONS_WORKTREE} and checkout branch
'translations' from Weblate's counterpart (weblate) of the SearXNG
(origin)::
remote weblate https://weblate.bubu1.eu/git/searxng/searxng/
- babel.weblate.to.translations --> weblate.to.translations
Update 'translations' branch of SearXNG (origin) with last additions from
Weblate.
- babel.translations.to.master --> weblate.translations.commit
Update 'translations' branch of SearXNG (origin) with last additions from
Weblate. Copy the changes to the master branch, compile translations and
create a commit in the local branch (master)
- babel.master.to.translations --> weblate.push.translations
Push *translation changes* from SearXNG (origin) to Weblate's
counterpart (weblate).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In ./manage implement babel.*:
- extract : extract messages from source files and generate POT file
- update : update existing message catalogs from POT file
- compile : compile translation catalogs into binary MO files
Replace searx_extra/update/update_translations.sh by command:
- ci.babel.update
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
**new** utils/lib_install.sh:
Used to initialize installation procedures
- Modified source_dot_config function that
- loads .config.sh from an existing installation (at SEARX_SRC).
- initialize **SEARX_SRC_INIT_FILES**
- functions like:
- install_log_searx_instance()
- install_searx_get_state()
**modified** utils/searx.sh
- obsolete environment SEARX_INSTANCE_NAME has been replaced
by string 'SearXNG'.
**modified** utils/filtron.sh, utils/morty.sh, utils/searx.sh
- source utils/lib_install.sh
- normalize logging of environment variables using new function
install_log_searx_instance()
**modified** utils/lib.sh
- fix marginal typos
**Installation scripts**
The utils/lib_install.sh is sourced by the installations scripts:
- utils/searx.sh
- utils/morty.sh
- utils/filtron.sh
If '${SEARX_SRC}/.config.sh' exists, the modified source_dot_config() function
loads this configuration (instead of './.config.sh').
**SEARX_SRC_INIT_FILES**
Array of file names to sync into a installation at $SEARX_SRC. The file names
are relative to the $REPO_ROOT. Set by function init_SEARX_SRC_INIT_FILES().
Most often theses are files like:
- .config.sh
- searx/settings.yml
- utils/brand.env
- ...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The functions:
- static.build.commit
- static.build.commit.drop
- static.build.restore
are imported into the ./manage script. To avoid name collisions some variables
and fucntions has been renamed by adding the prefix *static_*.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
- add script to shellcheck,
- fix error messages from shellcheck and
- moved global variables to local variables (lower case)
No functional change!
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The idea is to avoid conflict when there are differents branches with changes are made on the static files.
A solution is to ask the administrators to build the files from the sources, but it requires to install the npm dependencies.
So the solution in this commit keep the sources and the build files in the same git repository.
In one branch, the modification of the source (*.less, *.js) are commited without the built files.
The built files are commited in a uniq commit, with a commit message "Static build"
In case of merge or rebase, this commit can be dropped.
New make targets:
* static.build.commit.drop:
drop the last "Static build".
The command checks that there are only build files in the commit.
* static.build.commit :
call "make static.build.commit.drop"
call "make themes.all"
commit the files
* static.git.restore.staged:
git restore --staged <build files>
* static.git.restore:
git restore <build files>
Related to https://github.com/searxng/searxng/issues/137
Usage::
make test.yamllint
./manage test.yamllint
test.yamllint is also added to the `test` makefile target.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Add script docker-entrypoint.sh to shellcheck and try to _simplify_ and
_normalize_ some parts:
- fix issues reported by shellcheck
- don't mix tab and space indent
- command 'help' replaced by '-h': ./dockerfiles/docker-entrypoint.sh -h
- replace printf in help() by 'cat <<EOF'
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
make docker.buildx : build and push multiarch build.
(it can't be only build)
use buildx with the --cache-from and --cache-to options to cache the layers
(only the last built is cached)
Renamed 'test.sh' to 'test.shell' seems to solve this issue on my ubu204::
$ make test.sh
...
TEST robot
...
INSTALL gecko.driver
...
Running 7 tests
cat test.sh >test
cat: test.sh: Datei oder Verzeichnis nicht gefunden
make: *** [<eingebaut>: test] Fehler 1
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Springer Nature is a global publisher dedicated to providing service to research
community [1] with official API [2].
To test this PR, first get your API key following this page:
https://dev.springernature.com/signup
In searx/engines/springer.py at line 24, add this API key. I left my own key,
commented out in the line aboce. Feel free to use it, if needed.
[1] https://www.springernature.com/
[2] https://dev.springernature.com/
