close
Skip to content

Replace Marshal by Granular_marshal in ocaml-index#1889

Merged
voodoos merged 8 commits into
ocaml:mainfrom
voodoos:granular-marshal
Feb 5, 2025
Merged

Replace Marshal by Granular_marshal in ocaml-index#1889
voodoos merged 8 commits into
ocaml:mainfrom
voodoos:granular-marshal

Conversation

@voodoos
Copy link
Copy Markdown
Collaborator

@voodoos voodoos commented Jan 17, 2025

Initial PR presentation by @Lucccyo:

The current implementation of ocaml-index uses Marshal to store on the disk the data.
Searching for occurrences on massive projects is time-consuming because the search loads all the data structures from the disk to perform the search.

This Pull Request aims to replace Marshal with a granular version to make the ocaml-index more efficient in reading.
It comes with two granular implementations of the data structures set and map, based on the Stdlib implementation.
During a search operation, the program lazily loads only the required part of the ocaml-index.
It works because the heavy nodes of the granular_map and granular_set have link indirections,
introducing serialization boundaries, which allows Marshal to delay the deserialization of their children.

voodoos added a commit to voodoos/merlin that referenced this pull request Jan 21, 2025
voodoos added a commit to voodoos/merlin that referenced this pull request Jan 22, 2025
voodoos added a commit to voodoos/merlin that referenced this pull request Jan 22, 2025
voodoos added a commit to voodoos/merlin that referenced this pull request Jan 22, 2025
voodoos added a commit to voodoos/merlin that referenced this pull request Jan 22, 2025
@voodoos
Copy link
Copy Markdown
Collaborator Author

voodoos commented Feb 5, 2025

On my own testing the indexing time is a slight (5%) increase of the indexing time and no increase in the file size. Fetches are now close to instantaneous and Merlin's memory usage vastly reduced since the index are not fully loaded into memory anymore.

Indexing one large library in Dune:

Benchmark 1: Main
  Time (mean ± σ):      1.116 s ±  0.011 s    [User: 0.941 s, System: 0.159 s]
  Range (min … max):    1.100 s …  1.132 s    10 runs

Benchmark 1: With the new granular-marshall:
  Time (mean ± σ):      1.177 s ±  0.011 s    [User: 1.004 s, System: 0.158 s]
  Range (min … max):    1.162 s …  1.195 s    10 runs

Impact on the larger Dune parallel build appear to be much lower: 1.6% increase.

@liam923 I am going to merge these improvements that make sense for most users, but we should also evaluate the impact of this PR on "larger" codebases.

@voodoos voodoos merged commit 102eee4 into ocaml:main Feb 5, 2025
voodoos added a commit to voodoos/merlin-jst that referenced this pull request Apr 9, 2025
voodoos added a commit to voodoos/merlin-jst that referenced this pull request Apr 10, 2025
voodoos added a commit to voodoos/merlin-jst that referenced this pull request Apr 23, 2025
voodoos added a commit to voodoos/opam-repository that referenced this pull request Jun 24, 2025
CHANGES:

Tue Jun 24 16:10:42 CEST 2025

  + merlin library
    - Expose utilities to manipulate typed-holes in `Merlin_analysis.Typed_hole`
      (ocaml/merlin#1888)
    - `locate` can now disambiguate between files with identical names and contents
      (ocaml/merlin#1882)
    - `occurrences` now reports stale files (ocaml/merlin#1885)
    - `inlay-hints` fix inlay hints on function parameters (ocaml/merlin#1923)
    - Fix issues with ident validation and Lid comparison for occurrences (ocaml/merlin#1924)
    - Handle class type in outline (ocaml/merlin#1932)
    - Handle locally defined value in outline (ocaml/merlin#1936)
    - Fix a typer issue triggering assertions in the short-paths graph (ocaml/merlin#1935,
      fixes ocaml/merlin#1913)
    - Downstreamed a typer fix from 5.3.X that would trigger assertions linked
      to scopes bit masks when backtracking the typer cache (ocaml/merlin#1935)
    - Add a new selection field to outline results that contains the location of
      the symbol itself. (ocaml/merlin#1942)
    - Fix destruct hanging when printing patterns with (::). (ocaml/merlin#1944, fixes
      ocaml/ocaml-lsp#1489)
    - Reproduce and fix a handful of jump-to-definition (locate) issues  (ocaml/merlin#1930,
      fixes ocaml/merlin#1580 and ocaml/merlin#1588, workaround for ocaml/merlin#1934)
  + ocaml-index
    - Improve the granularity of index reading by segmenting the marshalization
      of the involved data-structures. (ocaml/merlin#1889)
  + test suite
    - Add a test case illustrating wrong open order proposed in issue ocaml/merlin#1900. (ocaml/merlin#1901)
voodoos added a commit to voodoos/merlin-jst that referenced this pull request Jun 26, 2025
voodoos added a commit to voodoos/merlin-jst that referenced this pull request Jun 27, 2025
voodoos added a commit to voodoos/merlin-jst that referenced this pull request Jun 30, 2025
liam923 pushed a commit to oxcaml/merlin that referenced this pull request Jul 10, 2025
liam923 added a commit to oxcaml/merlin that referenced this pull request Jul 16, 2025
* Downstream: Fix occurrences when the definition's source is hidden (ocaml/merlin#1865)

* Downstream: Use new uid info to fix jumps and provide occurrences in both the interface and the implementation (ocaml/merlin #1857)

* Downstream: perform less merges when indexing (ocaml/merlin#1881)

Immediately grow the final index instead of building and merging.

* Merge project-wide renaming changes from  (ocaml/merlin#1877)

* Merge changes adding granular marshal (ocaml/merlin#1889)

* Disable new upstream test relying on Dune

* Downstream: Fixes for renaming(ocaml/merlin#1924)

* Reformat files

* Undo  ->  change

* Add fold to Granular_set

* Resolve conflicts in occurrences.ml

* Promote good test changes

* Fix json serialization for renaming scope

* Promote more good test changes

* Review changes in locate.ml

* Review 1857

* Handle store_shapes in index.ml

* Review occurrences.ml

* Make functor renaming test run

* Review r-modules-and-types.t

* Resolve cr

* Add missing .merlin file

---------

Co-authored-by: Ulysse Gérard <thevoodoos@gmail.com>
Co-authored-by: Ulysse <5031221+voodoos@users.noreply.github.com>
mshinwell added a commit to oxcaml/oxcaml that referenced this pull request May 6, 2026
* Fix indentation

* Add changelog entry for #1840

* Fix locate command in presence of let punning (#96)

* Modify locate behavior with let punning

* Add tests

* Add tests

* Improve test output

* Add broken occurrences tests

* Remove redundant ?attrs:None

* Cleanup compare_nodes function

* Stop using <<<

* Remove usage of <<

* Fix recovery from lock errors (#95)

* Add test demonstrating bug

* Resolve bug

* Improve tests

* Use ref to deal with Msupport.raise_error in Env

* Undo change in jane_syntax_parsing.ml

* Incomplete curried application recovery bugfix (#94)

* Add test demonstrating bug

* Fix bug

* Catch all errors

* Fix test to use $MERLIN

* Prepare changelog for releace 5.2-502

* Add missing changelog entries

* Remove .ocamlformat and .ocamlformat-enable

* Resolve conflicts

* Resolve conflicts

* Resolve conflicts

* Remove bad ocamlformat files in upstream

* Setup CI with formatting

* Add formatting merge commit to .git-blame-ignore-revs

* Import mixed block bugfixes for 5.2 (#102)

* Import ocaml sources for efe8f8dfb4

* Import 5.2 bugfixes

* Fix stack-overflow on mutually recursive definitions (#100)

* Fix type-enclosing crash on mutually recursive definitions

* Bump fuel to 5000

* Add test for old gadt destruct bug (#103)

* Avoid hidden dependencies with short-paths (#101)

* Add test demonstrating problem

* Track module visibility in short paths

* Add test for when there is no non-hidden path

* Resolve .merlin issue

* Refactor to not track visibility and deprecation

* Occurrences canonicalization bugfix (#105)

* Create test demonstrating issue

* Fix bug in resolving relative paths for occurrences

* Fix EXCLUDE_QUERY_DIR for cms/cmt files (#104)

* Fix EXCLUDE_QUERY_DIR for cms/cmt files

* Modify test to use cmt and cms files

* Incorporate PR feedback

* Stop running test on windows

* Refactor to share functionality between build_path and cmt_path

* Merge 5.2.0minus-1 (#106)

* Import ocaml sources for 69c04271e0

* Automatic merges

* Commit merge conflicts

* Resolve conflicts

* Resolve type errors

* Bump magic numbers

* Promote failing tests

* Format code

* Remove unused magic numbers (#107)

* Fix exec magic number

* Remove unused magic numbers

* Add script to update magic numbers (#108)

* Add script to update magic numbers

* Fix CI failure

* Override context when locating (#91)

* Add context override to locate

* Create tests

* Add more context and tests

* Remove query_protocol.mli

* Cleanup signature of Locate.from_string

* Stop using array literals in test

* Declare arrays

* Rewrite test without using arrays

* Fix if syntax

* Change == to =

* Format files

* Ignore `none` locations instead of ghost locations (#110)

* Add tests demonstrating occurrences on punned expressions

* Fix occurrences with let punning

* Add comment to should_ignore_lid

* Include location of entire identifier

* Don't filter hidden nodes when doing occurrences

* Merge 5.2.0minus 2 (#112)

* Import ocaml sources for 8a585cf242

* Update magic numbers

* Promote sherlodoc parser to sources (#113)

(cherry picked from commit 412c3303afe2a01e90d96717fe08cf1e387148fa)

Co-authored-by: Ulysse Gérard <thevoodoos@gmail.com>

* Create dump-file-stats command (#115)

* Create command for printing magic numbers (#117)

* Create command for printing magic numbers

* Fix ci failures

* Better LSP hover interaction in stack-or-heap (#116)

* add tests of behaviors we plan to change

Signed-off-by: David Vulakh <dvulakh@janestreet.com>

* implement [let]-bound function support

Signed-off-by: David Vulakh <dvulakh@janestreet.com>

* implement restricted constructor location

gated behind [-lsp-compat] flag

Signed-off-by: David Vulakh <dvulakh@janestreet.com>

* clean up reported location for let-bound functions

report the entire value binding when not in the lsp-compat regime

also move all the lsp-compat tests to a separate file to group them
together

Signed-off-by: David Vulakh <dvulakh@janestreet.com>

* sundry cleanup

clean up some artifacts of intermediate states to make the total PR
diff cleaner

Signed-off-by: David Vulakh <dvulakh@janestreet.com>

* pr comments

Signed-off-by: David Vulakh <dvulakh@janestreet.com>

* make fmt

Signed-off-by: David Vulakh <dvulakh@janestreet.com>

---------

Signed-off-by: David Vulakh <dvulakh@janestreet.com>

* Make more compatible with opam (#114)

* Merge 5.2.0minus 3 (#119)

* Update import scripts

* Import ocaml sources for e1efceb89a5

* Automatic merges

* Commit conflicts

* Remove files that were deleted in flambda

* Apply old diff to language_extension

* Resolve conflicts

* Resolve errors outside of merlin-specific code

* Resolve more issues

* Fix more issues

* Update magic number script

* Bump magic numbers

* Promote tests

* Add comments about subdirectory

* Use Ast_helper.Sg.mk

* Merge 5.2.0minus-4 (#120)

* Import ocaml sources for 581b385a599

* Automatic merges

* Commit conflicts

* Resolve conflicts

* Resolve typing errors

* Update magic numbers

* Promote tests

* Clean up jkind printing logic

* Remove ocaml version from test output

* Add params to ignore (#121)

* Merge 5.2.0minus-5 (#124)

* Import ocaml sources for df4a6e0ba4f

* Commit merge conflicts

* Resolve syntax errors

* Fix type issues in shared code

* Fix some merlin-specific stuff

* Fix construct

* Fix typedtree_utils

* Fix ptyp_of_type

* Fix stack_or_heap_enclosing

* Fix tail_anaylsis

* Fix context

* Fix env_lookup

* Fix completion

* Update magic numbers

* Add flags to ignore

* Promote failing tests

* Fix record error recovery

* Fix typed holes

* Promote failing test

* Create unboxed record tests

* Rename Tmod_hole to Tmod_typed_hole

* Add test for constructing in a Texp_hole

* Update ci to use ocaml/setup-ocaml@v3

* Fix formatting

* Disable flaky stack-or-heap test

* Fix reduce/reduce conflict (#126)

* Fix handling of instance names in `Persistent_env` (#127)

This change was already made in flambda-backend but looks to have been
inadvertently dropped during a merge.

Signed-off-by: Luke Maurer <lmaurer@janestreet.com>

* Process the `-parameter` flag

Signed-off-by: Luke Maurer <lmaurer@janestreet.com>

* Add tests

Signed-off-by: Luke Maurer <lmaurer@janestreet.com>

* Fix formatting

Signed-off-by: Luke Maurer <lmaurer@janestreet.com>

* Move typed_hole from ocaml-lsp (#129)

* Move typed_hole from ocaml-lsp

* Format

* Distinguish identical files (#130)

* Create test

* Fix bug

* Disable test on windows

* Fix uncaught exception

* Remove `.t` extension from cram config

* Disable test

---------

Co-authored-by: Ulysse Gérard <thevoodoos@gmail.com>

* Use `revert-newlines` and format jq output afterward

Signed-off-by: Luke Maurer <lmaurer@janestreet.com>

* Add tests: modifications, .merlin

Signed-off-by: Luke Maurer <lmaurer@janestreet.com>

* Fiddle with formatting

Signed-off-by: Luke Maurer <lmaurer@janestreet.com>

* Remove timestamps from diff output in test

Signed-off-by: Luke Maurer <lmaurer@janestreet.com>

* Stop server before and after parameters test (#134)

* Update merlin for 5.2.0minus-6 (#132)

* Import ocaml sources for 4eb95cdd48f

* Commit conflicts

* Manually update clflags.ml{,i}

* Manually fix conflicts

* Fix the build

* Promote new test output (after checking it all)

* Update test to new reality

* Update magic numbers

* Import ocaml sources for 6a83bbad9dd

* Commit conflicts

* Resolve conflicts

* Fix compilation problems

* Accept new version numbers in test

* Account for new flags

* format

* Make module paths in jkind match the compiler

* Fix formatting

* Promote generated files

* Import ocaml sources for db3778f932f

* Commit conflicts

* Resolve conflicts

* Revert test

* Remove unused declaration

* Remove unused module alias in ctype

* Bump setup-ocaml version

---------

Co-authored-by: Chris Casinghino <ccasinghino@janestreet.com>
Co-authored-by: Liam Stevenson <lstevenson@janestreet.com>

* Merge 5.2.0minus-7 (#135)

* Import ocaml sources for 9af08951c69

* Merge changes

* Bump magic numbers

* Merge 5.2.0minus 8 (#136)

* Import ocaml sources for dc108ccc92da

* Commit merge conflicts

* Resolve conflicts

* Fix build errors

* Update magic numbers

* Promote tests

* Fix merge mistake in parser

* Updates for 5.2.0minus-9 (#137)

* Import ocaml sources for c3c58afb390

* Commit conflicts

* Resolve conflicts

* Fix type errors

* Bump version numbers

* Update ignored flags

* Promote tests

* Format

* Fix menhir version (#138)

* Make CI fail if there are unpromoted changes

* Promote menhir changes

* Bump Menhir to 20231231

* Print useful info when git diff check fials

* Update for the latest minus9 (#141)

* Import ocaml sources for e06264f05b4

* Commit conflicts

* Fix conflicts

* Fix the build

* Fix jump to definition with matching compilation unit name (#142)

* Add test

* Fix bug

* Commit .merlin file

* Update merlin for 5.2.0minus-10 (#144)

* Import ocaml sources for d8dbd01dd6

* magic number updates for 5.2.0minus-10

* Merge 5.2.0-minus11 (#146)

* Import ocaml sources for 60158e06115

* Merge changes

* Move Unit_info from parsing to typing

* Now merlin needs Numbers (dependency from Jkind)

* Fix easy errors

* Harder fixes. Please review.

* More edits that require review

* Fix default pattern recovery

* Bump magic numbers

* Format files

* Promote some tests

* Fix test to use new mode syntax

* Promote failing parameters test

* Add new ignored flags

* Improve construction of unit_info

---------

Co-authored-by: Richard Eisenberg <reisenberg@janestreet.com>

* Merge 5.2.0minus 12 (#151)

* Import ocaml sources for 87a4cecacc0e

* Merge compiler changes (with conflicts)

* Resolve merge conflicts

* Promote failing tests

* Make record_declaration_dependency side-effectful

* Import ocaml sources for 3db036c957bd

* Merge changes (with conflicts)

* Resolve conflicts

* Resolve type errors

* Bump magic numbers

* Promote failing test

* Merge 5.2.0minus 13 (#153)

* Import ocaml sources for 1bd1ae3ae5

* Merge compiler changes

* Bump magic numbers

* Improve stale occurrences (#123)

* Fix bug with checking staleness

* Create staleness test

* Report stale occurrences

* Always include staleness in json

* Backport PR 1864 (#131)

* test: show issues with deduplication

* Refactor and fix deduplication in type enclosing.

Deduplication can only be done if the type have been printed. We perform that printing only at the junction between the reconstructed identifier enclosings and the ones from the tree nodes because we often want to keep both. All other duplicated ranges are removed.

* Large index alwways show the last type.

* Type-enclosing: revert deduplication in most cases

Since deduplciation has been introduced a while ago the results of type-enclosing (the number of enclosings) have been unstable. This cannot be solved easily on the server-side without printing more types which can lead to performance issues when large modules are involed. We now leave the responsability of deduplication to the clients.

* Promote tests after performing backport

* Stop deduplicating all results

---------

Co-authored-by: Ulysse Gérard <thevoodoos@gmail.com>

* Create compiler merge bot (#122)

* Create compiler merge bot

* Modify base-rev

* Unmodify base-rev

* Update name of workflow

* Fix compiler revision that was screwed up in rebase

* Promote failing tests (#154)

* Merge 5.2.0minus 14 (#156)

* Import ocaml sources for e6099099792

* Merge changes

* Bump magic numbers

* Add workflow_dispatch to workflow (#164)

* Add pull_request workflow trigger (#165)

* Add pull_request workflow trigger

* Only trigger workflow runs on pushes to main

* Merlin downstreams for renaming voodoos (#155)

* Downstream: Fix occurrences when the definition's source is hidden (ocaml/merlin#1865)

* Downstream: Use new uid info to fix jumps and provide occurrences in both the interface and the implementation (ocaml/merlin #1857)

* Downstream: perform less merges when indexing (ocaml/merlin#1881)

Immediately grow the final index instead of building and merging.

* Merge project-wide renaming changes from  (ocaml/merlin#1877)

* Merge changes adding granular marshal (ocaml/merlin#1889)

* Disable new upstream test relying on Dune

* Downstream: Fixes for renaming(ocaml/merlin#1924)

* Reformat files

* Undo  ->  change

* Add fold to Granular_set

* Resolve conflicts in occurrences.ml

* Promote good test changes

* Fix json serialization for renaming scope

* Promote more good test changes

* Review changes in locate.ml

* Review 1857

* Handle store_shapes in index.ml

* Review occurrences.ml

* Make functor renaming test run

* Review r-modules-and-types.t

* Resolve cr

* Add missing .merlin file

---------

Co-authored-by: Ulysse Gérard <thevoodoos@gmail.com>
Co-authored-by: Ulysse <5031221+voodoos@users.noreply.github.com>

* Adjust completion order (#157)

* De-prioritize types in expression completion

* Add tests

* Fix flakes in parameters tests

* Add tests for printing jkinds (#169)

* Merge 5.2.0minus-16 (#172)

* Import ocaml sources for 00e9f22e7c5

* Automatic (non-conflicty) merges

* Commit merge conflicts

* Resolve conflicts (with type errors outstanding)

* Fix some type errors

* Fix more errors

* Fix remaining type errors

* Bump magic numbers for 5.2.0minus-16

* Promote some tests

* Promote jkind formatting change

* Resolve crs

* Add ignored flags

* Comment out assertion that was causing tests to fail

* Bump dune version in CI

* Prevent short paths from inserting duplicates into the graph

* Merge 5.2.0minus 16 retag (#174)

* Import ocaml sources for 31a75865c15

* Automatic merges

* Remove removed flags

* Merge 5.2.0minus 16 retag2 (#175)

* Import ocaml sources for 2314e9cbd6a

* Automatic merges

* Ppx improvements (#177)

* merlin document support for PPXs (#167)

* initial working with tests

* temp update

* temp

* major refactors, add support for .mli files

* refactoring and cleanup

* move all logic into override_document.ml; cleanup code; add logging

* add many more tests; handle edge cases

* revert std.ml changes

* improve comments

* update more comments

* suggested changes

* test nested ppx

* document payload test

* some more tests

* resolve final comments

* add test on floating attribute

* add test for attribute in an extension's payload

* Fix function syntax

* merlin overrides: merlin locate support for PPXs (#171)

* merlin document support for PPXs (#167)

* initial working with tests

* temp update

* temp

* major refactors, add support for .mli files

* refactoring and cleanup

* move all logic into override_document.ml; cleanup code; add logging

* add many more tests; handle edge cases

* revert std.ml changes

* improve comments

* update more comments

* suggested changes

* test nested ppx

* document payload test

* some more tests

* resolve final comments

* add test on floating attribute

* add test for attribute in an extension's payload

* Fix function syntax

* working parsing of new merlin.document structure

* fix document tests with new merlin.document structure

* replicate merlin.document tests for merlin.locate

* generalize comments, code for all overrides

* delete duplicate files from merge conflict

* add test overriding locate of a non-PPX

* add test on overriding default behavior of document

* format

* combine document and locate tests into single file

* adjust for new attribute structure

* update comment documenting expected attribute structure

* fix typo

* remove named parameter

* merlin locate outputs absolute paths using source_root

---------

Co-authored-by: Liam Stevenson <lstevenson@janestreet.com>

* merlin overrides use interval tree instead of list (#173)

* interval tree interface

* interval tree implementation and basic test

* use array for construction, add more tests

* add and update comments

* fix error messages, validate inputs

* use list instead of array, add tests, add comments

* integrate into overrides.ml

* add test for empty tree

* resolve requested changes

* add test for zero-width interval

* undo test reformat

* update outdated comment

* update another oudated comment

* update comment

* Cache overrides interval trees using Phase_cache (#176)

* implementation but failing tests

* working overrides caching with tests

* add USE_PPX_CACHE, add all cache entries to tests

* update comments

* use Option.map

* move comment

* fix overrides caching mechanism, functorize caches, add cache invalidation test

* add comments, rename

* cleanup test output, and resolve other comments

* remove unnecessary sed commands from tests

* add commentsg

---------

Co-authored-by: Max Wang <orangemax888@gmail.com>

* Merge 5.2.0minus-17 (#178)

* Import ocaml sources for 342a11315b4

* Automatic merges

* Add new files

* Commit merge conflicts

* Trivial merge conflict resolution

* Fix some type errors

* Finish fixing type errors in merged files

* Teach merlin about Pexp_idx

* Promote tests

* Import ocaml sources for d4c133509a1

* Automatic merges

* Merge trivial conflicts

* Teach merlin about atomic_loc

* Import ocaml sources for 99a981ed891

* Automatic merges

* Commit merge conflicts

* Trivial merge conflict resolution

* Fix typing errors

* Bump magic numbers

* Add ignored flags

* Import ocaml sources for e50954f1a39

* Automatic merges

* Make Merlin look for dot-merlin-reader in the install directory (#179)

* Make Merlin look for dot-merlin-reader in the install directory

* Use realpath

* Merge 5.2.0minus-18 (#180)

* Import ocaml sources for dec889241a6

* Automatic merges

* Commit merge conflicts

* Fix conflicts

* Fix type errors in merlin-specific code

* Bump magic numbers

* Promote tests

* Add ignored flags

* Fix formatting

* Comment out probe error (#181)

* Add option to not print types when getting the outline (#183)

* Add option to not print types when getting the outline

* Fix formatting

* Fix typo in doc

* Fix exponential short-paths issue (#182)

* Fix exponential short-paths issue

* Revert "Fix exponential short-paths issue"

This reverts commit 263e7ac2e59e3367ceaba7b9d14b865794d41bab.

* Avoid exponential expansion by only calling find_module when necessary

* Add syntax hovers for oxcaml (#168)

* Add syntax hovers for oxcaml

* Add doc for jkinds

* Remove specialise annotations

* Add zero_alloc descriptions

* Add most inline annoation descs

* Do rest of inline annotations

* Add more

* Add noalloc desc

* Add modes, modalitys, and jkinds

* Fill in remaining todos

* Add tests

* Apply suggestion from @goldfirere

Co-authored-by: Richard Eisenberg <reisenberg@janestreet.com>

* Update src/analysis/syntax_doc.ml

Co-authored-by: Richard Eisenberg <reisenberg@janestreet.com>

* Rephrase comonadic mode descs

* Update src/analysis/syntax_doc.ml

Co-authored-by: Richard Eisenberg <reisenberg@janestreet.com>

* Update src/analysis/syntax_doc.ml

Co-authored-by: Richard Eisenberg <reisenberg@janestreet.com>

* Update mode descriptions

* of -> with

* Update src/analysis/syntax_doc.ml

Co-authored-by: Richard Eisenberg <reisenberg@janestreet.com>

* Update src/analysis/syntax_doc.ml

Co-authored-by: Richard Eisenberg <reisenberg@janestreet.com>

* Update src/analysis/syntax_doc.ml

Co-authored-by: Richard Eisenberg <reisenberg@janestreet.com>

* Update src/analysis/syntax_doc.ml

Co-authored-by: Richard Eisenberg <reisenberg@janestreet.com>

* zero-alloc desc changes

* Update src/analysis/syntax_doc.ml

Co-authored-by: Richard Eisenberg <reisenberg@janestreet.com>

* Update src/analysis/syntax_doc.ml

Co-authored-by: Richard Eisenberg <reisenberg@janestreet.com>

* Update src/analysis/syntax_doc.ml

Co-authored-by: Richard Eisenberg <reisenberg@janestreet.com>

* Update src/analysis/syntax_doc.ml

Co-authored-by: Richard Eisenberg <reisenberg@janestreet.com>

* Update src/analysis/syntax_doc.ml

Co-authored-by: Richard Eisenberg <reisenberg@janestreet.com>

* Update src/analysis/syntax_doc.ml

Co-authored-by: Richard Eisenberg <reisenberg@janestreet.com>

* Update exclave desc

* Update tests/test-dirs/syntax-document/language-extensions.t/run.t

Co-authored-by: Richard Eisenberg <reisenberg@janestreet.com>

* Add comment

---------

Co-authored-by: Richard Eisenberg <reisenberg@janestreet.com>

* Update menhir version in readme (#184)

* Merge 5.2.0minus-19 (#185)

* Import ocaml sources for 951524cd4f4e

* Automatic merges

* Add typing/mode_hint.mli

* Commit conflicts

* Resolve easy conflicts

* Fix some type errors

* Changes for new shapes

* Fixup errors in lexer

* Fix typing errors

* Fixup syntax_doc

* Bump magic numbers

* Fix tests

* Add ignored flags

* Import ocaml sources for c2a3ea6fbd0b

* Automatic merges

* Disable warning 47

* Use MB.of_int

* Add update-flake-lock workflow (#186)

* Bump menhir version in flake.nix (#187)

* Revert "Add update-flake-lock workflow (#186)" (#189)

This reverts commit efeefa0.

* Fix Nix Flake to continue building (#188)

The check phase for merlin had to be disabled as they depend on a
MERLIN_TEST_OCAML_PATH of a locally built ocaml.

Flake lock file updates:

• Updated input 'flake-utils':
    'github:numtide/flake-utils/b1d9ab70662946ef0850d488da1c9019f3a9752a?narHash=sha256-SZ5L6eA7HJ/nmkzGG7/ISclqe6oZdOZTNoesiInkXPQ%3D' (2024-03-11)
  → 'github:numtide/flake-utils/11707dc2f618dd54ca8739b309ec4fc024de578b?narHash=sha256-l0KFg5HjrsfsO/JpG%2Br7fRrqm12kzFHyUHqHCVpMMbI%3D' (2024-11-13)
• Updated input 'menhir-repository':
    'gitlab:fpottier/menhir/abb46d3d9c536bcbe30025f37474e3b4c8288590?host=gitlab.inria.fr&narHash=sha256-i/Xs6G1L/bZ1zj%2BLB5ehiaBC10ounc1koURV3vFolhI%3D' (2020-12-16)
  → 'gitlab:fpottier/menhir/d3d815e4f554da68b8c247241c8f8678926eecaa?host=gitlab.inria.fr&narHash=sha256-veB0ORHp6jdRwCyDDAfc7a7ov8sOeHUmiELdOFf/QYk%3D' (2023-12-31)
• Updated input 'nixpkgs':
    'github:nixos/nixpkgs/c58b4a9118498c1055c5908a5bbe666e56abe949?narHash=sha256-YtlyfqOdYMuu7gumZtK0Kg7jr4OKfHUhJkZfNUryw68%3D' (2024-06-17)
  → 'github:nixos/nixpkgs/8eaee110344796db060382e15d3af0a9fc396e0e?narHash=sha256-iCGWf/LTy%2BaY0zFu8q12lK8KuZp7yvdhStehhyX1v8w%3D' (2025-09-19)

* Add `merlin.ppx_template_generated` (#190)

* Add attribute to mark whether an identifier is generated by ppx_template

* Fix typo

* Merge 5.2.0minus-22 (#191)

* flambda-backend -> oxcaml

* Import ocaml sources for 81eecd7a1fa

* All 12 files with merge conflicts have been resolved:

  * src/ocaml/preprocess/parser_raw.mly (Hunks 3, 4, 10)
  - Added new grammar rules for quotation metaprogramming (spliceable_expr, Pexp_quote, Pexp_splice)

  * src/ocaml/parsing/pprintast.ml (Hunk 2)
  - Renamed jkind constructors with Pjk_ prefix while keeping Misc_stdlib.pp_parens_if

  * src/ocaml/typing/env.ml (Hunks 2, 5, 17, 23)
  - Added stage field to environment and integrated stage tracking helpers
  - Added Quotation_lock | Splice_lock cases for lock walking

  * src/ocaml/typing/env.mli (Hunk 7)
  - Added stage-related function signatures alongside existing fold functions

  * src/ocaml/typing/outcometree.mli (Hunk 1)
  - Added Oval_code constructor for quoted code values

  * src/ocaml/typing/printtyp.ml (Hunks 2, 3)
  - Kept Internal_names module structure and removed conflicting definitions

  * src/ocaml/typing/typecore.ml (Hunks 7, 16)
  - Added Texp_quotation, Texp_eval expression cases
  - Updated allocation mode error handling

  * src/ocaml/typing/typedtree.ml (Hunks 1, 3)
  - Added three new expression constructors: Texp_quotation, Texp_antiquotation, Texp_eval
  - Integrated fold_antiquote functions for traversing quotations

  * src/ocaml/typing/typedtree.mli (Hunks 1, 3)
  - Added new expression constructor signatures
  - Added fold_antiquote_exp function signature

  * src/ocaml/typing/typemod.ml (Hunk 1)
  - Combined quotation check with empty list parameter in transl_signature call

  * src/ocaml/typing/typeopt.ml (Hunks 1, 4)
  - Removed local is_always_gc_ignorable function (moved to Ctype module)
  - Updated calls to use Ctype.is_always_gc_ignorable

  * src/ocaml/typing/typetexp.ml (Hunk 4)
  - Updated lookup_local signature to return (type_expr * Env.stage) tuple

* Vibecoded compilation fixes

* Add camlinternalQuote.ml{,i} from 5.2.0minus-22

* update-magic-numbers.sh 569 5.2.0minus-22

* Formatting

* Import ocaml sources for 05b98d54a75

* Promote tests

* Small fixes

* Comment out line

* Update comments in parser

* Fix forkable mode doc

Co-authored-by: Max Slater <max@thenumb.at>

---------

Co-authored-by: Jack Rickard <jrickard@janestreet.com>
Co-authored-by: Liam Stevenson <lstevenson@janestreet.com>
Co-authored-by: Max Slater <max@thenumb.at>

* Ignore llvm flags (#192)

* Push forcing into short_paths_additions_desc

* Stop forcing subst when constructing short-paths graph

* Add --lalr flag to menhir invocations (#194)

* Refactor index caching. Get rid of the stamping.

* Only index the buffer if the query needs lit

Suggested-by: Liam Stevenson <lstevenson@janestreet.com>

* Filter ghost locations from syntax_doc

* fix param lib vs. short path

* address comments

* Add afl flags

* Fix missing semicolon

* Add test for outline with include statements

* Fix bug

* Import ocaml sources for 8abf835dda41

* Automatic merges

* Commit conflicts

* Resolve trivial conflicts

* Fix some errors

* Fix more errors

* Add doc for shareable

* Fix more errors

* Make lazier

* Bump magic numbers

* Benign test changes

* Fix documentation bug

* Add doc for staticity

* Undo + location change

* Fix parameterized modules test

* Update ignored flags

* Import ocaml sources for 996a6635f0b1

* Automatic merges

* Commit conflicts

* Resolve conflicts

* Fix type error

* Add type hovers for jkinds

* Add more tests

* Add more tests

* Add test for ignored flags and add to ignored flags list

* Fix formatting

* Use posix-compatible syntax

* Import ocaml sources for d6e630469425

* Automatic merges

* Commit conflicts

* Resolve trivial conflicts

* Fix laziness issues

* Nontrivial conflict resolution

* Resolve typing errors

* Bump magic numbers

* Promote tests and add ignored flags

* Add cram test to reproduce the issue

* Merge pull request #1951 from liam923/locate-type-multi

Add `locate_type_multi` query

* Resolve conflicts

* Fixup

* Remove stray space

* Merge pull request #2025 from liam923/include-variables-in-result

Add `Other` variant to locate-types command

* Merge pull request #2027 from liam923/strip-option-from-optional-params

Strip option from optional parameters

* Fix optional function params

* Add kind-enclosing query

* Use sh compliant syntax

* Use sh compliant syntax

* Add override_verbosity option

* import changes from ocaml/merlin#2028

* ocamlformat

* Update docs

* Merge `5.2.0minus-29` (#212)

* Import ocaml sources for 977afdd4ebd

* Automatic merges

* Commit conflicts

* Resolve trivial conflicts

* Resolve trivial type errors

* Fix remaining errors

* Bump magic numbers

* Promote tests and fix some issues

* Bump bootstrap compiler for flambda on github

* Fix formatting

* Import ocaml sources for 0a1dc8de0264

* Automatic merges

* Fixup jkind printing

* Add flags

* Import ocaml sources for 4ac226a124a5

* Commit conflicts

* Resolve conflicts

* Update version name

* Promote tests

* Resolve conflicts (#213)

* Allow the signature help only after the function name

* Signature help should not appear on the function name (#206)

* Add cram test to reproduce the issue

* Allow the signature help only after the function name

* Add mode_enclosing query (#210)

* Add mode-enclosing

* Remove duplicate test

* Add test

* Fix verbosity issue

* Import compiler changes from PR 5398 (#218)

* Backport changes

* Fix spacing

* Add reproduction test

* Check the optional active_param which should not trigger the signature help if none

* Change output for all legacy modes in hover (#219)

* Fix locate for punned record pattern (#147)

* Create test demonstrating issue with record field punning in patterns

* Refactor Mbrowse

* Fix record pun behavior

* Use . instead of source

* Fix positional arg issue

* Merge `5.2.0minus-31` (#221)

* Import ocaml sources for 9790921724a7

* Commit conflicts

* Resolve trivial conflicts

* Fix env conflicts

* Fix typemod conflicts

* Fix location conflicts

* Move format_doc

* Apply misc changes

* Fix some type errors

* Make Format_doc changes

* More type fixes

* Handle Trepr

* Handle unboxed bools and units

* Misc fixups

* Fixup jkinds

* Fix env shadowing

* Bump magic numbers

* Add ignored flags

* Promote some tests and do some fixes

* Fix jkind printing env

* Promote test changes

* Import ocaml sources for ae041759a243

* Automatic merges

* Fix formatting

* Eta-expand Buffer.add_string

* Import ocaml sources for 7d714cfb3f

* Merge conflicts

* Resolve trivial conflict

* Add ticket numbers

* Make Trepr safer

* Update compiler revision

* Merge 5.2.0minus-32 (#224)

* Import ocaml sources for 812c769361

* Commit conflicts

* Resolve conflicts

* Fixup errors

* Bump magic numbers

* Promote tests

* Fix formatting

* Recovery in application in local open (#222)

* Create test demonstrating error

* Add a recovery layer around type_argument.

Fixes ocaml/merlin#1580 and ocaml/merlin#1588

---------

Co-authored-by: Ulysse Gérard <thevoodoos@gmail.com>

* Merge 5.2.0minus-34 (#225)

* Import ocaml sources for oxcaml/oxcaml@5.2.0minus-30-121-gc7fb58867d3

* Commit conflicts

* Resolve trivial issues

* Stop using labeled tuples

* Fix Texp_idents

* Fixup lpoly stuff

* Bump magic numbers

* Add new ignored flags

* Fix formatting

* Add -linscan to list of ignored flags (#226)

* Import ocaml sources for oxcaml/oxcaml@5.2.0minus-30-192-g8cb0afc5252

* Commit conflicts

* Resolve trivial conflicts

* Trivial type errors

* Resolve try_expand_head conflict

* Fix type errors

* Bump magic numbers

* Promote typedtree test

* Add ignored flags

* Resolve cmi loading during destruct

* Add cr-somedays for tquote in locate_types

* Fix formatting (#229)

* Fix ptyp_of_type for Tquote_eval (#230)

* Upgrade to 5.4 (#227)

* Upgrade to 5.4

* Upgrade to ocamlformat 0.29.0

* Reformat for ocamlformat 0.29.0

* Update magic number mapping from 35 to 36 (#231)

* Test for compiler flags (#232)

* Test for ocamlopt flags

* Add missing flags

* Add cram test to reproduce the issue (#215)

* Signature help should suggest optional arguments (#216)

* Add tests to demonstrate wrong behaviour with labelled and optional arguments

* Update parameter_info type to include apply_arg

* Filter the parameters to not highlight parameters already written

* Import ocaml sources for oxcaml/oxcaml@5.2.0minus-30-266-geb63e0e4186

* Commit conflicts

* Fix easy conflicts

* Generic changes

* Load path changes

* Bump magic numbers

* Promote tests

* Update ignored flags

* Fix syntax_doc.ml

* Improve syntax docs for scannable axes

* Clean up load path init impl

* Add -Ix test

* Add clarifying comment

* Fix formatting

* New cmi cache invalidation (#237)

* Add test for new cmi file

* Remember all missing persistent structs regardless of the allow_hidden flag.

This fixes Merlin's invalidation process.

Suggested-by: Gabriel Scherer <gabriel.scherer@gmail.com>

* Make test stable with different touch flavors

---------

Co-authored-by: Ulysse Gérard <thevoodoos@gmail.com>

* Reduce diff in typeopt (#233)

* Reduce diff in typeopt

* Actually raise errors

* Bring in new compiler changes

* Accept suggestions

* Update flake to use OCaml 5.4 (#236)

* Update flake to use OCaml 5.4

* Added revision justification

* Move misc into src/ocaml/utils

* Move format_doc into src/ocaml/utils

* Copy compiler's misc into merlin

* Delete Misc_stdlib

* Apply Merlin diff to misc.ml

* Reduce diff from compiler

* Move functions from misc.ml to std.ml

* Fix formatting

* Update CI to work with Merlin

* Update Merlin's import script

* Update .github/workflows/merlin.yml

Co-authored-by: David Allsopp <dallsopp@janestreet.com>

* Write import script in terms of subtree_prefix

* opam

* Remove dune-workspace

---------

Signed-off-by: David Vulakh <dvulakh@janestreet.com>
Signed-off-by: Luke Maurer <lmaurer@janestreet.com>
Co-authored-by: Ulysse Gérard <thevoodoos@gmail.com>
Co-authored-by: Ulysse <5031221+voodoos@users.noreply.github.com>
Co-authored-by: Nick Roberts <nroberts@janestreet.com>
Co-authored-by: David Vulakh <33737747+dvulakh@users.noreply.github.com>
Co-authored-by: Luke Maurer <Luke.Maurer@alumni.carleton.edu>
Co-authored-by: Luke Maurer <lmaurer@janestreet.com>
Co-authored-by: Richard Eisenberg <reisenberg@janestreet.com>
Co-authored-by: Chris Casinghino <ccasinghino@janestreet.com>
Co-authored-by: dkalinichenko-js <118547217+dkalinichenko-js@users.noreply.github.com>
Co-authored-by: Max Wang <orangemax888@gmail.com>
Co-authored-by: George Shammas <georgyo@gmail.com>
Co-authored-by: Mark Shinwell <mshinwell@pm.me>
Co-authored-by: Jack Rickard <jrickard@janestreet.com>
Co-authored-by: Max Slater <max@thenumb.at>
Co-authored-by: Zesen Qian <github@riaqn.org>
Co-authored-by: Zesen Qian <riaqn@users.noreply.github.com>
Co-authored-by: Lucccyo <cha.git@mailo.fr>
Co-authored-by: Kostia Balytskyi <kostia.balitsky@gmail.com>
Co-authored-by: Charlène_Gros <78226952+Lucccyo@users.noreply.github.com>
Co-authored-by: Gavin Gray <20209337+gavinleroy@users.noreply.github.com>
Co-authored-by: David Allsopp <dallsopp@janestreet.com>
Co-authored-by: David Allsopp <david.allsopp@metastack.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants