1 """automatically manage newlines in repository files

     2 

     3 This extension allows you to manage the type of line endings (CRLF or

     4 LF) that are used in the repository and in the local working

     5 directory. That way you can get CRLF line endings on Windows and LF on

     6 Unix/Mac, thereby letting everybody use their OS native line endings.

     7 

     8 The extension reads its configuration from a versioned ``.hgeol``

     9 configuration file every time you run an ``hg`` command. The

    10 ``.hgeol`` file use the same syntax as all other Mercurial

    11 configuration files. It uses two sections, ``[patterns]`` and

    12 ``[repository]``.

    13 

    14 The ``[patterns]`` section specifies how line endings should be

    15 converted between the working copy and the repository. The format is

    16 specified by a file pattern. The first match is used, so put more

    17 specific patterns first. The available line endings are ``LF``,

    18 ``CRLF``, and ``BIN``.

    19 

    20 Files with the declared format of ``CRLF`` or ``LF`` are always

    21 checked out and stored in the repository in that format and files

    22 declared to be binary (``BIN``) are left unchanged. Additionally,

    23 ``native`` is an alias for checking out in the platform's default line

    24 ending: ``LF`` on Unix (including Mac OS X) and ``CRLF`` on

    25 Windows. Note that ``BIN`` (do nothing to line endings) is Mercurial's

    26 default behaviour; it is only needed if you need to override a later,

    27 more general pattern.

    28 

    29 The optional ``[repository]`` section specifies the line endings to

    30 use for files stored in the repository. It has a single setting,

    31 ``native``, which determines the storage line endings for files

    32 declared as ``native`` in the ``[patterns]`` section. It can be set to

    33 ``LF`` or ``CRLF``. The default is ``LF``. For example, this means

    34 that on Windows, files configured as ``native`` (``CRLF`` by default)

    35 will be converted to ``LF`` when stored in the repository. Files

    36 declared as ``LF``, ``CRLF``, or ``BIN`` in the ``[patterns]`` section

    37 are always stored as-is in the repository.

    38 

    39 Example versioned ``.hgeol`` file::

    40 

    41   [patterns]

    42   **.py = native

    43   **.vcproj = CRLF

    44   **.txt = native

    45   Makefile = LF

    46   **.jpg = BIN

    47 

    48   [repository]

    49   native = LF

    50 

    51 .. note::

    52    The rules will first apply when files are touched in the working

    53    copy, e.g. by updating to null and back to tip to touch all files.

    54 

    55 The extension uses an optional ``[eol]`` section in your hgrc file

    56 (not the ``.hgeol`` file) for settings that control the overall

    57 behavior. There are two settings:

    58 

    59 - ``eol.native`` (default ``os.linesep``) can be set to ``LF`` or

    60   ``CRLF`` to override the default interpretation of ``native`` for

    61   checkout. This can be used with :hg:`archive` on Unix, say, to

    62   generate an archive where files have line endings for Windows.

    63 

    64 - ``eol.only-consistent`` (default True) can be set to False to make

    65   the extension convert files with inconsistent EOLs. Inconsistent

    66   means that there is both ``CRLF`` and ``LF`` present in the file.

    67   Such files are normally not touched under the assumption that they

    68   have mixed EOLs on purpose.

    69 

    70 The ``win32text.forbid*`` hooks provided by the win32text extension

    71 have been unified into a single hook named ``eol.hook``. The hook will

    72 lookup the expected line endings from the ``.hgeol`` file, which means

    73 you must migrate to a ``.hgeol`` file first before using the hook.

    74 

    75 See :hg:`help patterns` for more information about the glob patterns

    76 used.

    77 """

    78 

    79 from mercurial.i18n import _

    80 from mercurial import util, config, extensions, match

    81 import re, os

    82 

    83 # Matches a lone LF, i.e., one that is not part of CRLF.

    84 singlelf = re.compile('(^|[^\r])\n')

    85 # Matches a single EOL which can either be a CRLF where repeated CR

    86 # are removed or a LF. We do not care about old Machintosh files, so a

    87 # stray CR is an error.

    88 eolre = re.compile('\r*\n')

    89 

    90 

    91 def inconsistenteol(data):

    92     return '\r\n' in data and singlelf.search(data)

    93 

    94 def tolf(s, params, ui, **kwargs):

    95     """Filter to convert to LF EOLs."""

    96     if util.binary(s):

    97         return s

    98     if ui.configbool('eol', 'only-consistent', True) and inconsistenteol(s):

    99         return s

   100     return eolre.sub('\n', s)

   101 

   102 def tocrlf(s, params, ui, **kwargs):

   103     """Filter to convert to CRLF EOLs."""

   104     if util.binary(s):

   105         return s

   106     if ui.configbool('eol', 'only-consistent', True) and inconsistenteol(s):

   107         return s

   108     return eolre.sub('\r\n', s)

   109 

   110 def isbinary(s, params):

   111     """Filter to do nothing with the file."""

   112     return s

   113 

   114 filters = {

   115     'to-lf': tolf,

   116     'to-crlf': tocrlf,

   117     'is-binary': isbinary,

   118 }

   119 

   120 

   121 def hook(ui, repo, node, hooktype, **kwargs):

   122     """verify that files have expected EOLs"""

   123     files = set()

   124     for rev in xrange(repo[node].rev(), len(repo)):

   125         files.update(repo[rev].files())

   126     tip = repo['tip']

   127     for f in files:

   128         if f not in tip:

   129             continue

   130         for pattern, target in ui.configitems('encode'):

   131             if match.match(repo.root, '', [pattern])(f):

   132                 data = tip[f].data()

   133                 if target == "to-lf" and "\r\n" in data:

   134                     raise util.Abort(_("%s should not have CRLF line endings")

   135                                      % f)

   136                 elif target == "to-crlf" and singlelf.search(data):

   137                     raise util.Abort(_("%s should not have LF line endings")

   138                                      % f)

   139 

   140 

   141 def preupdate(ui, repo, hooktype, parent1, parent2):

   142     #print "preupdate for %s: %s -> %s" % (repo.root, parent1, parent2)

   143     repo.readhgeol(parent1)

   144     return False

   145 

   146 def uisetup(ui):

   147     ui.setconfig('hooks', 'preupdate.eol', preupdate)

   148 

   149 def extsetup(ui):

   150     try:

   151         extensions.find('win32text')

   152         raise util.Abort(_("the eol extension is incompatible with the "

   153                            "win32text extension"))

   154     except KeyError:

   155         pass

   156 

   157 

   158 def reposetup(ui, repo):

   159     uisetup(repo.ui)

   160     #print "reposetup for", repo.root

   161 

   162     if not repo.local():

   163         return

   164     for name, fn in filters.iteritems():

   165         repo.adddatafilter(name, fn)

   166 

   167     ui.setconfig('patch', 'eol', 'auto')

   168 

   169     class eolrepo(repo.__class__):

   170 

   171         _decode = {'LF': 'to-lf', 'CRLF': 'to-crlf', 'BIN': 'is-binary'}

   172         _encode = {'LF': 'to-lf', 'CRLF': 'to-crlf', 'BIN': 'is-binary'}

   173 

   174         def readhgeol(self, node=None, data=None):

   175             if data is None:

   176                 try:

   177                     if node is None:

   178                         data = self.wfile('.hgeol').read()

   179                     else:

   180                         data = self[node]['.hgeol'].data()

   181                 except (IOError, LookupError):

   182                     return None

   183 

   184             if self.ui.config('eol', 'native', os.linesep) in ('LF', '\n'):

   185                 self._decode['NATIVE'] = 'to-lf'

   186             else:

   187                 self._decode['NATIVE'] = 'to-crlf'

   188 

   189             eol = config.config()

   190             # Our files should not be touched. The pattern must be

   191             # inserted first override a '** = native' pattern.

   192             eol.set('patterns', '.hg*', 'BIN')

   193             # We can then parse the user's patterns.

   194             eol.parse('.hgeol', data)

   195 

   196             if eol.get('repository', 'native') == 'CRLF':

   197                 self._encode['NATIVE'] = 'to-crlf'

   198             else:

   199                 self._encode['NATIVE'] = 'to-lf'

   200 

   201             for pattern, style in eol.items('patterns'):

   202                 key = style.upper()

   203                 try:

   204                     self.ui.setconfig('decode', pattern, self._decode[key])

   205                     self.ui.setconfig('encode', pattern, self._encode[key])

   206                 except KeyError:

   207                     self.ui.warn(_("ignoring unknown EOL style '%s' from %s\n")

   208                                  % (style, eol.source('patterns', pattern)))

   209 

   210             include = []

   211             exclude = []

   212             for pattern, style in eol.items('patterns'):

   213                 key = style.upper()

   214                 if key == 'BIN':

   215                     exclude.append(pattern)

   216                 else:

   217                     include.append(pattern)

   218 

   219             # This will match the files for which we need to care

   220             # about inconsistent newlines.

   221             return match.match(self.root, '', [], include, exclude)

   222 

   223         def _hgcleardirstate(self):

   224             self._eolfile = self.readhgeol() or self.readhgeol('tip')

   225 

   226             if not self._eolfile:

   227                 self._eolfile = util.never

   228                 return

   229 

   230             try:

   231                 cachemtime = os.path.getmtime(self.join("eol.cache"))

   232             except OSError:

   233                 cachemtime = 0

   234 

   235             try:

   236                 eolmtime = os.path.getmtime(self.wjoin(".hgeol"))

   237             except OSError:

   238                 eolmtime = 0

   239 

   240             if eolmtime > cachemtime:

   241                 ui.debug("eol: detected change in .hgeol\n")

   242                 # TODO: we could introduce a method for this in dirstate.

   243                 wlock = None

   244                 try:

   245                     wlock = self.wlock()

   246                     for f, e in self.dirstate._map.iteritems():

   247                         self.dirstate._map[f] = (e[0], e[1], -1, 0)

   248                     self.dirstate._dirty = True

   249                     # Touch the cache to update mtime. TODO: are we sure this

   250                     # always enought to update the mtime, or should we write a

   251                     # bit to the file?

   252                     self.opener("eol.cache", "w").close()

   253                 finally:

   254                     if wlock is not None:

   255                         wlock.release()

   256 

   257         def commitctx(self, ctx, error=False):

   258             for f in sorted(ctx.added() + ctx.modified()):

   259                 if not self._eolfile(f):

   260                     continue

   261                 data = ctx[f].data()

   262                 if util.binary(data):

   263                     # We should not abort here, since the user should

   264                     # be able to say "** = native" to automatically

   265                     # have all non-binary files taken care of.

   266                     continue

   267                 if inconsistenteol(data):

   268                     raise util.Abort(_("inconsistent newline style "

   269                                        "in %s\n" % f))

   270             return super(eolrepo, self).commitctx(ctx, error)

   271     repo.__class__ = eolrepo

   272     repo._hgcleardirstate()