Compare commits

...

619 Commits

Author SHA1 Message Date
8558838922 UefiCpuPkg/CommonFeature: Always set FEATURE_CONTROL.Lock
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1305

The patch reverts commit 1ed6498c4a
* UefiCpuPkg/CommonFeature: Skip locking when the feature is disabled

FEATURE_CONTROL.Lock bit is controlled by feature
CPU_FEATURE_LOCK_FEATURE_CONTROL_REGISTER. The commit 1ed649 fixes
a bug that when the feature is disabled, the Lock bit is cleared.
But it's a security hole if the bit is cleared when booting OS.
We can argue that platform needs to make sure the value
of PcdCpuFeaturesUserConfiguration should be set properly to make
sure feature CPU_FEATURE_LOCK_FEATURE_CONTROL_REGISTER is enabled.

But it's better to guarantee this in the generic core code.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Eric Dong <eric.dong@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Cc: Andrew Fish <afish@apple.com>
Cc: Leif Lindholm <leif.lindholm@linaro.org>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
2018-11-14 11:02:48 +08:00
bd224a5dad MdeModulePkg/NvmExpressPei: Refine data buffer & len check in PassThru
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=1142

The fix is similar to commit ebb6c7633b.
We found that a similar fix should be applied to the NVMe PEI driver as
well. Hence, this one is for the PEI counterpart driver.

According to the the NVM Express spec Revision 1.1, for some commands
(like Get/Set Feature Command, Figure 89 & 90 of the spec), the Memory
Buffer maybe optional although the command opcode indicates there is a
data transfer between host & controller (Get/Set Feature Command, Figure
38 of the spec).

Hence, this commit refine the checks for the 'TransferLength' and
'TransferBuffer' field of the
EDKII_PEI_NVM_EXPRESS_PASS_THRU_COMMAND_PACKET structure to address this
issue.

Cc: Andrew Fish <afish@apple.com>
Cc: Leif Lindholm <leif.lindholm@linaro.org>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Jiewen Yao <Jiewen.yao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Hao Wu <hao.a.wu@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Philippe Mathieu-Daude <philmd@redhat.com>
2018-11-13 20:54:44 +08:00
da2c81ee96 Maintainers.txt: Update EDK II Releases to EDK-II-Release-Planning wiki
In V2, add Liming Gao as the maintainer of EDK II Releases.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
2018-11-12 22:18:16 +08:00
246604e3a4 Reapply "OvmfPkg/QemuVideoDxe: list "UnalignedIoInternal.h" in the INF file"
This reverts commit e038bde267, reapplying
b2959e9f1a.

Note that the commit now being reverted is technically correct; the only
reason we're reverting it is because it should not have been pushed past
the Soft Feature Freeze for the edk2-stable201811 tag.

Cc: Anthony Perard <anthony.perard@citrix.com>
Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Julien Grall <julien.grall@linaro.org>
Cc: Philippe Mathieu-Daudé <philmd@redhat.com>
Cc: yuchenlin <yuchenlin@synology.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1319
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: yuchenlin <yuchenlin@synology.com>
2018-11-12 12:24:05 +01:00
2e77f0e7b5 Reapply "OvmfPkg/QemuVideoDxe: VMWare SVGA device support"
This reverts commit 98856a724c, reapplying
c137d95081.

Note that the commit now being reverted is technically correct; the only
reason we're reverting it is because it should not have been pushed past
the Soft Feature Freeze for the edk2-stable201811 tag.

Cc: Anthony Perard <anthony.perard@citrix.com>
Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Julien Grall <julien.grall@linaro.org>
Cc: Philippe Mathieu-Daudé <philmd@redhat.com>
Cc: yuchenlin <yuchenlin@synology.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1319
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: yuchenlin <yuchenlin@synology.com>
2018-11-12 12:24:03 +01:00
615c2c766e Reapply "OvmfPkg/QemuVideoDxe: Helper functions for unaligned port I/O."
This reverts commit 438ada5aa5,
reapplying 05a5379458.

Note that the commit now being reverted is technically correct; the only
reason we're reverting it is because it should not have been pushed past
the Soft Feature Freeze for the edk2-stable201811 tag.

Cc: Anthony Perard <anthony.perard@citrix.com>
Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Julien Grall <julien.grall@linaro.org>
Cc: Philippe Mathieu-Daudé <philmd@redhat.com>
Cc: yuchenlin <yuchenlin@synology.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1319
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: yuchenlin <yuchenlin@synology.com>
2018-11-12 12:24:00 +01:00
98449a1678 Reapply "OvmfPkg: VMWare SVGA display device register definitions"
This reverts commit 328409ce8d, reapplying
9bcca53fe4.

Note that the commit now being reverted is technically correct; the only
reason we're reverting it is because it should not have been pushed past
the Soft Feature Freeze for the edk2-stable201811 tag.

Cc: Anthony Perard <anthony.perard@citrix.com>
Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Julien Grall <julien.grall@linaro.org>
Cc: Philippe Mathieu-Daudé <philmd@redhat.com>
Cc: yuchenlin <yuchenlin@synology.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1319
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: yuchenlin <yuchenlin@synology.com>
2018-11-12 12:23:44 +01:00
ade71c52a4 IntelFrameworkModulePkg: Fix UEFI and Tiano Decompression logic issue
https://bugzilla.tianocore.org/show_bug.cgi?id=1317

This is a regression issue caused by 684db6da64.
In Decode() function, once mOutBuf is fully filled, Decode() should return.
Current logic misses the checker of mOutBuf after while() loop.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-11-11 11:48:37 +08:00
1c4cecc9fd MdePkg BaseUefiDecompressLib: Fix UEFI Decompression logic issue
https://bugzilla.tianocore.org/show_bug.cgi?id=1317

This is a regression issue caused by 2ec7953d49.
In Decode() function, once mOutBuf is fully filled, Decode() should return.
Current logic misses the checker of mOutBuf after while() loop.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
Cc: Michael Kinney <michael.d.kinney@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-11-11 11:48:34 +08:00
5e45a1fdcf BaseTools: Fix UEFI and Tiano Decompression logic issue
https://bugzilla.tianocore.org/show_bug.cgi?id=1317

This is a regression issue caused by 041d89bc0f.
In Decode() function, once mOutBuf is fully filled, Decode() should return.
Current logic misses the checker of mOutBuf after while() loop.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-11-11 11:48:31 +08:00
114a75ad86 Revert "MdeModulePkg/DisplayEngine: Remove useless NULL ptr check for NewPos"
This reverts commit 8cd4e734cc.

It is not a real bug fix. It should not be pushed after
Hard Feature Freeze for edk2-stable201811 tag.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
Cc: Dandan Bi <dandan.bi@intel.com>
Reviewed-by: Bi Dandan <dandan.bi@intel.com>
2018-11-11 11:42:36 +08:00
e9ab1635a2 MdePkg: Fix incorrect check for DisplayOnly text format in AcpiEx
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1312

Text format for AcpiEx device path in UEFI Spec:
AcpiEx(HID,CID,UID,HIDSTR,CIDSTR,UIDSTR)
AcpiEx(HID|HIDSTR,(CID|CIDSTR,UID|UIDSTR))(Display Only)

When convert device path to text for AcpiEx device path,
current code check AllowShortcuts parameter to convert
the device path to DisplayOnly text format(shorter text
representation) by mistake.
It should check DisplayOnly parameter.

This commit is to fix this issue.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Dandan Bi <dandan.bi@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-11-11 11:41:04 +08:00
9bae7811d9 UefiCpuPkg/PiSmmCpuDxeSmm: Separate semaphore container.
In current implementation, core and package level sync uses same semaphores.
Sharing the semaphore may cause wrong execution order.
For example:
1. Feature A has CPU_FEATURE_CORE_BEFORE dependency with Feature B.
2. Feature C has CPU_FEATURE_PACKAGE_AFTER dependency with Feature B.
The expected feature initialization order is A B C:
A ---- (Core Depends) ----> B ---- (Package Depends) ----> C

For a CPU has 1 package, 2 cores and 4 threads. The feature initialization
order may like below:

   Thread#1             Thread#2       Thread#3         Thread#4
   [A.Init]             [A.Init]                        [A.Init]
Release(S1, S2)        Release(S1, S2)                Release(S3, S4)
Wait(S1) * 2           Wait(S2) * 2  <------------------------------- Core sync

   [B.Init]             [B.Init]
Release (S1,S2,S3,S4)
Wait (S1) * 4  <----------------------------------------------------- Package sync
                                                      Wait(S4 * 2) <- Core sync
                                                        [B.Init]

In above case, for thread#4, when it syncs in core level, Wait(S4) * 2 isn't
blocked and [B.Init] runs. But [A.Init] hasn't run in thread#3. It's wrong!
Thread#4 should execute [B.Init] after thread#3 executes [A.Init] because B
core level depends on A.

The reason of the wrong execution order is that S4 is released in thread#1
by calling Release (S1, S2, S3, S4) and in thread #4 by calling
Release (S3, S4).

To fix this issue, core level sync and package level sync should use separate
semaphores.

In above example, the S4 released in Release (S1, S2, S3, S4) should not be the
same semaphore as that in Release (S3, S4).

Related BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1311

Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
2018-11-11 10:02:47 +08:00
91ff5bba02 UefiCpuPkg/RegisterCpuFeaturesLib: Separate semaphore container.
In current implementation, core and package level sync uses same semaphores.
Sharing the semaphore may cause wrong execution order.
For example:
1. Feature A has CPU_FEATURE_CORE_BEFORE dependency with Feature B.
2. Feature C has CPU_FEATURE_PACKAGE_AFTER dependency with Feature B.
The expected feature initialization order is A B C:
A ---- (Core Depends) ----> B ---- (Package Depends) ----> C

For a CPU has 1 package, 2 cores and 4 threads. The feature initialization
order may like below:

   Thread#1             Thread#2       Thread#3         Thread#4
   [A.Init]             [A.Init]                        [A.Init]
Release(S1, S2)        Release(S1, S2)                Release(S3, S4)
Wait(S1) * 2           Wait(S2) * 2  <------------------------------- Core sync

   [B.Init]             [B.Init]
Release (S1,S2,S3,S4)
Wait (S1) * 4  <----------------------------------------------------- Package sync
                                                      Wait(S4 * 2) <- Core sync
                                                        [B.Init]

In above case, for thread#4, when it syncs in core level, Wait(S4) * 2 isn't
blocked and [B.Init] runs. But [A.Init] hasn't run in thread#3. It's wrong!
Thread#4 should execute [B.Init] after thread#3 executes [A.Init] because B
core level depends on A.

The reason of the wrong execution order is that S4 is released in thread#1
by calling Release (S1, S2, S3, S4) and in thread #4 by calling
Release (S3, S4).

To fix this issue, core level sync and package level sync should use separate
semaphores.

In above example, the S4 released in Release (S1, S2, S3, S4) should not be the
same semaphore as that in Release (S3, S4).

Related BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1311

Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
2018-11-11 10:02:45 +08:00
c1528b855c UefiCpuPkg/RegisterCpuFeaturesLib: Adjust Order.
V2 changes:
V1 change has regression which caused by change feature order.
V2 changes logic to detect dependence not only for the
neighborhood features. It need to check all features in the list.

V1 Changes:
In current code logic, only adjust feature position if current
CPU feature position not follow the request order. Just like
Feature A need to be executed before feature B, but current
feature A registers after feature B. So code will adjust the
position for feature A, move it to just before feature B. If
the position already met the requirement, code will not adjust
the position.

This logic has issue when met all below cases:
1. feature A has core or package level dependence with feature B.
2. feature A is register before feature B.
3. Also exist other features exist between feature A and B.

Root cause is driver ignores the dependence for this case, so
threads may execute not follow the dependence order.

Fix this issue by change code logic to adjust feature position
for CPU features which has dependence relationship.

Related BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1311

Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Ruiyu Ni <Ruiyu.ni@intel.com>
2018-11-11 10:02:43 +08:00
8cd4e734cc MdeModulePkg/DisplayEngine: Remove useless NULL ptr check for NewPos
In function UiDisplayMenu, the NewPos ptr which used to point to the
highlight menu entry. It will always point to the menu entry which
need to be highlighted or the gMenuOption menu if the highlight menu
is not found.
So we can remove the NULL ptr check for NewPos in this function.
And add the ASSERT code to avoid if any false positive reports
of NULL pointer dereference issue raised from static analysis.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Cc: Star Zeng <star.zeng@intel.com>
Cc: Hao Wu <hao.a.wu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Dandan Bi <dandan.bi@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-11-08 13:40:08 +08:00
3eacb4751a SourceLevelDebugPkg/Readme.txt: Update UDK Debugger Tool & manual URL
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=1289

The commit updates the link to get access to the Intel(R) UDK Debugger
Tool and its detailed user manual.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-11-08 12:00:34 +08:00
b46de50913 ShellPkg/ShellLib: Fix potential NULL deference issue
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1310

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Jim Dailey <jim_dailey@dell.com>
Reviewed-by: Hao A Wu <hao.a.wu@intel.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2018-11-08 10:25:53 +08:00
e6459b9e6c MdePkg/BaseSynchronizationLib: Fix InternalSync[De|In]crement
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1303

Today's code generates assembly code as below for
InternalSyncIncrement:
  __asm__ __volatile__ (
    "movl    $1, %%eax  \n\t"
    "lock               \n\t"
    "xadd    %%eax, %1  \n\t"
    "inc     %%eax      \n\t"
    : "=a" (Result),          // %0
      "+m" (*Value)           // %1
    :                         // no inputs that aren't also outputs
    : "memory",
      "cc"
    );

 0:       55      pushl   %ebp
 1:       89 e5   movl    %esp, %ebp
 3:       8b 45 08        movl    8(%ebp), %eax
 6:       b8 01 00 00 00  movl    $1, %eax
 b:       f0      lock
 c:       0f c1 00        xaddl   %eax, _InternalSyncIncrement(%eax)
 f:       40      incl    %eax
10:       5d      popl    %ebp
11:       c3      retl

Line #3 and Line #6 both use EAX as destination register.
Line #c uses EAX and (EAX).

The output operand "=a" tells GCC that EAX is used for output.
But GCC only assumes that EAX will be used in the very last
instruction.

Per GCC document,
"Use the '&' constraint modifier on all output operands that must
 not  overlap an input. Otherwise, GCC may allocate the output
 operand in the same register as an unrelated input operand, on
 the assumption that the assembler code consumes its inputs before
 producing outputs. This assumption may be false if the assembler
 code actually consists of more than one instruction."

"=&a" should be used to tell GCC not use EAX before the assembly.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Michael Kinney <michael.d.kinney@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Philippe Mathieu-Daudé <philmd@redhat.com>
Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Suggested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Fixes: 8a94eb9283
Fixes: 17634d026f
2018-11-08 10:00:05 +08:00
c60d36b4d1 UefiCpuPkg/SmmCpu: Block access-out only when static paging is used
When static paging is disabled, page table for below 4GB is created
and page table for above 4GB is created dynamically in page fault
handler.

Today's implementation only allow SMM access-out to below types of
memory address no matter static paging is enabled or not:
1. Reserved, run time and ACPI NVS type
2. MMIO

But certain platform feature like RAS may need to access other types
of memory from SMM. Today's code blocks these platforms.
This patch simplifies the policy to only block when static paging
is used so that the static paging can be disabled in these platforms
to meet their SMM access-out need.

Setting PcdCpuSmmStaticPageTable to FALSE can disable the static
paging.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jiewen Yao <jiewen.yao@intel.com>
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Reviewed-by: Jiewen Yao <jiewen.yao@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
2018-11-08 00:13:26 +08:00
b66e38b501 ArmPkg/ArmGicDxe: Fix GICv3 interrupt routing mode bug
Setting GICD_IROUTERn.IRM and GICD_IROUTERn.{Aff3,Aff2,Aff1,Aff0}
at the same time is nonsensical (see 8.9.13 in the GICv3 spec, which
says of GICD_IROUTERn.IRM that "When this bit is set to 1,
GICD_IROUTER<n>.{Aff3,Aff2,Aff1,Aff0} are UNKNOWN"). There is also no
guarantee that IRM is implemented (see GICD_TYPER.No1N which indicates
whether the implementation supports this or not).

Let's thus not set this bit, as we want all SPIs to be delivered to the
same CPU, and not be broadcast to all of them.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ming Huang <ming.huang@linaro.org>
Acked-by: Marc Zyngier <marc.zyngier@arm.com>
2018-11-07 16:19:22 +01:00
0adc6eae94 MdeModulePkg/Core: fix ineffective guard page issue
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1295

This issue originates from following patch which allows to enable
paging if PcdImageProtectionPolicy and PcdDxeNxMemoryProtectionPolicy
(in addition to PcdSetNxForStack) are set to enable related features.

  5267926134

Due to above change, PcdImageProtectionPolicy will be set to 0 by
default in many platforms, which, in turn, cause following code in
MdeModulePkg\Core\Dxe\Misc\MemoryProtection.c fail the creation of
notify event of CpuArchProtocol.

1138:  if (mImageProtectionPolicy != 0 ||
           PcdGet64 (PcdDxeNxMemoryProtectionPolicy) != 0) {
1139:  Status = CoreCreateEvent (
...
1142:             MemoryProtectionCpuArchProtocolNotify,
...
1145:             );

Then following call flow won't be done and Guard pages will not be
set as not-present in SetAllGuardPages() eventually.

   MemoryProtectionCpuArchProtocolNotify()
=> HeapGuardCpuArchProtocolNotify()
=> SetAllGuardPages()

The solution is removing the if(...) statement so that the notify
event will always be created and registered. This won't cause
unnecessary code execution because, in the notify event handler,
the related PCDs like

    PcdImageProtectionPolicy and
    PcdDxeNxMemoryProtectionPolicy

will be checked again before doing related jobs.

Cc: Star Zeng <star.zeng@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Leif Lindholm <leif.lindholm@linaro.org>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jian J Wang <jian.j.wang@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-11-07 23:08:27 +08:00
8a7ff892ca MdeModulePkg/Core: fill logic hole in MemoryProtectionCpuArchProtocolNotify
At the end of of MemoryProtectionCpuArchProtocolNotify there's cleanup
code to free resource. But at line 978, 994, 1005 the function returns
directly. This patch use "goto" to replace "return" to make sure the
resource is freed before exit.

1029:  CoreCloseEvent (Event);
1030:  return;

There's another memory leak after calling gBS->LocateHandleBuffer() in
the same function:

  Status = gBS->LocateHandleBuffer (
                  ByProtocol,
                  &gEfiLoadedImageProtocolGuid,
                  NULL,
                  &NoHandles,
                  &HandleBuffer
                  );

HandleBuffer is allocated in above call but never freed. This patch
will also add code to free it.

Cc: Star Zeng <star.zeng@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Leif Lindholm <leif.lindholm@linaro.org>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jian J Wang <jian.j.wang@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-11-07 23:08:26 +08:00
d98fc9adfb Revert "XhciDxe: Use common buffer for AsyncInterruptTransfer"
There is concern at the thread
https://lists.01.org/pipermail/edk2-devel/2018-November/031951.html.
And the time point is a little sensitive as it is near edk2-stable201811.

This reverts commit 7779209971.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-11-07 22:48:07 +08:00
f87db25620 Revert "EhciDxe: Use common buffer for AsyncInterruptTransfer"
There is concern at the thread
https://lists.01.org/pipermail/edk2-devel/2018-November/031951.html.
And the time point is a little sensitive as it is near edk2-stable201811.

This reverts commit 0cd6452503.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-11-07 22:48:01 +08:00
1ed6498c4a UefiCpuPkg/CommonFeature: Skip locking when the feature is disabled
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1305

Today's code unconditionally sets the IA32_FEATURE_CONTROL.Lock to 1
no matter the feature is enabled or not.

The patch fixes this issue by only setting the Lock bit to 1 when
the feature is enabled.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Eric Dong <eric.dong@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
2018-11-07 17:05:49 +08:00
328409ce8d Revert "OvmfPkg: VMWare SVGA display device register definitions"
This reverts commit 9bcca53fe4.

We reverted VMWare SVGA driver. We don't need these definitions too.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: yuchenlin <yuchenlin@synology.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Regression-tested-by: Laszlo Ersek <lersek@redhat.com>
2018-11-06 12:56:22 +01:00
438ada5aa5 Revert "OvmfPkg/QemuVideoDxe: Helper functions for unaligned port I/O."
This reverts commit 05a5379458.

The VMWare SVGA display device implemented by Qemu (-vga vmware) uses
an I/O-type BAR which is laid out such that some register offsets are
not aligned to the read/write width with which they are expected to be
accessed. However, we reverted the initialization of VMWare SVGA device,
we don't need such unaligned I/O.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: yuchenlin <yuchenlin@synology.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Regression-tested-by: Laszlo Ersek <lersek@redhat.com>
2018-11-06 12:56:20 +01:00
98856a724c Revert "OvmfPkg/QemuVideoDxe: VMWare SVGA device support"
This reverts commit c137d95081.

The VMWare SVGA model now -- since commit 104bd1dc70 in QEMU --
falls back to stdvga (that is, Bochs) if we don't setup VMWare SVGA
FIFO.

To simplify QemuVideoDxe, we don't intend to implement the VMWare SVGA
FIFO setup feature. It means our current VMW SVGA driver code is
basically dead. To simplify the problem, we will replace the old
VMWare SVGA driver to Bochs interface. It should work on all QEMU
version.

The first step for using Bochs interface is to revert old driver.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: yuchenlin <yuchenlin@synology.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Regression-tested-by: Laszlo Ersek <lersek@redhat.com>
2018-11-06 12:56:17 +01:00
e038bde267 Revert "OvmfPkg/QemuVideoDxe: list "UnalignedIoInternal.h" in the INF file"
This reverts commit b2959e9f1a.

The VMWare SVGA display device implemented by Qemu (-vga vmware) uses
an I/O-type BAR which is laid out such that some register offsets are
not aligned to the read/write width with which they are expected to be
accessed. However, we will revert the initialization of VMWare SVGA
device later, we don't need such unaligned I/O.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: yuchenlin <yuchenlin@synology.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Regression-tested-by: Laszlo Ersek <lersek@redhat.com>
2018-11-06 12:56:03 +01:00
62ea70e312 ShellPkg/TftpDynamicCommand: Clarify the retry count option in command.
[-c <retry count>] is to define the number of times to transmit request
packets and wait for a response. The default value is 6. But it doesn't
specify the behavior of zero value. Here, The patch is to clear that:
Set to zero also means to use the default value.

Cc: Carsey Jaben <jaben.carsey@intel.com>
Cc: Ye Ting <ting.ye@intel.com>
Cc: Fu Siyuan <siyuan.fu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Wu Jiaxin <jiaxin.wu@intel.com>
Reviewed-by: Fu Siyuan <siyuan.fu@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2018-11-06 15:23:38 +08:00
fc5b790b39 MdeModulePkg/Mtftp4Dxe: Fix invalid configuration of MTFTP local port.
This patch is to fix the invalid setting of MTFTP local port. The
issue can be reproduced by tftp shell command by using [-l port]
option.

Cc: Ye Ting <ting.ye@intel.com>
Cc: Fu Siyuan <siyuan.fu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Wu Jiaxin <jiaxin.wu@intel.com>
Reviewed-by: Fu Siyuan <siyuan.fu@intel.com>
2018-11-06 15:22:35 +08:00
d2f1f6423b OvmfPkg: Replace obsoleted network drivers from platform DSC/FDF.
V2:
Add missed library instance for NetworkPkg iSCSI driver.

This patch replaces the MdeModulePkg TCP, PXE and iSCSI driver with those
ones in NetworkPkg. These 3 drivers in MdeModulePkg are not being actively
maintained and will be removed from edk2 master soon.

Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Cc: Anthony Perard <anthony.perard@citrix.com>
Cc: Julien Grall <julien.grall@linaro.org>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Fu Siyuan <siyuan.fu@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
2018-11-06 09:07:30 +08:00
5f1371270e Vlv2TbltDevicePkg: Replace obsoleted drivers from platform DSC/FDF.
V2:
Add missed library instance for NetworkPkg iSCSI driver.

This patch replaces the MdeModulePkg TCP, PXE and iSCSI driver with those
ones in NetworkPkg. These 3 drivers in MdeModulePkg are not being actively
maintained and will be removed from edk2 master soon.

Cc: David Wei <david.wei@intel.com>
Cc: Mang Guo <mang.guo@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Fu Siyuan <siyuan.fu@intel.com>
Reviewed-by: David Wei <david.wei@intel.com>
2018-11-06 09:06:12 +08:00
5ae3184d8c Maintainer.txt: Add Chao to be co-maintainer of SignedCapsulePkg
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhang, Chao B <chao.b.zhang@intel.com>
Reviewed-by: Jiewen Yao <jiewen.yao@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
2018-11-05 23:07:15 +08:00
e40f8efb8b ArmPkg/OpteeLib: Fix compilation issues for ARM (32-bit)
Correct usage of EFI_PHYSICAL_ADDRESS (always 64-bit) vs. pointers
(depending on architecture).

Reported-by: Leif Lindholm <leif.lindholm@linaro.org>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Sumit Garg <sumit.garg@linaro.org>
Reviewed-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
2018-11-05 12:04:35 +01:00
6cf3aeaeb1 EmulatorPkg: Replace obsoleted network drivers from platform DSC/FDF.
This patch replaces the MdeModulePkg TCP, PXE and iSCSI driver with those
ones in NetworkPkg. These 3 drivers in MdeModulePkg are not being actively
maintained and will be removed from edk2 master soon.

Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Andrew Fish <afish@apple.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Fu Siyuan <siyuan.fu@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-11-05 16:02:24 +08:00
2eee9d6581 Nt32Pkg: Replace obsoleted network drivers from NT32 platform DSC/FDF.
This patch replaces the MdeModulePkg TCP, PXE and iSCSI driver with those
ones in NetworkPkg. These 3 drivers in MdeModulePkg are not being actively
maintained and will be removed from edk2 master soon.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Hao Wu <hao.a.wu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Fu Siyuan <siyuan.fu@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-11-05 16:01:54 +08:00
806c28ef50 CorebootPayloadPkg: Remove EdkShellBinPkg in FDF
Remove EdkShellBinPkg in CorebootPayloadPkg.fdf.
https://bugzilla.tianocore.org/show_bug.cgi?id=1108

v3:Remove FULL_BIN and change SHELL_TYPE from FULL_BIN
to UEFI_BIN.

Cc: Maurice Ma <maurice.ma@intel.com>
Cc: Prince Agyeman <prince.agyeman@intel.com>
Cc: Benjamin You <benjamin.you@intel.com>
Change-Id: I4db7068a3a1f68a1f6303079b73dc548c9feb2e3
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
Reviewed-by: Benjamin You <benjamin.you@intel.com>
2018-11-05 13:58:24 +08:00
c95f600e93 BaseTools: Use common cc flag for building PcdValueInit.
V3:

Update the patch to avoid "reduce" function fail.

V2:
Support to extract the common cc flag from a
combined cc flag string. For example
MSFT:*_*_IA32_CC_FLAGS = /D DISABLE_NEW_DEPRECATED_INTERFACES /DDEF_IA32
MSFT:*_*_X64_CC_FLAGS  = /DDEF_X64 /D DISABLE_NEW_DEPRECATED_INTERFACES

V1:
Use common cc flags for building PcdValueInit. The common
cc flags include the cc flag which is under common arch and
under all build arches.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-11-05 11:02:46 +08:00
fc5217a999 BaseTools build_rule.template: Update aslc rule for XCODE tool chain
Update aslc rule to rename the temp output file from .efi to .pecoff.
This change can avoid the conflict .efi file name in output directory.
One is the driver image, another is aslc temp output file.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-11-05 11:02:46 +08:00
764c21f4aa ShellPkg-ShellApp: Provide fully-qualified path to shell scripts
Provide fully-qualified path to shell scripts

Section 3.6.2 of version 2.2 of the shell specification requires that
the first positional argument (i.e. arg 0) of a shell script resolves
to "the full path name of the script itself."

Ensure that the startup script and any scripts launched by the shell
meet this requirement.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jim Dailey <jim_dailey@dell.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-11-05 10:52:51 +08:00
9ba404d664 ShellPkg-UefiShellLib: Add a function to fully-qualify paths
Add a function to return a clean, fully-qualified version of some path.

This function handles a (possibly "dirty") input path that may or may
not include a file system reference.

If it does not include a file system reference, then if the input path
does not begin with a forward or backward slash, then the input path is
relative to the current working directory of the current file system.
Otherwise, it is an absolute path within the current file system.

If it does include a file system reference, it may be a reference to the
current or some other file system.  If the file system reference is not
immediately followed by a forward or backward slash, then the input path
is relative to the current working directory of the given file system.
Otherwise, it is an absolute path within the given file system.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jim Dailey <jim_dailey@dell.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-11-05 10:52:40 +08:00
e048ce883c UefiCpuPkg/MpInitLib: Rollback old change 2a5997f8.
In some special cases, after BSP sends Init-sipi-sipi signal
AP needs more time to start the Ap procedure. In this case
BSP may think AP has finished its task but in fact AP hasn't began
yet.

Rollback former change to keep the status which only be used
when AP really finished task.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-11-05 10:39:13 +08:00
2552661848 Maintainers.txt: Change package maintainer of UefiCpuPkg
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Eric Dong <eric.dong@intel.com>
2018-11-05 10:26:07 +08:00
23d14cae07 Maintainers.txt: Change package maintainer of IntelSiliconPkg
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Jiewen Yao <jiewen.yao@intel.com>
2018-11-05 10:26:05 +08:00
93f9898582 BaseTools: Check the max size for string PCD.
According to PCD_DATABASE_INIT in
edk2\MdeModulePkg\Include\Guid\PcdDataBaseSignatureGuid.h,
the max size for string PCD should not exceed USHRT_MAX 65535(0xffff).

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: ZhiqiangX Zhao <zhiqiangx.zhao@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2018-11-02 22:39:08 +08:00
498cb29dfd MdePkg/Base.h: Implement BASE_CR() via OFFSET_OF().
Replace the current NULL pointer dereference to retrieve Field's
offset with a call to OFFSET_OF().  This is implemented via
__builtin_offsetof for GCC and Clang, which eliminates UB caught by
Clang UndefinedBehaviorSanitizer.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Marvin Haeuser <Marvin.Haeuser@outlook.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-11-02 22:15:54 +08:00
88e8498f8a BaseTools tools_def.template: Add GCC link script option in ASLDLINK_FLAGS
GCC link script is used to discard the unused section data from ELF image.
ASLDLINK_FLAGS requires it to remove the unnecessary section data, then
GenFw can be used to retrieve the correct data section from ELF image.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-11-02 22:15:54 +08:00
5af5153ad9 BaseTools: Add checking to EFI variable attribute.
This patch is going to add a check that only if the
attribute of a EFI variable include 'NV', it will be
added into PcdNvStoreDefaultValueBuffer.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-11-02 22:15:53 +08:00
dbeaf58555 MdeModulePkg: Remove the trailing space in three source files.
The recent changes in these three source files introduce the trailing space.
This patch removes them to follow edk2 coding style.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
Cc: Star Zeng <star.zeng@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-11-02 22:15:52 +08:00
c4f4984c69 BaseTools/Eot: Remove a duplication code in EotMain class
Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Hess Chen <hesheng.chen@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-11-02 11:26:32 +08:00
d3d97b378f BaseTools: Add special handle for '\' use in Pcd Value
V2: Follow PEP8 to not multiples import on one line

Case:
gEfiOzmosisPkgTokenSpaceGuid.PcdBootLogFolderPath|L"\\Logs\\"|VOID*|12

Fixes: https://bugzilla.tianocore.org/show_bug.cgi?id=1287
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2018-11-01 10:27:13 +08:00
5b9639e697 BaseTools: Fix merge in FDF parser for OptionROM override
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Tomas Pilar <tpilar@solarflare.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-11-01 10:23:23 +08:00
bfc87aa78e MdeModulePkg/PiSmmIpl: Do not reset SMRAM to UC when CPU driver runs
Today's PiSmmIpl implementation initially sets SMRAM to WB to speed
up the SMM core/modules loading before SMM CPU driver runs.
When SMM CPU driver runs, PiSmmIpl resets the SMRAM to UC. It's done
in SmmIplDxeDispatchEventNotify(). COMM_BUFFER_SMM_DISPATCH_RESTART
is returned from SMM core that SMM CPU driver is just dispatched.

Since now the SMRR is widely used to control the SMRAM cache setting.
It's not needed to reset the SMRAM to UC anymore.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Jiewen Yao <jiewen.yao@intel.com>
Cc: Michael Kinney <michael.d.kinney@intel.com>
2018-11-01 10:06:30 +08:00
fd95015673 NetworkPkg/TlsDxe: Fix failure to process multiple TLS records.
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1290.

Current implementation failed to parse the multiple TLS record
messages due to the incorrect pointer of TLS record header. This
patch is to resolve that problem.

Cc: Ye Ting <ting.ye@intel.com>
Cc: Fu Siyuan <siyuan.fu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Wu Jiaxin <jiaxin.wu@intel.com>
Reviewed-by: Fu Siyuan <siyuan.fu@intel.com>
2018-11-01 09:37:41 +08:00
c41da21704 NetworkPkg/Mtftp6Dxe: Remove the trailing white spaces.
Cc: Fu Siyuan <siyuan.fu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Wu Jiaxin <jiaxin.wu@intel.com>
Reviewed-by: Fu Siyuan <siyuan.fu@intel.com>
2018-11-01 09:37:34 +08:00
e23366d3c3 FmpDevicePkg DSC: Add NOOPT target to verify NOOPT target
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
Cc: Star Zeng <star.zeng@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-10-31 22:08:10 +08:00
ce283fd609 BaseTools ConvertFceToStructurePcd: Fix the array value with empty string
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-10-31 22:07:57 +08:00
269f3b5180 CryptoPkg/BaseCryptLib: Fix potential integer overflow issue.
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1275

The LookupFreeMemRegion() in RuntimeMemAllocate.c is used to look-up
free memory region for runtime resource allocation, which was designed
to support runtime authenticated variable service.
The ReqPages in this function is the required pages to be allocated,
which depends on the malloc() call in internal OpenSSL routines. The
direct offset subtractions on ReqPages may bring possible integer
overflow issue.

This patch is to add the extra parameter checks to remove this possible
overflow risk.

Cc: Ye Ting <ting.ye@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Long Qin <qin.long@intel.com>
Reviewed-by: Ye Ting <ting.ye@intel.com>
2018-10-31 11:07:53 +08:00
beabfd5800 UefiCpuPkg/RegisterCpuFeaturesLib: Fix build failure.
Build UefiCpuPkg with below configuration:
Architecture(s)  = IA32
Build target     = NOOPT
Toolchain        = VS2015x86

Below error info shows up:
DxeRegisterCpuFeaturesLib.lib(CpuFeaturesInitialize.obj) :
error LNK2001: unresolved external symbol __allmul

Valid mDependTypeStr type only have 5 items, use UINT32 type cast
to fix this error.

Cc: Dandan Bi <dandan.bi@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-10-31 09:43:41 +08:00
27b9cb33e7 MdeModulePkg/UdfDxe: Memory free/use after free in ResolveSymlink()
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=1279

For function ResolveSymlink(), the below codes:

    if (CompareMem ((VOID *)&PreviousFile, (VOID *)Parent,
                    sizeof (UDF_FILE_INFO)) != 0) {
      CleanupFileInformation (&PreviousFile);
    }

    CopyMem ((VOID *)&PreviousFile, (VOID *)File, sizeof (UDF_FILE_INFO));

If the contents in 'PreviousFile' and 'File' are the same, call to
"CleanupFileInformation (&PreviousFile);" will free the buffers in 'File'
as well. This will lead to potential memory double free/use after free
issues.

This commit will add additional check to address the above issue.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Paulo Alcantara <palcantara@suse.de>
Reviewed-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Leif Lindholm <leif.lindholm@linaro.org>
2018-10-31 08:57:04 +08:00
bfb8c64cbf MdeModulePkg/UdfDxe: Content check for 'File' in ResolveSymlink()
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=1279

The content within 'File' is the output data for ResolveSymlink(). This
commit will add checks to ensure the content in 'File' is valid.
Otherwise, possible null pointer dereference issue will occur during the
subsequent usage of the data returned by ResolveSymlink().

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Paulo Alcantara <palcantara@suse.de>
Reviewed-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Leif Lindholm <leif.lindholm@linaro.org>
2018-10-31 08:57:04 +08:00
542b03d0a0 MdeModulePkg/UdfDxe: Check 'Component Type' within a Path Component
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=1279

According to the ECMA-167 standard (3rd Edition - June 1997), Section
14.16.1.1, valid values are 1 to 5. All other values will be treated as a
corrupted volume.

This commit will add such check within function ResolveSymlink().

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Paulo Alcantara <palcantara@suse.de>
Reviewed-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Leif Lindholm <leif.lindholm@linaro.org>
2018-10-31 08:57:04 +08:00
a0f01e8a22 ArmPkg/OpteeLib: Add APIs to communicate with OP-TEE
Add following APIs to communicate with OP-TEE pseudo/early TAs:
1. OpteeInit
2. OpteeOpenSession
3. OpteeCloseSession
4. OpteeInvokeFunc

Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Cc: Leif Lindholm <leif.lindholm@linaro.org>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Sumit Garg <sumit.garg@linaro.org>
Reviewed-by: Leif Lindholm <leif.lindholm@linaro.org>
2018-10-30 14:21:45 +00:00
4222e8e7e4 UefiCpuPkg/PiSmmCpuDxeSmm: Fix ASSERT for success.
Index is initialized to MAX_UINT16 as default failure value, which
is what the ASSERT is supposed to test for.  The ASSERT condition
however can never return FALSE for INT16 != int, as due to
Integer Promotion[1], Index is converted to int, which can never
result in -1.

Furthermore, Index is used as a for loop index variable inbetween its
initialization and the ASSERT, so the value is unconditionally
overwritten too.

Fix the ASSERT check to compare Index to its upper boundary, which it
will be equal to if the loop was not broken out of on success.

[1] ISO/IEC 9899:2011, 6.5.9.4

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Marvin Haeuser <Marvin.Haeuser@outlook.com>
Reviewed-by: Eric Dong <eric.dong@intel.com>
2018-10-30 10:21:33 +08:00
37fba7c276 UefiCpuPkg/MpInitLib: Fix ASSERT for success.
Index is initialized to MAX_UINT16 as default failure value, which
is what the ASSERT is supposed to test for.  The ASSERT condition
however can never return FALSE for INT16 != int, as due to
Integer Promotion[1], Index is converted to int, which can never
result in -1.

Furthermore, Index is used as a for loop index variable inbetween its
initialization and the ASSERT, so the value is unconditionally
overwritten too.

Fix the ASSERT check to compare Index to its upper boundary, which it
will be equal to if the loop was not broken out of on success.

[1] ISO/IEC 9899:2011, 6.5.9.4

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Marvin Haeuser <Marvin.Haeuser@outlook.com>
Reviewed-by: Eric Dong <eric.dong@intel.com>
2018-10-30 10:21:29 +08:00
61a62fc258 MdeModulePkg/Core: fix an issue of potential NULL pointer access
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1286

This issue is introduced by bb685071c2.

The *MemorySpaceMap assigned with NULL (line 1710) value might be
accessed (line 1726/1730) without any sanity check. Although it won't
happen in practice because of line 1722, we still need to add check
against NULL to make static code analyzer happy.

1710  *MemorySpaceMap       = NULL;
....  ...
1722  if (DescriptorCount == *NumberOfDescriptors) {
....  ...
1726    Descriptor = *MemorySpaceMap;
....  ...
1730        BuildMemoryDescriptor (Descriptor, Entry);

Tests:
  Pass build and boot to shell.

Cc: Hao Wu <hao.a.wu@intel.com>
Cc: Star Zeng <star.zeng@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jian J Wang <jian.j.wang@intel.com>
Reviewed-by: Hao Wu <hao.a.wu@intel.com>
2018-10-30 10:16:03 +08:00
c87ac38cf2 MdeModulePkg/UsbBusDxe: Add missing "return NULL" in UsbCreateDesc()
When (Len < Offset) is TRUE, indicating the data to visit is beyond
the boundary, the error message is printed but the function doesn't
return NULL.

It's a typo when modifying the commit 4c034bf62.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-10-30 09:17:40 +08:00
c09b254bdc IntelFsp2Pkg: Fixed potentially NULL pointer accessing
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1280

When copying IDT table in SecMain, the pointer might be
NULL so added the check to fix it.

Test: Verified on internal platform and boots successfully.

Cc: Jiewen Yao <Jiewen.yao@intel.com>
Cc: Desimone Nathaniel L <nathaniel.l.desimone@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Chasel Chiu <chasel.chiu@intel.com>
2018-10-29 11:24:58 +08:00
70425456da MdeModulePkg/UsbBusPei: Reject descriptor whose length is bad
Today's implementation doesn't check whether the length of
descriptor is valid before using it.

The patch fixes this issue by syncing the similar fix to UsbBusDxe.
70c3c2370a
*MdeModulePkg/UsbBus: Reject descriptor whose length is bad

Additionally the patch also rejects the data when length is
larger than sizeof (PeiUsbDevice->ConfigurationData).

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
2018-10-29 11:11:03 +08:00
c96de1dbae MdeModulePkg/UsbBusPei: Fix out-of-bound read access to descriptors
Today's implementation reads the Type/Length field in the USB
descriptors data without checking whether the offset to read is
beyond the data boundary.

The patch fixes this issue by syncing the fix in commit
4c034bf62c
*MdeModulePkg/UsbBus: Fix out-of-bound read access to descriptors

ParsedBytes in UsbBusPei.GetExpectedDescriptor() is different from
Consumed in UsbBusDxe.UsbCreateDesc().
ParsedBytes is the offset of found descriptor while Consumed is
offset of next descriptor of found one.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
2018-10-29 11:11:02 +08:00
2f6693c283 NetworkPkg/Mtftp6Dxe: Correct the total received and saved block number.
The block returned from Mtftp6RemoveBlockNum is not the total received and
saved block number if it works in passive (Slave) mode.

The issue was exposed by the EMS test.

Cc: Ye Ting <ting.ye@intel.com>
Cc: Fu Siyuan <siyuan.fu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Wu Jiaxin <jiaxin.wu@intel.com>
Reviewed-by: Ye Ting <ting.ye@intel.com>
2018-10-29 08:31:18 +08:00
9202304c18 MdeModulePke/Mtftp4Dxe: Correct the total received and saved block number.
The block returned from Mtftp4RemoveBlockNum is not the total received and
saved block number if it works in passive (Slave) mode.

The issue was exposed by the EMS test.

Cc: Ye Ting <ting.ye@intel.com>
Cc: Fu Siyuan <siyuan.fu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Wu Jiaxin <jiaxin.wu@intel.com>
Reviewed-by: Ye Ting <ting.ye@intel.com>
2018-10-29 08:31:10 +08:00
0cd6452503 MdeModulePkg EhciDxe: Use common buffer for AsyncInterruptTransfer
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1274

In current code, EhcMonitorAsyncRequests (timer handler) will do
unmap and map operations for AsyncIntTransfers to "Flush data from
PCI controller specific address to mapped system memory address".
EhcMonitorAsyncRequests
  EhcFlushAsyncIntMap
    PciIo->Unmap
      IoMmu->SetAttribute
    PciIo->Map
      IoMmu->SetAttribute

This may impact the boot performance.

Since the data buffer for EhcMonitorAsyncRequests is internal
buffer, we can allocate common buffer by PciIo->AllocateBuffer
and map the buffer with EfiPciIoOperationBusMasterCommonBuffer,
then the unmap and map operations can be removed.

///
/// Provides both read and write access to system memory by
/// both the processor and a bus master. The buffer is coherent
/// from both the processor's and the bus master's point of view.
///
EfiPciIoOperationBusMasterCommonBuffer,

Test done:
USB KB works normally.
USB disk read/write works normally.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Hao Wu <hao.a.wu@intel.com>
Cc: Jian J Wang <jian.j.wang@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Hao Wu <hao.a.wu@intel.com>
2018-10-28 21:20:08 +08:00
7779209971 MdeModulePkg XhciDxe: Use common buffer for AsyncInterruptTransfer
V3:
Call XhcFreeUrb after XhcCreateTransferTrb fails in XhcCreateTrb.

REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1274

In current code, XhcMonitorAsyncRequests (timer handler) will do
unmap and map operations for AsyncIntTransfers to "Flush data from
PCI controller specific address to mapped system memory address".
XhcMonitorAsyncRequests
  XhcFlushAsyncIntMap
    PciIo->Unmap
      IoMmu->SetAttribute
    PciIo->Map
      IoMmu->SetAttribute

This may impact the boot performance.

Since the data buffer for XhcMonitorAsyncRequests is internal
buffer, we can allocate common buffer by PciIo->AllocateBuffer
and map the buffer with EfiPciIoOperationBusMasterCommonBuffer,
then the unmap and map operations can be removed.

///
/// Provides both read and write access to system memory by
/// both the processor and a bus master. The buffer is coherent
/// from both the processor's and the bus master's point of view.
///
EfiPciIoOperationBusMasterCommonBuffer,

Test done:
USB KB works normally.
USB disk read/write works normally.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Hao Wu <hao.a.wu@intel.com>
Cc: Jian J Wang <jian.j.wang@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Hao Wu <hao.a.wu@intel.com>
2018-10-28 21:20:08 +08:00
4f792685e4 MdeModulePkg EhciDxe: Extract new EhciInsertAsyncIntTransfer function
V3:
Match function parameter name and description between
EhciSched.c and EhciSched.h.

V2:
Add the missing "gBS->FreePool (Data);".

REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1274

Extract new EhciInsertAsyncIntTransfer function from
EhcAsyncInterruptTransfer.

It is code preparation for following patch,
no essential functional change.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Hao Wu <hao.a.wu@intel.com>
Cc: Jian J Wang <jian.j.wang@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Hao Wu <hao.a.wu@intel.com>
2018-10-28 21:20:07 +08:00
6681582dcc MdeModulePkg XhciDxe: Extract new XhciInsertAsyncIntTransfer function
V3:
Match function parameter name and description between
XhciSched.c and XhciSched.h.

V2:
Add the missing "FreePool (Data);".
Remove the unnecessary indentation change.

REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1274

Extract new XhciInsertAsyncIntTransfer function from
XhcAsyncInterruptTransfer.

It is code preparation for following patch,
no essential functional change.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Hao Wu <hao.a.wu@intel.com>
Cc: Jian J Wang <jian.j.wang@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Hao Wu <hao.a.wu@intel.com>
2018-10-28 21:20:06 +08:00
a6eb94eedb UefiCpuPkg/RegisterCpuFeaturesLib: Fix GCC build failure.
Cc: Liming Gao <liming.gao@intel.com>
Cc: Dandan Bi <dandan.bi@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
2018-10-27 22:39:48 +08:00
895415ed47 MdeModulePkg/Core: correct one coding style
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1284

Non-Boolean comparisons should use a compare operator
(==, !=, >, < >=, <=)

Cc: Dandan Bi <dandan.bi@intel.com>
Cc: Star Zeng <star.zeng@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jian J Wang <jian.j.wang@intel.com>
Reviewed-by: Dandan Bi <dandan.bi@intel.com>
2018-10-27 21:33:13 +08:00
23a62c6990 BaseTools: Sync the DevicePath Function update from MdePkg
Update the AcpiExp, USBxxx, Sata, VENDOR_DEVICE_PATH.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-10-27 19:49:26 +08:00
7a283cb3f0 BaseTools: Rollback Filter out unused structure pcds
This reverts commit 51d17bb7b0.
commit 51d17bb7b0 adds new check
of Pcds in the platform unused library INF files.
It breaks the existing platform.
To avoid the impact, roll back this change first.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob C Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-27 15:28:27 +08:00
3da7d99506 MdeModulePkg RegularExpressionDxe: Remove unknown build option for XCODE
This patch makes RegularExpressionDxe pass XCODE5 build.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
Cc: Star Zeng <star.zeng@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-10-26 22:01:46 +08:00
f157f97301 BaseTools: Allow multiple PciDeviceId in Fdf OptionROM override
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Tomas Pilar <tpilar@solarflare.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-10-26 20:47:54 +08:00
b28d406b5a BaseTools: Fix BPDG tool print traceback info issue
Fix BPDG tool print traceback info issue
and remove abundant code

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-10-26 20:27:35 +08:00
e5001ab7a9 MdeModulePkg/Core: fix an IA32 build failure
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1277

The failure is caused by data type conversion between UINTN and UINT64,
which is checked in at 63ebde8ef6.

Cc: Star Zeng <star.zeng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jian J Wang <jian.j.wang@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-10-26 16:04:37 +08:00
979b7d802c MdeModulePkg/HiiDB: Make sure database update behaviors are atomic
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1235

When update contents in HiiDatabase, like:
1. Add/update/remove package list
2. Add/update string
3. Add/update image
We should make these operations atomic to prevent the
potential issue that the one update operation with
higher TPL may interrupt another.
This commit is to make the HiiDatabase update behaviors
atomic by adding EfiAcquireLock/EfiReleaseLock function.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Dandan Bi <dandan.bi@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-26 15:08:44 +08:00
adb2c05012 MdeModulePkg/HiiDB: Reorganize codes of exporting HII settings
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1235

Function "HiiGetConfigurationSetting" may be called
when HiiDatabase contents have been updated.
And it is just to call function "HiiGetDatabaseInfo" to
get the contents in HiiDatabase and call function
"HiiGetConfigRespInfo" and get the configuration response
string form HII drivers.

So here we can remove function "HiiGetConfigurationSetting"
and make caller to call "HiiGetDatabaseInfo" and
"HiiGetConfigRespInfo" directly.

And thus it also can distinguish which code blocks are to
operate HiiDatabase contents and which code blocks are not.
Then it's easy to know which code blocks should be atomic
when updating HiiDatabase contents.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Dandan Bi <dandan.bi@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-26 15:08:40 +08:00
599bb2be1f BaseTools: Roll back code modify by commit 9e47e6f908
Roll back code modify by commit 9e47e6f908,
if ForceRebase not False or True, the GenFv command not need add parameter

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
2018-10-26 14:13:47 +08:00
1475b83f06 UefiCpuPkg/RegisterCpuFeaturesLib: Support combo CPU feature style.
Current code assume only one dependence (before or after) for one
feature. Enhance code logic to support feature has two dependence
(before and after) type.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-10-26 12:39:49 +08:00
901ae29dfe IntelFsp2Pkg: Fix GCC49/XCODE build failure
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1276

Fixed potentially uninitialized variable build failure
caused by commit: b1cc6f672f

Test: Verified on internal platform and boots successfully.

Cc: Jiewen Yao <Jiewen.yao@intel.com>
Cc: Desimone Nathaniel L <nathaniel.l.desimone@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Chasel Chiu <chasel.chiu@intel.com>
Reviewed-by: Jiewen Yao <Jiewen.yao@intel.com>
2018-10-26 11:34:21 +08:00
7db4034f9a UefiCpuPkg/PiSmmCpuDxeSmm: Fix build failure for VS2012 and GCC49.
Code initialized in function can't be correctly detected by build tool.
Add code to clearly initialize the local variable before use it.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Dandan Bi <dandan.bi@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-10-26 11:28:37 +08:00
31fb333431 UefiCpuPkg/PiSmmCpuDxeSmm: Clean up useless code.
Remove useless code after change 93324390.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Dandan Bi <dandan.bi@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-10-26 11:28:37 +08:00
e07e3cebc7 UefiCpuPkg/PiSmmCpuDxeSmm: Remove white space at line end.
Remove extra white space at the end of line.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Dandan Bi <dandan.bi@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-10-26 11:28:36 +08:00
1c15179c8f UefiCpuPkg/RegisterCpuFeaturesLib: Fix ECC issues.
Changes include:
1. Remove extra white space at the end of line.
2. Add comments for the new add function parameter.
3. Update IN OUT tag for function parameter.

Cc: Dandan Bi <dandan.bi@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-10-26 11:28:36 +08:00
2b0c199465 UefiCpuPkg/CpuCommonFeaturesLib: Remove white space at line end.
Remove extra white space at the end of line.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Dandan Bi <dandan.bi@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-10-26 11:28:36 +08:00
692e318d9d UefiCpuPkg/RegisterCpuFeaturesLib: Fix build failure for VS2012 and GCC49.
Code initialized in function can't be correctly detected by build tool.
Add code to clearly initialize the local variable before use it.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Dandan Bi <dandan.bi@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-10-26 11:28:36 +08:00
8022aca8fd BaseTools:Not miss the full assign value of FixedAtBuild structure PCD
For structure PCD, if it is a FixedAtBuild PCD, the full assign value in
dsc file should not be missed when updating the structure PCD value.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: ZhiqiangX Zhao <zhiqiangx.zhao@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2018-10-26 10:49:49 +08:00
63ebde8ef6 MdeModulePkg/Core: add freed-memory guard feature
Freed-memory guard is used to detect UAF (Use-After-Free) memory issue
which is illegal access to memory which has been freed. The principle
behind is similar to pool guard feature, that is we'll turn all pool
memory allocation to page allocation and mark them to be not-present
once they are freed.

This also implies that, once a page is allocated and freed, it cannot
be re-allocated. This will bring another issue, which is that there's
risk that memory space will be used out. To address it, the memory
service add logic to put part (at most 64 pages a time) of freed pages
back into page pool, so that the memory service can still have memory
to allocate, when all memory space have been allocated once. This is
called memory promotion. The promoted pages are always from the eldest
pages which haven been freed.

This feature brings another problem is that memory map descriptors will
be increased enormously (200+ -> 2000+). One of change in this patch
is to update MergeMemoryMap() in file PropertiesTable.c to allow merge
freed pages back into the memory map. Now the number can stay at around
510.

Cc: Star Zeng <star.zeng@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jian J Wang <jian.j.wang@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-10-26 10:30:35 +08:00
bb685071c2 MdeModulePkg/Core: prevent re-acquire GCD memory lock
This issue is hidden in current code but exposed by introduction
of freed-memory guard feature due to the fact that the feature
will turn all pool allocation to page allocation.

The solution is moving the memory allocation in CoreGetMemorySpaceMap()
to be out of the GCD memory map lock.

   CoreDumpGcdMemorySpaceMap()
=> CoreGetMemorySpaceMap()
=> CoreAcquireGcdMemoryLock () *
   AllocatePool()
=> InternalAllocatePool()
=> CoreAllocatePool()
=> CoreAllocatePoolI()
=> CoreAllocatePoolPagesI()
=> CoreAllocatePoolPages()
=> FindFreePages()
=> PromoteMemoryResource()
=> CoreAcquireGcdMemoryLock()  **

Cc: Star Zeng <star.zeng@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jian J Wang <jian.j.wang@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-10-26 10:30:34 +08:00
54efcfea81 UefiCpuPkg/CpuDxe: prevent recursive calling of InitializePageTablePool
The freed-memory guard feature will cause a recursive calling
of InitializePageTablePool(). This is due to a fact that
AllocateAlignedPages() is used to allocate page table pool memory.
This function will most likely call gBS->FreePages to free unaligned
pages and then cause another round of page attributes change, like
below (freed pages will be always marked not-present if freed-memory
guard is enabled)

   FreePages() <===============|
=> CpuSetMemoryAttributes()    |
=> <if out of page table>      |
=> InitializePageTablePool()   |
=> AllocateAlignedPages()      |
=> FreePages() ================|

The solution is add a global variable as a lock in page table pool
allocation function and fail any other requests if it has not been
done.

Since this issue will only happen if free-memory guard is enabled,
it won't affect CpuSetMemoryAttributes() in default build of a BIOS.

If free-memory guard is enabled, it only affect the pages
(failed to mark them as not-present) freed in AllocateAlignedPages().

Since those freed pages haven't been used yet (their addresses not
yet exposed to code outside AllocateAlignedPages), it won't compromise
the freed-memory guard feature.

This change will just fail the CpuSetMemoryAttributes() called from
FreePages() but it won't fail the FreePages(). So the error status
won't be propagated upper layer of code.

Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Star Zeng <star.zeng@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jian J Wang <jian.j.wang@intel.com>
Reviewed-by: Eric Dong <eric.dong@intel.com>
2018-10-26 10:30:34 +08:00
02ba936dfd UefiCpuPkg/CpuDxe: consider freed-memory guard in non-stop mode
Non-stop mode was introduced / explained in commit 8f2613628a
("MdeModulePkg/MdeModulePkg.dec: add new settings for PCDs",
2018-08-30).

The macro HEAP_GUARD_NONSTOP_MODE was added to CpuDxe in commit
dcc026217f ("UefiCpuPkg/CpuDxe: implement non-stop mode for uefi",
2018-08-30).

Another instance of the macro HEAP_GUARD_NONSTOP_MODE was added to
PiSmmCpuDxeSmm -- with BIT1|BIT0 replaced with BIT3|BIT2 -- in commit
09afd9a42a ("UefiCpuPkg/PiSmmCpuDxeSmm: implement non-stop mode for
SMM", 2018-08-30)

Since the freed-memory guard is for UEFI-only. This patch only updates
HEAP_GUARD_NONSTOP_MODE in "UefiCpuPkg/CpuDxe/CpuDxe.h" (add BIT4).

Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Star Zeng <star.zeng@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jian J Wang <jian.j.wang@intel.com>
Reviewed-by: Eric Dong <eric.dong@intel.com>
2018-10-26 10:27:21 +08:00
5323e918c3 MdeModulePkg: introduce UEFI freed-memory guard bit in HeapGuard PCD
UAF (Use-After-Free) memory issue is kind of illegal access to memory
which has been freed. It can be detected by a new freed-memory guard
enforced onto freed memory.

BIT4 of following PCD is used to enable the freed-memory guard feature.

  gEfiMdeModulePkgTokenSpaceGuid.PcdHeapGuardPropertyMask

Please note this feature is for debug purpose and should not be enabled
in product BIOS, and cannot be enabled with pool/page heap guard at the
same time. It's disabled by default.

Cc: Star Zeng <star.zeng@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jian J Wang <jian.j.wang@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
2018-10-26 10:27:20 +08:00
a6bde8a644 MdeModulePkg: cleanup Heap Guard pool/page type PCD documentation
This cleanup is meant for avoiding misuse of newly introduced BIT4
(UAF detection) of PCD PcdHeapGuardPropertyMask, because it applies
to all types of physical memory. In another words,
PcdHeapGuardPoolType and PcdHeapGuardPageType don't have effect to
the BIT4 of PcdHeapGuardPropertyMask.

Cc: Star Zeng <star.zeng@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jian J Wang <jian.j.wang@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
2018-10-26 10:27:20 +08:00
d2aefa0adf SecurityPkg/Include/Library/TcgStorageOpalLib.h: Update Pyrite spec revision.
Pyrite 2.0 spec has been published, update the spec link info for this file.

Cc: Hao Wu <hao.a.wu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Hao Wu <hao.a.wu@intel.com>
2018-10-26 09:44:47 +08:00
ea98a8254a BaseTools/GenFds: create and use new variable in FdfParser
replace lots of '}' and "}" with a shared new consistent variable.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-10-25 20:17:38 +08:00
ca957eb5cd BaseTools/GenFds: refactor FdfParser warnings
make functions for common error messages
refactor to use these functions

Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-10-25 20:17:14 +08:00
5a6a268d51 BaseTools/GenFds: remove MacroDict parameter
The MacroDict parameter goes around in circles through 4 functions without use.
1. GetSectionData calls into GetLeafSection, otherwise doesn?t use MacroDict
2. GetLeafSection calls into GetFileStatement, otherwise doesn?t use MacroDict
3. GetFileStatement calls into GetFilePart, otherwise doesn?t use MacroDict
4. GetFilePart calls into GetSectionData, otherwise doesn?t use MacroDict
	Go to 1 and repeat forever.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-10-25 20:16:31 +08:00
9ffaaac2e5 Basetools/GenFds: refactor class FV
1) initialize UiFvName via __init__ parameter. No change to default behavior.
2) initialize 3 empty lists in __init__. Curently not guarenteed initialized.

Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-10-25 20:15:32 +08:00
5a264f28ee BaseTools/GenFds: change objects to sets
Change lists and tuples used solely for "in" testing to sets.
These operations are not order dependent.
fixed some line length for PEP8 compliance on some.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-10-25 20:15:07 +08:00
9e47e6f908 BaseTools/GenFds: cleanup GenFds
1) remove wildcard imports and use explicit imports
2) refactor to use shared variables from Common/DataType
3) rename to not shadow imports
4) don't assign a variable in a loop (just do final assignment)
5) remove spaces, parens, unused or commented out code, etc.
6) merge unnecessary parent classes into child
7) refactor to share DXE and PEI apriori GUIDs from one place
  this includes changes to Build and EOT files
8) for PEP8, dont use __ for custom methods.

Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob C Feng <bob.c.feng@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-10-25 20:14:49 +08:00
0019375fbc Maintainers.txt: add DynamicTablesPkg
DynamicTablesPkg has been in edk2-staging[1] for some time now, and it is
time for it to move into the main tree.
[1] https://github.com/tianocore/edk2-staging/tree/dynamictables

Add Evan and Sami as maintainers of the new package, and let them bring
it in themselves.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Leif Lindholm <leif.lindholm@linaro.org>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Evan Lloyd <evan.lloyd@arm.com>
Reviewed-by: Sami Mujawar <sami.mujawar@arm.com>
2018-10-25 11:04:28 +01:00
b1cc6f672f IntelFsp2Pkg: FSP should not override IDT
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1265

FSP should not override IDT table when it is initialized
by boot loader. IDT should be re-initialized in FSP only
when it is invalid.
To mitigate temporary memory usage a PCD
PcdFspMaxInterruptSupported created for platform to decide
how many interrupts the FSP IDT table can support.

Test: Verified on internal platform and boots successfully.

Cc: Jiewen Yao <Jiewen.yao@intel.com>
Cc: Desimone Nathaniel L <nathaniel.l.desimone@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Chasel Chiu <chasel.chiu@intel.com>
Reviewed-by: Jiewen Yao <Jiewen.yao@intel.com>
2018-10-25 17:01:36 +08:00
5061efe70d BaseTools: list .nasm include inc files as its dependency
.nasm source file may include some header files.
header file should be listed in Makefile as .nasm source file
dependency.
But now, BaseTools doesn't find them and list them in Makefile.
This is a missing, because original ASM file supports it.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-25 16:43:00 +08:00
ebed920ffd BaseTools: Fix VPD PCD Sub-section display bug
original we get the VPD PCD items from the VPDGuid.map file in the FV
output folder. but this logic doesn't work when 1) there only have
single non Default SKU, 2) there have multiple SKU with same value.
Now we change it to get the really VPD Pcd items that already display
in the PCD section of the report.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-25 16:40:18 +08:00
5d54ab94db BaseTools: Fix the bug for Pcd used in command line's override
V2: remove the not used parameter i

Fix the bug for Pcd used in command line not override the Pcd used
in the [component] driver's sub-section.

Case:
DSC file:
[PcdsFixedAtBuild]
TokenSpaceGuid.PcdTest

[Components]
 TestPkg/TestDriver.inf {
  <PcdsFixedAtBuild>
  TokenSpaceGuid.PcdTest|"b"
  }

build command with --pcd TokenSpaceGuid.PcdTest="AAAABB"

Then we found the Pcd value in the AutoGen.c file is incorrect,
because of the incorrect logic that use the pcd in the [component]
section to re-override it.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-25 15:16:16 +08:00
8411c9d5c4 MdeModulePkg/NvmExpressDxe: Refine PassThru IO queue creation behavior
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=1260

For the PassThru() service of NVM Express Pass Through Protocol, the
current implementation (function NvmExpressPassThru()) will only use the
IO Completion/Submission queues created internally by this driver during
the controller initialization process. Any other IO queues created will
not be consumed.

So the value is little to accept external IO Completion/Submission queue
creation request. This commit will refine the behavior of function
NvmExpressPassThru(), it will only accept driver internal IO queue
creation commands and will return "EFI_UNSUPPORTED" for external ones.

Cc: Jiewen Yao <Jiewen.yao@intel.com>
Cc: Star Zeng <star.zeng@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-10-25 12:41:37 +08:00
5687ae5723 MdeModulePkg/NvmExpressDxe: Always copy CQ entry to PassThru packet
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=1259

According to the the NVM Express spec Revision 1.1, for some commands,
command-related information will be stored in the Dword 0 of the
completion queue entry.

One case is for the Get Features Command (Section 5.9.2 of the spec),
Dword 0 of the completion queue entry may contain feature information.

Hence, this commit will always copy the content of completion queue entry
to the PassThru packet regardless of the execution result of the command.

Cc: Liangcheng Tang <liangcheng.tang@intel.com>
Cc: Star Zeng <star.zeng@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-10-25 12:41:19 +08:00
ebb6c7633b MdeModulePkg/NvmExpressDxe: Refine data buffer & len check in PassThru
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=1142

According to the the NVM Express spec Revision 1.1, for some commands
(like Get/Set Feature Command, Figure 89 & 90 of the spec), the Memory
Buffer maybe optional although the command opcode indicates there is a
data transfer between host & controller (Get/Set Feature Command, Figure
38 of the spec).

Hence, this commit refine the checks for the 'TransferLength' and
'TransferBuffer' field of the EFI_NVM_EXPRESS_PASS_THRU_COMMAND_PACKET
structure to address this issue.

Cc: Liangcheng Tang <liangcheng.tang@intel.com>
Cc: Star Zeng <star.zeng@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-10-25 12:41:17 +08:00
27bf6712b4 IntelSiliconPkg VTdDxe: Report status code for VTd error
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1273

Current code only uses DEBUG() for VTd error.
This patch updates to also report status code for VTd error.

Test done:
Created case that has VTd error and confirmed the error
status code could be reported as expected.

Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Rangasai V Chaganty <rangasai.v.chaganty@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Jiewen Yao <jiewen.yao@intel.com>
2018-10-25 11:50:49 +08:00
8c09f30010 IntelSiliconPkg VTdDxe: Option to force no early access attr request
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1272

To have high confidence in usage for platform, add option (BIT2 of
PcdVTdPolicyPropertyMask) to force no IOMMU access attribute request
recording before DMAR table is installed.

Check PcdVTdPolicyPropertyMask BIT2 before RequestAccessAttribute()
and ProcessRequestedAccessAttribute(), then RequestAccessAttribute(),
ProcessRequestedAccessAttribute() and mAccessRequestXXX variables
could be optimized by compiler when PcdVTdPolicyPropertyMask BIT2 = 1.

Test done:
1: Created case that has IOMMU access attribute request before DMAR
   table is installed, ASSERT was triggered after setting
   PcdVTdPolicyPropertyMask BIT2 to 1.

2. Confirmed RequestAccessAttribute(), ProcessRequestedAccessAttribute()
   and mAccessRequestXXX variables were optimized by compiler after
   setting PcdVTdPolicyPropertyMask BIT2 to 1.

Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Rangasai V Chaganty <rangasai.v.chaganty@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Jiewen Yao <jiewen.yao@intel.com>
2018-10-25 11:50:43 +08:00
0bc7448ad2 MdeModulePkg/UsbBus: Removing trailing spaces
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-10-25 11:19:29 +08:00
c637c60273 BaseTools: Move PcdValueInit to platform build folder
PcdValueInit tool is platform scope.
It should be generated into Platform output directory.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob C Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-25 08:15:19 +08:00
51d17bb7b0 BaseTool: Filter out unused structure pcds
The current code handle all the structure pcds
even if there is no module or library use them.
This patch is going to filter out the unused structure pcds.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-25 08:15:14 +08:00
951fb2d4e5 BaseTools: Remove Arch specific build options for PcdValueInit tool.
PcdValueInit tool is Arch independent, the Arch specific
build options should be removed from PcdValueInit makefile.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob C Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-24 22:27:05 +08:00
2855e9c330 BaseTools:Translate the StructurePCD value in field to correct format.
For StructurePCD value got from DSC file, translate its field value in to
correct format in report.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: ZhiqiangX Zhao <zhiqiangx.zhao@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2018-10-24 22:27:05 +08:00
da2d4f76b7 BaseTools: Not convert the void* pcd string in command line to array.
For void* type pcd in command line, if its value is string, code should
not convert the void* pcd string in command line to array, otherwise it
will make the pcd value in report not match its real raw value.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: ZhiqiangX Zhao <zhiqiangx.zhao@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2018-10-24 22:27:04 +08:00
9f2b0594ed BaseTools: add ASSERT checker for array buffer in fdf and command line
For structure PCD in fdf file and command line,
1. use compiler time assert to check the array index, report error
if array index exceeds the array number.
2. use compiler time assert to check the array size, report error
if the user declared size in header file is smaller than the user
used in fdf file and command line.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: ZhiqiangX Zhao <zhiqiangx.zhao@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2018-10-24 22:27:04 +08:00
a8b5750901 MdePkg: Handle AcpiExp device path when optional para is not specified
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1243

AcpiExp text device path: AcpiExp(HID,CID,UIDSTR)
And according to UEFI spec, the CID parameter is optional
and has a default value of 0. But current implementation
miss to check following cases for the AcpiExp.
FromText:when text device is AcpiExp(HID,,UIDSTR)/AcpiExp(HID,0,UIDSTR)
ToText: when the CID is 0 in the node structure

This commit is to do the enhancement.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Dandan Bi <dandan.bi@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-10-24 22:15:57 +08:00
3874108034 MdePkg: Handle USBxxx device path when optional para is not specified
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1243

According to UEFI spec,
for the Messaging Device Path with USB Class SubType, some paras
are optional in the text device path.
Take UsbClass(VID,PID,Class,SubClass,Protocol) for example,
The VID is an integer between 0 and 65535 and is optional. The
default value is 0xFFFF.
The PID is an integer between 0 and 65535 and is optional. The
default value is 0xFFFF.
The Class is an integer between 0 and 255 and is optional. The
default value is 0xFF.
The SubClass is an integer between 0 and 255 and is optional. The
default value is 0xFF.
The Protocol is an integer between 0 and 255 and is optional. The
default value is 0xFF.
So if any the optional para is not specified in the text device,
we should set related para in the node structure to default value.

This commit is to do the enhancement for USB Class device path
when optional para is not specified

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Dandan Bi <dandan.bi@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-10-24 22:15:56 +08:00
6d9b9bbb61 MdePkg: Handle Sata device path when optional para is not specified
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1243

Sata device path format:Sata(HPN, PMPN, LUN)
According to UEFI Spec, the PMPN is an integer between
0 and 65535 and is optional. If not provided, the default is 0xFFFF.

This commit is to do the enhancement for Sata device path
when optional para is not specified.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Dandan Bi <dandan.bi@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-10-24 22:15:56 +08:00
ea1486c281 FatPkg: Correct the line ending to CRLF
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-10-24 15:22:31 +08:00
b08b045ca9 MdePkg-BaseLib: Fix PathCleanUpDirectories() issue with "\\..\\.."
Replace multiple, consecutive "\" characters prior to other processing
involving "\" characters.  This fixes an issue where "\\..\\..",
"//..//..", and similar input paths are not cleaned properly.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jim Dailey <jim_dailey@dell.com>
Reviewed-by: Ruiyu Ni <Ruiyu.ni@Intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-24 13:36:35 +08:00
add1310df5 FatPkg/EnhancedFatDxe Fix Double Cluster Allocation
This is a fix for a double cluster allocation when the disk is full.
The double allocation happens because FatGrowEof calls
FatAllocateCluster without immediately marking the each returned
cluster as allocated. The fix is to move the FatSetFatEntry call
inside the loop.
I've also include some improvements to the sanity checks that I added
while tracking this down. They are optional.

Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Herbie Robinson <Herbie.Robinson@stratus.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-10-24 13:30:43 +08:00
8a2794f6f3 FatBinPkg: Remove FatBinPkg and refresh document
Remove FatBinPkg and refresh Maintainers.txt.
https://bugzilla.tianocore.org/show_bug.cgi?id=1105

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Andrew Fish <afish@apple.com>
Reviewed-by: Leif Lindholm <leif.lindholm@linaro.org>
2018-10-24 13:23:54 +08:00
f1a7d73a68 ShellPkg/dmem: Only dump sizeof (EFI_SYSTEM_TABLE) bytes for gST
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1236

When "dmem" runs without additional arguments, it dumps the memory
content of EFI_SYSTEM_TABLE. But today's implementation dumps 512
bytes. It's not correct because sizeof (EFI_SYSTEM_TABLE) is less
than 512, the 512-read causes page fault exception in a heap-guard
enabled environment.

The patch changes the implementation to only dump
sizeof (EFI_SYSTEM_TABLE) bytes for gST.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Jim Dailey <jim_dailey@.com>
2018-10-24 12:18:22 +08:00
a58a421c36 MdeModulePkg/UsbMass: Fix USB key write failure
Commit e59db6a732
* MdeModulePkg/UsbMass: Merge UsbBoot(Read|Write)Blocks(16)
introduces a bug that causes writing to USB key always fails.

When that patch is verified, only reading was verified.

The root cause is when the writing operation is performed,
the data direction EfiUsbDataIn is wrongly used. Instead, it
should be EfiUsbDataOut.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-10-24 12:00:19 +08:00
90c64aada8 BaseTools/UPT: Fix an issue of UNI string checking.
The tool now can detect the error that the content between double
quotes contains another double quotes or enter key.

Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Hess Chen <hesheng.chen@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-10-24 08:41:05 +08:00
7a57490c2e BaseTools/Ecc: Update a checkpoint criteria.
Change the criteria of the checkpoint of "#ifndef" to remove the requirement of prefix '_'.

Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Hess Chen <hesheng.chen@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-10-24 08:41:04 +08:00
3a244b8db4 BaseTools/ECC: Fix an identification issue of typedef function.
Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Hess Chen <hesheng.chen@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-10-24 08:41:04 +08:00
33a211d009 BaseTools/ECC: Add a checkpoint to check no usage for deprecated functions.
Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Hess Chen <hesheng.chen@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-10-24 08:41:03 +08:00
041d89bc0f BaseTools: Add more checker in Decompress algorithm to access the valid buffer (CVE FIX)
Fix CVE-2017-5731,CVE-2017-5732,CVE-2017-5733,CVE-2017-5734,CVE-2017-5735
https://bugzilla.tianocore.org/show_bug.cgi?id=686

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Holtsclaw Brent <brent.holtsclaw@intel.com>
Signed-off-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
2018-10-24 08:24:15 +08:00
684db6da64 IntelFrameworkModulePkg: Add more checker in UefiTianoDecompressLib (CVE FIX)
Fix CVE-2017-5731,CVE-2017-5732,CVE-2017-5733,CVE-2017-5734,CVE-2017-5735
https://bugzilla.tianocore.org/show_bug.cgi?id=686
To make sure the valid buffer be accessed only.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Holtsclaw Brent <brent.holtsclaw@intel.com>
Signed-off-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
2018-10-24 08:24:14 +08:00
2ec7953d49 MdePkg: Add more checker in UefiDecompressLib to access the valid buffer only (CVE FIX)
Fix CVE-2017-5731,CVE-2017-5732,CVE-2017-5733,CVE-2017-5734,CVE-2017-5735
https://bugzilla.tianocore.org/show_bug.cgi?id=686

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Holtsclaw Brent <brent.holtsclaw@intel.com>
Signed-off-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
2018-10-24 08:24:14 +08:00
68099b52b0 MdeModulePkg/UdfDxe: Avoid possible use of already-freed data
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=1255

For function ReadFile():

If the line

  Status = GetAedAdsData (
   ...
   );

is reached multiple times during the 'for' loop, freeing the data pointed
by variable 'Data' may potentially lead to variable 'Ad' referencing the
already-freed data.

After calling function GetAllocationDescriptor(), 'Data' and 'Ad' may
point to the same memory (with some possible offset). Hence, this commit
will move the FreePool() call backwards to ensure the data will no longer
be used.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Paulo Alcantara <palcantara@suse.de>
Acked-by: Star Zeng <star.zeng@intel.com>
2018-10-23 14:25:04 +08:00
85acb5e8ff MdeModulePkg/UdfDxe: Add more check when getting PD from LongAd
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=1254

This commit will add an additional check within function GetPdFromLongAd()
when getting a Partition Descriptor from given a Long Allocation
Descriptor.

According to UDF 2.60 Spec, Section 2.2.13:

> The partition reference numbers used are determined by the order of the
> Partition Maps in the LVD.

(Also the picture comes before the above contents)

And a more detailed explanation of the partition reference numbers is at
https://sites.google.com/site/udfintro/ (seems not a formal documentation
though), Section 5.3.6.

Based on the above findings, the 'PartitionReferenceNumber' field in a
Long Allocation Descriptor is used as an index to access the Partition
Maps data within a Logical Volume Descriptor.

Hence, the new check focuses on the validity of this
'PartitionReferenceNumber' field in a Long Allocation Descriptor. Since
the current implementation of UdfDxe driver supports only one partition on
a Logical Volume, so the value of 'PartitionReferenceNumber' should be 0.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Paulo Alcantara <palcantara@suse.de>
Acked-by: Star Zeng <star.zeng@intel.com>
2018-10-23 14:24:58 +08:00
32698a8f01 MdeModulePkg/UdfDxe: Update GetInfo() for FS VolumeLabel info request
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=1175

This commit will update the UdfGetInfo() function with the support of
EFI_FILE_SYSTEM_VOLUME_LABEL data information request.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Paulo Alcantara <palcantara@suse.de>
Acked-by: Star Zeng <star.zeng@intel.com>
2018-10-23 14:24:50 +08:00
6a926aaed7 MdeModulePkg/UdfDxe: Fix a typo within SetFileInfo()
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=1253

Within function SetFileInfo():
This commit will fix a typo where 'Minute' should be used instead of
'Second'.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Paulo Alcantara <palcantara@suse.de>
Acked-by: Star Zeng <star.zeng@intel.com>
2018-10-23 14:24:23 +08:00
6df465b212 MdeModulePkg/UdfDxe: Correct behavior for UdfSetPosition()
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=1252

Calling the 'SetPosition' service of the EFI_FILE_PROTOCOL with 'Position'
equals to 0xFFFFFFFFFFFFFFFF for a file is to set the current position to
the end of the file. But the current implementation of function
UdfSetPosition() is to set it to the last byte (not EOF).

This commit will resolve this issue.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Paulo Alcantara <palcantara@suse.de>
Acked-by: Star Zeng <star.zeng@intel.com>
2018-10-23 14:24:17 +08:00
3b30351b75 MdeModulePkg/UdfDxe: Add boundary check for getting volume (free) size
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=828

Within GetVolumeSize():

The boundary check will validate the 'NumberOfPartitions' field of a
Logical Volume Integrity Descriptor matches the data within the relating
Logical Volume Descriptor.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Paulo Alcantara <palcantara@suse.de>
Acked-by: Star Zeng <star.zeng@intel.com>
2018-10-23 14:23:57 +08:00
89f75aa04a MdeModulePkg/UdfDxe: Add boundary check for ComponentIdentifier decode
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=828

Within ResolveSymlink():

The boundary check will validate the 'LengthofComponentIdentifier' field
of a Path Component matches the data within the relating (Extended) File
Entry.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Paulo Alcantara <palcantara@suse.de>
Acked-by: Star Zeng <star.zeng@intel.com>
2018-10-23 14:23:35 +08:00
5c0748f43f MdeModulePkg/UdfDxe: Add boundary check the read of FE/EFE
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=828

Within ReadFile():

Add checks to ensure that when getting the raw data or the Allocation
Descriptors' data from a FE/EFE, it will not consume data beyond the
size of a FE/EFE.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Paulo Alcantara <palcantara@suse.de>
Acked-by: Star Zeng <star.zeng@intel.com>
2018-10-23 14:23:25 +08:00
b9ae1705ad MdeModulePkg/UdfDxe: Refine boundary checks for file/path name string
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=828

The commit refines the boundary checks for file/path name string to
prevent possible buffer overrun.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Paulo Alcantara <palcantara@suse.de>
Acked-by: Star Zeng <star.zeng@intel.com>
2018-10-23 14:23:18 +08:00
4df8f5bfa2 MdeModulePkg/PartitionDxe: Add check for underlying device block size
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=828

Within FindAnchorVolumeDescriptorPointer():

Add a check for the underlying device block size to ensure it is greater
than the size of an Anchor Volume Descriptor Pointer.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Paulo Alcantara <palcantara@suse.de>
Acked-by: Star Zeng <star.zeng@intel.com>
2018-10-23 14:23:09 +08:00
4a723d3d7f MdeModulePkg Xhci: Handle value 5 in Port Speed field of PORTSC
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1267

The value 5 Port Speed field of PORTSC is new defined in
XHCI 1.1 spec November 2017.

This patch updates XhciDxe and XhciPei to handle it, otherwise
the USB 3.1 device may not be recognized with the XHCI controller
following XHCI 1.1 spec November 2017.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Hao Wu <hao.a.wu@intel.com>
Cc: Jian J Wang <jian.j.wang@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-10-23 11:17:32 +08:00
fed6cf25b8 MdeModulePkg XhciDxe: Assign Usb2Hc.XXXRevision based on SBRN
Current hard code Usb2Hc.XXXRevision may be not accurate.
This patch updates code to assign Usb2Hc.XXXRevision based on
SBRN (Serial Bus Release Number, PCI configuration space offset
0x60) although there is no code consuming them.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Hao Wu <hao.a.wu@intel.com>
Cc: Jian J Wang <jian.j.wang@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-10-23 11:17:31 +08:00
a6a326770b BaseTools: Fix the *B and *F Flag display for Structure Pcd
Because of we newly add the PcdFieldValueFromComm and
PcdFieldValueFromFdf in early parser phase, so in the report we use
the saved value in this two variables to print it.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-23 10:40:31 +08:00
0dab57708d MdePkg/BaseLib: AsciiStrToUnicodeStr(S) not handle EASCII properly
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=1262

Current implementation of BaseLib APIs:

AsciiStrToUnicodeStr()
AsciiStrToUnicodeStrS()
AsciiStrnToUnicodeStrS()

do not handle EASCII properly.

More specifically, if the value of ASCII character is larger than 0x7F,
then the converted Unicode character will have all '1's in the higher 8
bits.

An example:
  0xC9 => 0xFFC9 (current implementations)
and it should be:
  0xC9 => 0x00C9

This commit will address this issue.

Cc: Bin.Lain <bin_601@mail2000.com.tw>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-23 08:45:00 +08:00
073891a3e7 BaseTools: Convert "Unicode string" to "byte array" if value type diff
V2:
Fixed 3 typo.
Use startswith(('L"',"L'")) to check if a string is Unicode string.
Use a set PcdValueTypeSet instead of a list PcdValueTypeList to save
memory.

V1:
For the same one VOID* pcd, if the default value type of one SKU is
"Unicode string", the other SKUs are "OtherVOID*"(ASCII string or
byte array),Then convert "Unicode string" to "byte array".

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: ZhiqiangX Zhao <zhiqiangx.zhao@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2018-10-22 15:19:27 +08:00
32be12235d BaseTool: Support different PCDs that refers to the same EFI variable.
V2:
Make the code of patch both compatible for Python2 and Python3.

V1:
If different PCDs refer to the same EFI variable, then do EFI variable
combination, according to the VariableOffset of different PCDS.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: ZhiqiangX Zhao <zhiqiangx.zhao@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2018-10-22 15:19:26 +08:00
d28daaddb3 UefiCpuPkg/CpuCommonFeaturesLib: Register MSR base on scope Info.
Because MSR has scope attribute, driver has no needs to set
MSR for all APs if MSR scope is core or package type. This patch
updates code to base on the MSR scope value to add MSR to the register
table.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-10-22 11:19:49 +08:00
38381e18bf UefiCpuPkg/CpuS3DataDxe: Keep old data if value already existed.
AcpiCpuData add new fields, keep these fields if old data already existed.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Regression-tested-by: Laszlo Ersek <lersek@redhat.com>
2018-10-22 11:19:48 +08:00
9332439058 UefiCpuPkg/PiSmmCpuDxeSmm: Add logic to support semaphore type.
V4 changes:
1. Serial console log for different threads when program register table.
2. Check the AcpiCpuData before use it to avoid potential ASSERT.

V3 changes:
1. Use global variable instead of internal function to return string for register type
   and dependence type.
2. Add comments for some complicated logic.

V1 changes:
Because this driver needs to set MSRs saved in normal boot phase, sync
semaphore logic from RegisterCpuFeaturesLib code which used for normal boot phase.

Detail see below change for RegisterCpuFeaturesLib:
  UefiCpuPkg/RegisterCpuFeaturesLib: Add logic to support semaphore type.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Regression-tested-by: Laszlo Ersek <lersek@redhat.com>
2018-10-22 11:19:48 +08:00
b3c71b472d UefiCpuPkg/RegisterCpuFeaturesLib: Add logic to support semaphore type.
V4 changes include:
1. Serial debug message for different threads when program the register table.

V3 changes include:
1. Use global variable instead of internal function to return string for register type
   and dependence type.
2. Add comments for some complicated logic.

V2 changes include:
1. Add more description for the code part which need easy to understand.
2. Refine some code base on feedback for V1 changes.

V1 changes include:
In a system which has multiple cores, current set register value task costs huge times.
After investigation, current set MSR task costs most of the times. Current logic uses
SpinLock to let set MSR task as an single thread task for all cores. Because MSR has
scope attribute which may cause GP fault if multiple APs set MSR at the same time,
current logic use an easiest solution (use SpinLock) to avoid this issue, but it will
cost huge times.

In order to fix this performance issue, new solution will set MSRs base on their scope
attribute. After this, the SpinLock will not needed. Without SpinLock, new issue raised
which is caused by MSR dependence. For example, MSR A depends on MSR B which means MSR A
must been set after MSR B has been set. Also MSR B is package scope level and MSR A is
thread scope level. If system has multiple threads, Thread 1 needs to set the thread level
MSRs and thread 2 needs to set thread and package level MSRs. Set MSRs task for thread 1
and thread 2 like below:

            Thread 1                 Thread 2
MSR B          N                        Y
MSR A          Y                        Y

If driver don't control execute MSR order, for thread 1, it will execute MSR A first, but
at this time, MSR B not been executed yet by thread 2. system may trig exception at this
time.

In order to fix the above issue, driver introduces semaphore logic to control the MSR
execute sequence. For the above case, a semaphore will be add between MSR A and B for
all threads. Semaphore has scope info for it. The possible scope value is core or package.
For each thread, when it meets a semaphore during it set registers, it will 1) release
semaphore (+1) for each threads in this core or package(based on the scope info for this
semaphore) 2) acquire semaphore (-1) for all the threads in this core or package(based
on the scope info for this semaphore). With these two steps, driver can control MSR
sequence. Sample code logic like below:

  //
  // First increase semaphore count by 1 for processors in this package.
  //
  for (ProcessorIndex = 0; ProcessorIndex < PackageThreadsCount ; ProcessorIndex ++) {
    LibReleaseSemaphore ((UINT32 *) &SemaphorePtr[PackageOffset + ProcessorIndex]);
  }
  //
  // Second, check whether the count has reach the check number.
  //
  for (ProcessorIndex = 0; ProcessorIndex < ValidApCount; ProcessorIndex ++) {
    LibWaitForSemaphore (&SemaphorePtr[ApOffset]);
  }

Platform Requirement:
1. This change requires register MSR setting base on MSR scope info. If still register MSR
   for all threads, exception may raised.

Known limitation:
1. Current CpuFeatures driver supports DXE instance and PEI instance. But semaphore logic
   requires Aps execute in async mode which is not supported by PEI driver. So CpuFeature
   PEI instance not works after this change. We plan to support async mode for PEI in phase
   2 for this task.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-10-22 11:19:48 +08:00
87e9395109 UefiCpuPkg/RegisterCpuFeaturesLib.h: Add new dependence types.
V4 changes:
  1. Update comments.

v3 changes:
  1. Move CPU_FEATURE_DEPENDENCE_TYPE definition to AcpiCpuData.h.
  2. Add comments for CPU_FEATURE_BEFORE/CPU_FEATURE_AFTER.

v1 changes:
Add new core/package dependence types which consumed by different MSRs.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-10-22 11:19:47 +08:00
d5aa2078f7 UefiCpuPkg/Include/AcpiCpuData.h: Add Semaphore related Information.
v3 changes:
1. Move CPU_FEATURE_DEPENDENCE_TYPE definition here from RegisterCpuFeaturesLib.h file.
2. Add Invalid type for REGISTER_TYPE which will be used in code.

v2 changes:
1. Add more description about why we do this change.
2. Change structure field type from pointer to EFI_PHYSICAL_ADDRESS because it will
   be share between PEI and DXE.

v1 Changes:
In order to support semaphore related logic, add new definition for it.

In a system which has multiple cores, current set register value task costs huge times.
After investigation, current set MSR task costs most of the times. Current logic uses
SpinLock to let set MSR task as an single thread task for all cores. Because MSR has
scope attribute which may cause GP fault if multiple APs set MSR at the same time,
current logic use an easiest solution (use SpinLock) to avoid this issue, but it will
cost huge times.

In order to fix this performance issue, new solution will set MSRs base on their scope
attribute. After this, the SpinLock will not needed. Without SpinLock, new issue raised
which is caused by MSR dependence. For example, MSR A depends on MSR B which means MSR A
must been set after MSR B has been set. Also MSR B is package scope level and MSR A is
thread scope level. If system has multiple threads, Thread 1 needs to set the thread level
MSRs and thread 2 needs to set thread and package level MSRs. Set MSRs task for thread 1
and thread 2 like below:

            Thread 1                 Thread 2
MSR B          N                        Y
MSR A          Y                        Y

If driver don't control execute MSR order, for thread 1, it will execute MSR A first, but
at this time, MSR B not been executed yet by thread 2. system may trig exception at this
time.

In order to fix the above issue, driver introduces semaphore logic to control the MSR
execute sequence. For the above case, a semaphore will be add between MSR A and B for
all threads. Semaphore has scope info for it. The possible scope value is core or package.
For each thread, when it meets a semaphore during it set registers, it will 1) release
semaphore (+1) for each threads in this core or package(based on the scope info for this
semaphore) 2) acquire semaphore (-1) for all the threads in this core or package(based
on the scope info for this semaphore). With these two steps, driver can control MSR
sequence. Sample code logic like below:

  //
  // First increase semaphore count by 1 for processors in this package.
  //
  for (ProcessorIndex = 0; ProcessorIndex < PackageThreadsCount ; ProcessorIndex ++) {
    LibReleaseSemaphore ((UINT32 *) &SemaphorePtr[PackageOffset + ProcessorIndex]);
  }
  //
  // Second, check whether the count has reach the check number.
  //
  for (ProcessorIndex = 0; ProcessorIndex < ValidApCount; ProcessorIndex ++) {
    LibWaitForSemaphore (&SemaphorePtr[ApOffset]);
  }

Platform Requirement:
1. This change requires register MSR setting base on MSR scope info. If still register MSR
   for all threads, exception may raised.

Known limitation:
1. Current CpuFeatures driver supports DXE instance and PEI instance. But semaphore logic
   requires Aps execute in async mode which is not supported by PEI driver. So CpuFeature
   PEI instance not works after this change. We plan to support async mode for PEI in phase
   2 for this task.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Regression-tested-by: Laszlo Ersek <lersek@redhat.com>
2018-10-22 11:19:47 +08:00
95aea2fac9 NetworkPkg/IpSecDxe: Fix issue to parse SA Payload.
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1251

*v2: Correct the type of parameters in Ikev2ParseProposalData(), and refined
the corresponding description.

IpSecDxe failed to create the Child SA during parsing SA Payload, the issue
was caused by the below commit:

SHA-1: 1e0db7b119
* MdeModulePkg/NetworkPkg: Checking for NULL pointer before use.

In above commit, it changed the value of IsMatch in Ikev2ChildSaParseSaPayload()
to FALSE. That's correct but it exposed the potential bug in to match the correct
proposal Data, which will cause the issue happen.

Cc: Fu Siyuan <siyuan.fu@intel.com>
Cc: Ye Ting <ting.ye@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Wu Jiaxin <jiaxin.wu@intel.com>
Reviewed-by: Ye Ting <ting.ye@intel.com>
2018-10-22 09:04:57 +08:00
ddc6d41d12 NetworkPkg: Correct the time stamp and fix the integer overflow issue.
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=883.

Cc: Fu Siyuan <siyuan.fu@intel.com>
Cc: Ye Ting <ting.ye@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Wu Jiaxin <jiaxin.wu@intel.com>
Reviewed-by: Ye Ting <ting.ye@intel.com>
2018-10-22 09:04:56 +08:00
1682cc9353 NetworkPkg/TlsDxe: Remove the redundant library class.
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1018.

Cc: Fu Siyuan <siyuan.fu@intel.com>
Cc: Ye Ting <ting.ye@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Wu Jiaxin <jiaxin.wu@intel.com>
Reviewed-by: Fu Siyuan <siyuan.fu@intel.com>
Reviewed-by: Ye Ting <ting.ye@intel.com>
2018-10-22 09:04:55 +08:00
2c65efac57 BaseTools: Fix one crash bug in the report for Fixed structure Pcd
The case is:
in the DSC file:
SKUID_IDENTIFIER               = ALL

[SkuIds]
  0|DEFAULT
  1|A

[PcdsFixedAtBuild.common.A]
  TokenSpaceGuid.Test401|{0x0F, 0x12}
  TokenSpaceGuid.Test401.TEST401INT8ARRAY[0]|'B'

in the build report, Data = OverrideValues[Keys[0]], but the Keys[0]
is the keyword "DEFAULT", and in this case the "DEFAULT" SKU doesn't
save any value, then it cause the Data is empty, in the next code
when we use the code it cause crash.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2018-10-21 20:10:21 +08:00
7cedfa1c81 BaseTools: delete unused file
this file is not imported/used.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-10-21 20:06:03 +08:00
eb99b52f98 BaseTools: Fix the crash issue when Dynamic structure Pcd use in FDF
The case is use Dynamic structure Pcd in the FDF file.
Current code already save the  Guid, Name and Filed info for those Pcd,
but it directly use the dict key as [Name, Guid] and cause this crash
issue.

Fixes: https://bugzilla.tianocore.org/show_bug.cgi?id=1264
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2018-10-21 20:03:37 +08:00
c3d0f526c0 BaseTools: Fix the bug that PcdValueFromComm is not set
the bug is PcdValueFromComm is not set, but the Pcd have been override
by the command line option.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2018-10-21 20:03:02 +08:00
e7ac09b957 MdePkg UefiLib: Check Table against NULL in ScanTableInSDT
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1266

af5e952159 abstracted generic functions
from different modules (IntelVTdDxe, S3SaveStateDxe, PcRtc,
DpDynamicCommand and PiSmmCpuDxeSmm). Some of them (IntelVTdDxe and
PcRtc) checked Table against NULL before accessing Table->Signature,
some (S3SaveStateDxe, DpDynamicCommand and PiSmmCpuDxeSmm did not.

The ScanTableInSDT() in Acpi.c of UefiLib was mainly from
S3SaveStateDxe, so it does not check Table against NULL before
accessing Table->Signature.

This patch updates ScanTableInSDT() to check Table against NULL first
before accessing Table->Signature.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Jian J Wang <jian.j.wang@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Jian J Wang <jian.j.wang@intel.com>
2018-10-21 09:57:44 +08:00
a2e61f341d IntelFsp2Pkg: Support FSP Dispatch mode
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1241

Add support for both API (original mode) and DISPATCH mode:
1. Add FspMode field from reserved byte of Global
   Data Structure to tell which mode is selected by boot
   loader. If boot loader invoking FSP-M API this field
   will remain as default 0 (API mode), otherwise platform
   FSP should set this field to 1 (Dispatch mode) when
   initializing Global Data Structure.
2. gFspInApiModePpiGuid will be instaled when FSP running in API
   mode and modules only for API mode should have this in depex.
3. If it is DISPATCH mode, FSP will return to PEI dispatcher,
   not directly return to boot loader.
4. DISPATCH mode supports DXE NotifyPhase drivers so FSP
   will not wait for PEI NotifyPhase callbacks, instead it
   will install gFspReadyForNotifyPhasePpiGuid PPI for
   platform to complete late initialization before transferring
   to DXE.

Test: Verified FSP API and DISPATCH modes on 2 internal
      platforms and both boot successfully.

Cc: Jiewen Yao <Jiewen.yao@intel.com>
Cc: Desimone Nathaniel L <nathaniel.l.desimone@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Chasel Chiu <chasel.chiu@intel.com>
Reviewed-by: Jiewen Yao <Jiewen.yao@intel.com>
2018-10-19 15:01:00 +08:00
7b500c606a BaseTools: Remove the step to freeze python tool
https://bugzilla.tianocore.org/show_bug.cgi?id=1257
Binary python tool is not supported anymore. So, the freeze python tool
step is not required.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-10-19 08:21:47 +08:00
1c3399d73d MdeModulePkg BrotliDecompressLib: Add the checker to avoid array out of bound
This change is to pass static analysis.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-10-19 08:21:47 +08:00
eae7b476c2 UefiCpuPkg/CpuExceptionHandlerLib: always clear descriptor data in advance
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1237

Sometimes the memory will be contaminated by random data left in last
boot (warm reset). The code should not assume the allocated memory is
always filled with zero. This patch add code to clear data structure
used for stack switch to prevent such problem from happening.

Cc: Eric Dong <eric.dong@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jian J Wang <jian.j.wang@intel.com>
Reviewed-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-10-18 13:19:14 +08:00
fea5e28658 BaseTools: Fix bug caused by 03c36c36a3
In the expression for unicode string and general string compare, it
should check whether it startswith "L'" or 'L"', but not "L".

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-18 09:36:49 +08:00
53c64f4286 BaseTools: Fix a bug --pcd option enable and use the pcd in expression
the case is:
in the DSC:
[PcdsFixedAtBuild.common]
 TokenSpaceGuid.TestFixedPcd|0xFFEAA000

[PcdsDynamicExDefault.common.DEFAULT]
!if TokenSpaceGuid.PcdFlag == TRUE
TokenSpaceGuid.PcdTest|TokenSpaceGuid.TestFixedPcd
!endif

Then build with --pcd TokenSpaceGuid.PcdFlag=TRUE, it report failure,
but if we build without this --pcd option, it could build success.
we found when the --pcd is enabled, the fixedatbuild pcds are not be
collected into expression to calculate.

Fixes: https://bugzilla.tianocore.org/show_bug.cgi?id=1256
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-18 09:36:27 +08:00
5317e9ccaf MdeModulePkg/UdfDxe: Handle dead codes in FileSystemOperations.c
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=1249

We found potential dead codes within File.c during the code coverage test.

After manual review, we think the below ones are positive reports:

A. For function GetAllocationDescriptor():
Due to the all the calling places for this function, the input parameter
'RecordingFlags' can only with value 'LongAdsSequence' or
'ShortAdsSequence'. Moreover, this is also mentioned in the function
description comments for GetAllocationDescriptor().

So the below code will never be reached:

  return EFI_DEVICE_ERROR;

This commit will add "ASSERT (FALSE);" before the above line to indicate
this and thus matching the function description comments.

B. For function GetAllocationDescriptorLsn():
Due to the all the calling places for this function, the input parameter
'RecordingFlags' can only with value 'LongAdsSequence' or
'ShortAdsSequence'. Moreover, this is also mentioned in the function
description comments for GetAllocationDescriptorLsn().

So the below code will never be reached:

  return EFI_UNSUPPORTED;

This commit will add "ASSERT (FALSE);" before the above line to indicate
this and thus matching the function description comments.

Cc: Paulo Alcantara <palcantara@suse.de>
Cc: Paulo Alcantara <paulo@paulo.ac>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-10-18 09:13:40 +08:00
11941d3b1b MdeModulePkg/UdfDxe: Handle dead codes in File.c
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=1249

We found potential dead codes within File.c during the code coverage test.

After manual review, we think the below ones are positive reports:

A. In function UdfRead():
  else if (IS_FID_DELETED_FILE (Parent->FileIdentifierDesc)) {
    Status = EFI_DEVICE_ERROR;
  }

A File Identifier Descriptor will be get from the UDF media only by
function ReadDirectoryEntry(). And within this function, all the File
Identifier Descriptor with 'DELETED_FILE' characteristics will be skipped
and will not be returned. Hence, the above codes in function UdfRead()
will never be hit.

This commit will add "ASSERT (FALSE);" before the above line to indicate
this.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Paulo Alcantara <palcantara@suse.de>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-10-18 09:12:51 +08:00
f355dd3404 MdeModulePkg/UdfDxe: Use debug msg instead of ASSERT in UdfOpen()
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=1248

Within function UdfOpen():
This commit will use debug messages instead of using ASSERT when an error
occurs after calling GetFileSize().

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Paulo Alcantara <palcantara@suse.de>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-10-18 09:12:45 +08:00
6086569e7e MdeModulePkg/UdfDxe: Use error handling when fail to return LSN
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=1248

This commit will refine the ASSERTs in function GetLongAdLsn() and
GetAllocationDescriptorLsn() when the logical sector number cannot be
returned due to unrecognized media format.

Error handling logic will be used for those ASSERTs.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Paulo Alcantara <palcantara@suse.de>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-10-18 09:12:43 +08:00
5fb22f5920 MdeModulePkg/UdfDxe: ASSERT for false positives of NULL ptr deref
This commit adds ASSERTs to address false positive reports of NULL
pointer dereference issues raised from static analysis with regard to
function ReadDirectoryEntry().

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Paulo Alcantara <palcantara@suse.de>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-10-18 09:12:41 +08:00
ebb12f5142 MdeModulePkg/UdfDxe: Use error handling for memory allocation failure
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=1247

For functions DuplicateFid() and DuplicateFe(), this commit will use error
handling logic instead of ASSERTs for memory allocation failure.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Paulo Alcantara <palcantara@suse.de>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-10-18 09:12:38 +08:00
3a0329bed2 MdePkg/BaseSynchronizationLib GCC: simplify IA32 InternalSyncCompareExchange64()
The IA32 variant of InternalSyncCompareExchange64() is correct, but we can
simplify it. We don't need to load the lower 32 bits of ExchangeValue into
EBX in two steps (first into a general register, then into EBX); we can
ask GCC to populate EBX like that itself.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1208
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Acked-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-17 18:44:51 +02:00
e5d4e7500f MdePkg/BaseSynchronizationLib GCC: fix X64 InternalSyncCompareExchange64()
(This patch is identical to the X64 half of the last one, except for the
InternalSyncCompareExchange32() -> InternalSyncCompareExchange64() and
"cmpxchgl" -> "cmpxchgq" replacements.)

The CMPXCHG instruction has the following operands:
- AX (implicit, CompareValue):    input and output
- destination operand (*Value):   input and output
- source operand (ExchangeValue): input

The X64 version of InternalSyncCompareExchange64() attempts to mark both
CompareValue and (*Value) as input/output, but it doesn't use the
appropriate constraints for either operand.

Fix these issues. Furthermore, prefer the short "+" constraint for I/O
operands over the <output-operand-number> constraint that can be applied
to the input instances of I/O operands.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1208
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Acked-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-17 18:44:48 +02:00
7149d40938 MdePkg/BaseSynchronizationLib GCC: fix InternalSyncCompareExchange32()
(This patch is identical to the last one, except for the
InternalSyncCompareExchange16() -> InternalSyncCompareExchange32() and
"cmpxchgw" -> "cmpxchgl" replacements.)

The CMPXCHG instruction has the following operands:
- AX (implicit, CompareValue):    input and output
- destination operand (*Value):   input and output
- source operand (ExchangeValue): input

The IA32 version of InternalSyncCompareExchange32() correctly marks
CompareValue as input/output, but it marks (*Value) only as input.

The X64 version of InternalSyncCompareExchange32() attempts to mark both
CompareValue and (*Value) as input/output, but it doesn't use the
appropriate constraints for either operand.

Fix these issues. Furthermore, prefer the short "+" constraint for I/O
operands over the <output-operand-number> constraint that can be applied
to the input instances of I/O operands.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1208
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Acked-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-17 18:44:46 +02:00
c6fedbd791 MdePkg/BaseSynchronizationLib GCC: fix InternalSyncCompareExchange16()
The CMPXCHG instruction has the following operands:
- AX (implicit, CompareValue):    input and output
- destination operand (*Value):   input and output
- source operand (ExchangeValue): input

The IA32 version of InternalSyncCompareExchange16() correctly marks
CompareValue as input/output, but it marks (*Value) only as input.

The X64 version of InternalSyncCompareExchange16() attempts to mark both
CompareValue and (*Value) as input/output, but it doesn't use the
appropriate constraints for either operand.

Fix these issues. Furthermore, prefer the short "+" constraint for I/O
operands over the <output-operand-number> constraint that can be applied
to the input instances of I/O operands.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1208
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Acked-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-17 18:44:43 +02:00
310ddb638b MdePkg/BaseSynchronizationLib GCC: fix whitespace and comments
The "GccInline.c" files have some inconsistent whitespace, and missing (or
incorrect) operand comments. Fix and unify them.

This patch doesn't change behavior.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1208
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Acked-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-17 18:44:34 +02:00
b7dc8888f3 MdeModulePkg Variable: Fix Timestamp zeroing issue on APPEND_WRITE
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=415

When SetVariable() to a time based auth variable with APPEND_WRITE
attribute, and if the EFI_VARIABLE_AUTHENTICATION_2.TimeStamp in
the input Data is earlier than current value, it will cause timestamp
zeroing.

This issue may bring time based auth variable downgrade problem.
For example:
A vendor released three certs at 2014, 2015, and 2016, and system
integrated the 2016 cert. User can SetVariable() with 2015 cert and
APPEND_WRITE attribute to cause timestamp zeroing first, then
SetVariable() with 2014 cert to downgrade the cert.

This patch fixes this issue.

Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Chao Zhang <chao.b.zhang@intel.com>
Cc: Jian J Wang <jian.j.wang@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Jiewen Yao <jiewen.yao@intel.com>
2018-10-17 11:24:44 +08:00
25d310d9b6 MdeModulePkg/UsbMass: Reject device whose block size is 0 or > 64K
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Star Zeng <star.zeng@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-10-17 11:04:06 +08:00
8894c90d74 MdeModulePkg/Bus/Ufs: Ensure device not return more data than expected
This commit adds checks to make sure the UFS devices do not return more
data than the driver expected.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Star Zeng <star.zeng@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-10-17 11:04:05 +08:00
b2252bab12 MdeModulePkg/UsbBus: Deny when the string descriptor length is odd
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Star Zeng <star.zeng@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-10-17 11:04:04 +08:00
6c46cbbd5e MdeModulePkg/UsbMouse: Don't access key codes when length is wrong
Per USB HID spec, the buffer holding key codes should at least 3-byte
long.
Today's code assumes that the key codes buffer length is longer than
3-byte and unconditionally accesses the key codes buffer.
It's incorrect.
The patch fixes the issue by returning Device Error when the
length is less than 3-byte.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Star Zeng <star.zeng@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-10-17 11:04:03 +08:00
0dd6065520 MdeModulePkg/AbsPointer: Don't access key codes when length is wrong
Per USB HID spec, the buffer holding key codes should at least 3-byte
long.
Today's code assumes that the key codes buffer length is longer than
3-byte and unconditionally accesses the key codes buffer.
It's incorrect.
The patch fixes the issue by returning Device Error when the
length is less than 3-byte.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Star Zeng <star.zeng@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-10-17 11:04:02 +08:00
8bcbe587e7 MdeModulePkg/UsbKb: Don't access key codes when length is wrong
Per USB HID spec, the buffer holding key codes should be 8-byte
long.
Today's code assumes that the key codes buffer length is 8-byte
long and unconditionally accesses the key codes buffer.
It's incorrect.
The patch fixes the issue by returning Device Error when the
length is less than 8-byte.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Star Zeng <star.zeng@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-10-17 11:04:01 +08:00
4d2b506631 SourceLevelDebugPkg/Usb3: Make sure data from HW can fit in buffer
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Hao A Wu <hao.a.wu@intel.com>
Reviewed-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-10-17 11:03:59 +08:00
7fb7259fc0 MdeModulePkg/Usb: Make sure data from HW is no more than expected
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Star Zeng <star.zeng@intel.com>
Cc: Hao A Wu <hao.a.wu@intel.com>
Reviewed-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-10-17 11:03:58 +08:00
70c3c2370a MdeModulePkg/UsbBus: Reject descriptor whose length is bad
Today's implementation doesn't check whether the length of
descriptor is valid before using it.

The patch fixes this issue.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Star Zeng <star.zeng@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-10-17 11:03:57 +08:00
4c034bf62c MdeModulePkg/UsbBus: Fix out-of-bound read access to descriptors
Today's implementation reads the Type/Length field in the USB
descriptors data without checking whether the offset to read is
beyond the data boundary.

The patch fixes this issue.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Star Zeng <star.zeng@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-10-17 11:03:56 +08:00
4f8b2f9d72 MdeModulePkg/UsbMass: Fix integer overflow when BlockSize is 1
UsbBootReadWriteBlocks() and UsbBootReadWriteBlocks16() use a UINT16
local variable to hold the value of
USB_BOOT_MAX_CARRY_SIZE (=0x10000) / BlockSize.
When BlockSize is 1, the UINT16 local variable is set to 0x10000
but the high-16 bits are truncated resulting the final value be 0.

It causes the while-loop in the two functions accesses 0 block in
each loop, resulting the loop never ends.

The patch fixes the two functions to make sure no integer overflow
happens.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Star Zeng <star.zeng@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-10-17 11:03:55 +08:00
e59db6a732 MdeModulePkg/UsbMass: Merge UsbBoot(Read|Write)Blocks(16)
The change doesn't have functionality impact.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Star Zeng <star.zeng@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-10-17 11:03:54 +08:00
76d1c03cbd CorebootPayloadPkg: don't use serial output for Release build
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Wonkyu Kim <wonkyu.kim@intel.com>
Reviewed-by: Maurice Ma <maurice.ma@intel.com>
Reviewed-by: Benjamin You <benjamin.you@intel.com>
2018-10-17 10:35:16 +08:00
6d665168b0 MdeModulePkg/RegularExpressionDxe: Add null pointer check
This is the part of commit 3948c510ed.

Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Dongao Guo <dongao.guo@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-16 15:24:23 +08:00
47f15da160 BaseTools/EOT: Change to call a program instead of calling Python API.
Update the EOT tool to call the program itself instead of calling the Python API when parsing FV images.

Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Hess Chen <hesheng.chen@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
Acked-by: Jaben Carsey <jaben.carsey@intel.com>
2018-10-16 14:52:02 +08:00
ff4d0f851d BaseTools: Support to use struct name as datum type before max size
Original it hard code to use "VOID*", this patch extend it to both
support VOID* and valid struct name.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-16 14:48:49 +08:00
0fab42ba27 BaseTools: Fix bugs use special character in the --pcd option
Cases:
--pcd Token.Name="!"
--pcd Token.Name="\'W&\'"
--pcd Token.Name="2*h"

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2018-10-16 14:45:29 +08:00
03c36c36a3 BaseTools: Add check for the string type whether is same
Relational and equality operators require both operands to be of
the same type.
Treat the string 'A' and "A" as same type, but for "A" and L"A"
are not same type since one is general string, another is unicode
string.

True:'A'<'B', "A"<"B" 'A'<"B", L'A'<L'B', L"A"<L"B", L'A'<L"B"
Error:'A'<L'B', 'A'<L"B", "A'<L'B'

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2018-10-16 14:44:50 +08:00
aa52648c1e BaseTools:Fix issue caused by 84a52d4d03
The commit 84a52d4d03 will introduce
the new issue "Invalid offset value for Dynamic Vpd PCD". This patch
will fix the new issue, add 'DEFAULT' sku in to the AvailableSkuIdSet
variable.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: ZhiqiangX Zhao <zhiqiangx.zhao@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-16 08:59:51 +08:00
a96133d2d1 BaseTools: Convert string value of void* pcd in command line to array.
For void* type pcd in command line, if its value is string, then
convert the string value to array format. For example,
build --pcd gUefiOvmfPkgTokenSpaceGuid.PcdTest="c",
convert the pcd value from "c" to "{0x63,0x00}".

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: ZhiqiangX Zhao <zhiqiangx.zhao@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-16 08:59:50 +08:00
0caf903913 BaseTools: Code should not update the variable that save the raw data.
Code should not update the value of variable which save the original
raw content of meta file.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: ZhiqiangX Zhao <zhiqiangx.zhao@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-16 08:59:50 +08:00
e451aaa637 BaseTools: add ASSERT checker for array buffer value assignment.
V3:
Update the error message for array checker.
V2:
1. Add comments for each ASSERT.
2. ASSERT need to skip the case of array size of array as zero. For
example, TestArray[] in struct in header file.
V1:
For structure PCD,
1. use compiler time assert to check the array index, report error
if array index exceeds the array number.
2. use compiler time assert to check the array size, report error
if the user declared size in header file is smaller than the user
defined in DEC/DSC file.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: ZhiqiangX Zhao <zhiqiangx.zhao@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-16 08:59:49 +08:00
272ecccd79 BaseTools: Fix regression boot hang issue by commit 0e982cf03d
Fix regression boot hang issue by commit
0e982cf03d

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-10-15 18:21:28 +08:00
b1cd4061e3 MdeModulePkg BrotliCustomDecompressLib: Don't build it for EBC arch
BrotliCustomDecompressLib has the definition with float type. But,
Floating-point types are not supported by EBC compiler.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
Cc: Star Zeng <star.zeng@intel.com>
Cc: Bi Dandan <dandan.bi@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-10-15 15:55:55 +08:00
2215855957 MdeModulePkg RegularExpressionDxe: Enable STDARG for variable arguments
Set macro for variable arguments, and remove the hard code definition
to avoid the potential duplicated definition.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Dongao Guo <dongao.guo@intel.com>
2018-10-15 15:55:54 +08:00
3948c510ed MdeModulePkg/RegularExpressionDxe: Add null pointer check
There are five check not necessary in logic ,just for pass static
analysis. More detail please refer to comment in code.
And the rest changes are all accepted by owner, they are reasonable.

Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Dongao Guo <dongao.guo@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-15 15:55:53 +08:00
df8be9e50f MdeModulePkg/RegularExpressionDxe: Miss null pointer check
Oniguruma https://github.com/kkos/oniguruma
this change is merged from oniguruma develop branch.
from commit 1db8a2726dfad0401f928cb8474bd770f07040a7.

Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Dongao Guo <dongao.guo@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-15 15:55:53 +08:00
a5def17714 MdeModulePkg/RegularExpressionDxe: Miss null pointer check
Oniguruma https://github.com/kkos/oniguruma
this change is merged from oniguruma develop branch.
from commit ea36d810f1d9b28f3ef20bd8d453bea2f7fb598b

Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Dongao Guo <dongao.guo@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-15 15:55:52 +08:00
16bd7c35df MdeModulePkg/RegularExpressionDxe: Miss null pointer check
Oniguruma https://github.com/kkos/oniguruma
this change is merged from oniguruma develop branch.
from commit 396a757dffafc0c7eb269433c29a0ba961d73ad6.

Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Dongao Guo <dongao.guo@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-15 15:55:51 +08:00
f1400101a7 BaseTools: Add --uefi option to enable UefiCompress method
Add one new option --uefi to enable UefiCompress.
(re-add this patch since it be reverted in Python3 migration patches,
but this patch is not related with Python3)

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-10-15 15:37:45 +08:00
f88113789b BaseTools: Enhance the *P Flag display for Structure Pcd
Cover the case:
1.only define the structure Pcd in DEC file, it should not have any
Flag.
2.In the DEC file and DSC file only have the PCD's default value, and
without the field value, it should have *P Flag.
(re-add this patch since it be reverted in Python3 migration patches,
but this patch is not related with Python3)

Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-15 15:26:48 +08:00
425084cd45 MdeModulePkg PcdDxe: ASSERT PcdSetNvStoreDefaultId set
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1245

In current implementation and according to the description in
MdeModulePkg.dec, PcdSetNvStoreDefaultId should be set in PEI
phase to take effect.

This patch ASSERTs PcdSetNvStoreDefaultId set in PcdDxe to alert
the invalid operation.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Jian J Wang <jian.j.wang@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Jian J Wang <jian.j.wang@intel.com>
2018-10-15 14:11:31 +08:00
04aa423c8d BaseTools: increment build generate inf file lost .depex file
increment build generate inf file in output directory lost .depex file info.
(re-add this patch since it be reverted in Python3 migration patches,
but this check is not related with Python3)

Fixes: https://bugzilla.tianocore.org/show_bug.cgi?id=1244
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-15 10:28:05 +08:00
6974c784cb BaseTools: remove the not used PyUtility file
the PyUtility is not used, so we remove it.
(re-add this patch since it be reverted in Python3 migration patches,
but this check is not related with Python3)

Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-15 10:23:36 +08:00
0e982cf03d BaseTools: do basic check in FvImage with header size and signature
Add some basic check in FvImage with header size and signature.

Fixes: https://bugzilla.tianocore.org/show_bug.cgi?id=1181
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-10-15 10:16:07 +08:00
55e8ff01af UefiCpuPkg/S3Resume2Pei: disable paging before creating new page table.
V5:
1. Add ASSERT to indicate this assumption that environment is 32 bit mode.
2. Add description in INF about this driver's expected result
   in different environment.

V4:
Only disable paging when it is enabled.

V3 changes:
No need to change inf file.

V2 changes:
Only disable paging in 32 bit mode, no matter it is enable or not.

V1 changes:
PEI Stack Guard needs to enable paging. This might cause #GP if code
trying to write CR3 register with PML4 page table while the processor
is enabled with PAE paging.

Simply disabling paging before updating CR3 can solve this conflict.

It's an regression caused by change: 0a0d5296e4

BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1232

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Jian J Wang <jian.j.wang@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Regression-tested-by: Laszlo Ersek <lersek@redhat.com>
2018-10-15 08:36:40 +08:00
1ccc4d895d Revert BaseTools: PYTHON3 migration
This reverts commit 6693f359b3c213513c5096a06c6f67244a44dc52..
678f851312.

Python3 migration is the fundamental change. It requires every developer
to install Python3. Before this migration, the well communication and wide
verification must be done. But now, most people is not aware of this change,
and not try it. So, Python3 migration is reverted and be moved to edk2-staging
Python3 branch for the edk2 user evaluation.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
2018-10-15 08:29:14 +08:00
678f851312 BaseTools/Tests: Update GNUmakefile to use python3 variable
Cover the case user do make -C BaseTools before run the .edksetup.sh
file.

Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Cc: Leif Lindholm <leif.lindholm@linaro.org>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
2018-10-14 00:50:35 +08:00
913795f7a2 BaseTools: Not compare the VOID* difference in the DSC and DEC file
For structure Pcd, the type defined in the DEC file is the struct name
while if this Pcd used in the DSC file, it should add VOID* keywords
when it have max size info. so this patch filter the type compare for
this case.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-13 10:07:16 +08:00
15298db95c BaseTools: Enhance the *P Flag display for Structure Pcd
Cover the case:
1.only define the structure Pcd in DEC file, it should not have any
Flag.
2.In the DEC file and DSC file only have the PCD's default value, and
without the field value, it should have *P Flag.

Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-13 10:06:37 +08:00
27f38c6d6e BaseTools: remove the not used PyUtility file
the PyUtility is not used, so we remove it.

Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-13 10:06:12 +08:00
b87099f336 BaseTools: increment build generate inf file lost .depex file
increment build generate inf file in output directory lost .depex file info.

Fixes: https://bugzilla.tianocore.org/show_bug.cgi?id=1244
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-13 10:01:36 +08:00
ea787b6b55 BaseTools: update Test scripts support python3
update Test scripts support python3

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-13 09:57:19 +08:00
22c4de1ac8 BaseTools/Ecc/EOT: Add Python 3 support on ECC and EOT tools.
1. Add Python 3 support on ECC and EOT tools
2. Add C grammar file of ANTLR4 and fix some bugs

Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Hess Chen <hesheng.chen@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-13 09:57:18 +08:00
472eb3b896 BaseTools: Add --uefi option to enable UefiCompress method
Add one new option --uefi to enable UefiCompress.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-13 09:57:17 +08:00
1b2e077260 BaseTools/UPT: Porting UPT Tool from Python2 to Python3
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-13 09:57:16 +08:00
9d7e1e56a7 BaseTools/Scripts: Porting PackageDocumentTools code to use Python3
Porting PackageDocumentTools code to Python3
DoxyGen 1.8.6
wxpython 4.0.3

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-13 09:57:16 +08:00
c9486c8c9d BaseTools: Update Makefile to ignore python tools make
ignore make python tools, replace PYTHON_HOME by PYTHON3

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-13 09:57:15 +08:00
d25c096d27 BaseTools: Use PYTHON3 variable instead of PYTHON_HOME
Use PYTHON3 variable instead of PYTHON_HOME path

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-13 09:57:15 +08:00
04c47cd401 BaseTools: Update windows and linux run scripts file to use Python3
Modify windows script, PosixLike script, edksetup.sh, edksetup.bat to
use Python3

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-13 09:57:14 +08:00
8be15c61e8 BaseTools: Fix some build and report file issue
1. increment build not skip make file when not change any file
2. report file generate abundant blank line
3. Build encounter Database is locked on some platform, using database
   auto commit
4. Fv BaseAddress must have if set

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-13 09:57:13 +08:00
0e3bfc6f7a BaseTools: Rename get_bytes_le() to bytes_le
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-13 09:56:56 +08:00
60dc40db2a BaseTools: Update argparse arguments since it not have version now
argparse.ArgumentParser not have version parameter

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-13 09:54:54 +08:00
2e300969ae BaseTools: change the Division Operator in the expression
PEP 238 -- Changing the Division Operator
x/y to return a reasonable approximation of the mathematical result
    of the division ("true division")
x//y to return the floor ("floor division")

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-13 09:54:53 +08:00
00fcce9153 BaseTools: do the list and iterator translation
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-13 09:54:52 +08:00
86e6cf98a8 BaseTools: Handle the bytes and str difference
Deal with bytes and str is different, remove the unicode()
Using utcfromtimestamp instead of fromtimestamp.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-13 09:54:07 +08:00
a09f4c91f7 BaseTools: fix the open file's read and write bugs
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-13 09:50:45 +08:00
fe3991d635 BaseTools: use map and filter to replace the itertools function
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-13 09:50:44 +08:00
4ce4f757d7 BaseTools: remove the super() function argument
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-13 09:50:43 +08:00
5135cc4852 BaseTools: replace the list iteritems by items
replace the list iteritems by items in Python3.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-13 09:50:43 +08:00
d3678942b4 BaseTools: Rename raw_input() to input()
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-13 09:50:42 +08:00
8371d87412 BaseTools: Rename long() to int()
Because the long() function was not exist in Python3.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-13 09:50:42 +08:00
261eee25de BaseTools: Rename xrange() to range()
Because the xrange() was not exist in Python3

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-13 09:50:41 +08:00
487062c048 BaseTools: Remove the "from __future__ import" items
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-13 09:50:40 +08:00
6693f359b3 BaseTools: Remove the outdated verbose parameter of namedtuple
the verbose is outdated.
in 3.6: The verbose and rename parameters became keyword-only arguments

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-10-13 09:50:39 +08:00
301402fa47 ArmPkg: Add support for GICv4
Updated Redistributor base calculation to allow for the fact that
GICv4 has 2 additional 64KB frames (for VLPI and a reserved frame).
The code now tests the VLPIS bit in the GIC Redistributor Type
Register (GICR_TYPER) and calculates the Redistributor granularity
accordingly.

The code changes are:
  GICR_TYPER register fields, etc, added to the header.
  Loop updated to pay attention to GICR_TYPER.Last.
  Derive frame "stride" size from GICR_TYPER.VLPIS.

Note: The assumption is that the redistributors are adjacent for
all CPUs. However this may not be the case for NUMA systems.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Sami Mujawar <sami.mujawar@arm.com>
Reviewed-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
2018-10-12 17:40:33 +02:00
bbce001515 MdeModulePkg/SdMmcPciHcDxe: Allow additional SDHCI versions
Clock control is similar in SDHCI versions greater then version 2. Add
support for more recent versions of the controller specification.

SD card support for 1.8V is also present in controller versions 3 and
greater.

https://bugzilla.tianocore.org/show_bug.cgi?id=1233

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jeff Brasen <jbrasen@nvidia.com>
Reviewed-by: Hao Wu <hao.a.wu@intel.com>
2018-10-12 15:46:10 +08:00
bdf038ccff MdeModulePkg/SdMmcPciHcDxe: Add controller version defines
Add SDHCI controller defines, this is useful as the version in the
register does not explictly map to a specification version. For example
vesion 4.10 of the specification is version 0x04.

https://bugzilla.tianocore.org/show_bug.cgi?id=1233

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jeff Brasen <jbrasen@nvidia.com>
Reviewed-by: Hao Wu <hao.a.wu@intel.com>
2018-10-12 15:46:04 +08:00
9343d0a1cd MdePkg: Use VENDOR_DEVICE_PATH structure for Debug Port device path
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1229

When converting DebugPort device path from text,
current code use VENDOR_DEFINED_MESSAGING_DEVICE_PATH structure
for Debug port device node.
typedef struct {
  EFI_DEVICE_PATH_PROTOCOL  Header;
  EFI_GUID                  Guid;
  UINT8                     VendorDefinedData[1];
} VENDOR_DEFINED_MESSAGING_DEVICE_PATH;

And Debugport Device Path is a vendor-defined messaging
device path with no data, only a GUID. So it's better to
use VENDOR_DEVICE_PATH to create the Debug port device node.
typedef struct {
  EFI_DEVICE_PATH_PROTOCOL        Header;
  EFI_GUID                        Guid;
} VENDOR_DEVICE_PATH;

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Dandan Bi <dandan.bi@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-10-12 14:46:01 +08:00
b1df6c2582 ShellPkg-Cd: Ensure all valid cd targets are handled properly
ShellPkg-Cd: Ensure all valid cd targets are handled properly

Make sure that PathCleanUpDirectories() is called on all valid targets
of the cd command.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jim Dailey <jim_dailey@dell.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-10-12 13:45:34 +08:00
7177be0bd8 MdePkg-BaseLib: Fix PathCleanUpDirectories() error involving "\..\.."
MdePkg-BaseLib: Fix PathCleanUpDirectories() error involving "\..\.."

The loop that removes "xxxx\..\" errs when multiple "\.." sequences are
in the path.  Before this change the code would modify a path like
"FS0:\efi\tools\..\.." to "FS0:\efi\\.." and then to "FS0:\efi\", but
the correct path is "FS0:\".

You can test the effect of this change in the shell by setting the
current directory to something like FS0:\efi\boot and then executing
the command "ls ..\..".  Before the change you will see the files in
the FS0:\efi directory; after the change, you will see the files in
the root directory of FS0:.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jim Dailey <jim_dailey@dell.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-10-12 11:46:17 +08:00
8122c6bc8b MdeModulePkg/RegularExpressionDxe:omit unused variable
comment unused variable to avoid warning,and modify inf build option.

Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Dongao Guo <dongao.guo@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-11 13:19:43 +08:00
4ee787cc1b MdeModulePkg/RegularExpressionDxe:disable wraning to pass gcc4.8 build
There are three warnings reported by GCC 4.8 and the later GCC release
are workaround with them.
And all the three warnings are invalid,so I just disable warnings rather
than fix them at now.

Following is the analysis from Laszlo Ersek.
(1)

> MdeModulePkg/Universal/RegularExpressionDxe/Oniguruma/regcomp.c: In
> function 'compile_length_tree':
> MdeModulePkg/Universal/RegularExpressionDxe/Oniguruma/regcomp.c:1516:7:
> warning: 'len' may be used uninitialized in this function
> [-Wmaybe-uninitialized]
>    int len;
>        ^

I think this is an invalid warning; the type of the controlling expression
(node->type) is enum GimmickType, and the case labels cover all values of
the enum. An assert(0) could be added, I guess, but again, upstream
Oniguruma would be justified to reject the idea.

(2)

> MdeModulePkg/Universal/RegularExpressionDxe/Oniguruma/regparse.c: In
> function 'parse_callout_args.isra.10.constprop.30':
> MdeModulePkg/Universal/RegularExpressionDxe/Oniguruma/regparse.c:6753:25:
> warning: 'rl' may be used uninitialized in this function
> [-Wmaybe-uninitialized]
>   vals[n].l = rl;
>               ^

This warning is invalid, given:

  6749    if (cn > 0) {
  6750      long rl;
  6751      r = parse_long(enc, buf, bufend, 1, LONG_MAX, &rl);
  6752      if (r == ONIG_NORMAL) {
  6753        vals[n].l = rl;

Because parse_long() only returns ONIG_NORMAL after it sets (*rl).

(3)

> MdeModulePkg/Universal/RegularExpressionDxe/Oniguruma/regparse.c: In
> function 'parse_callout_of_name.constprop.29':
> MdeModulePkg/Universal/RegularExpressionDxe/Oniguruma/regparse.c:6861:38:
> warning: 'tag_end' may be used uninitialized in this function
> [-Wmaybe-uninitialized]
>      if (! is_allowed_callout_tag_name(enc, tag_start, tag_end))

This warning is also invalid, given:

  6852    if (c == '[') {
  6853      if (PEND) return ONIGERR_END_PATTERN_IN_GROUP;
  6854      tag_start = p;
  6855      while (! PEND) {
  6856        if (PEND) return ONIGERR_END_PATTERN_IN_GROUP;
  6857        tag_end = p;
  6858        PFETCH_S(c);
  6859        if (c == ']') break;
  6860      }
  6861      if (! is_allowed_callout_tag_name(enc, tag_start, tag_end))
  6862        return ONIGERR_INVALID_CALLOUT_TAG_NAME;
  6863

To see that, first we should note:
 #define PEND         (p < end ?  0 : 1)

therefore PEND doesn't change if neither "p" nor "end" change.

Second, when we reach line 6855 (the "while") for the very first time,
(!PEND) is certainly true (i.e., PEND is false), because otherwise we
would have bailed at line 6853. Therefore we reach line 6857, and assign
"tag_end". Regardless of whether we iterate zero or more *additional*
times around the loop, "tag_end" will have been set, whenever we reach
line 6861.

Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Dongao Guo <dongao.guo@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
2018-10-11 13:19:41 +08:00
b9c055f72b IntelFsp2Pkg/GenCfgOpt.py: Support PCD input from command line
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1231

Build system already support override PCD value by command
line so add this support to GenCfgOpt.py
Also update revision to 0.53

Test: Verified UPD header files generated can reflect different
      PCD values from --pcd build command input

Cc: Jiewen Yao <Jiewen.yao@intel.com>
Cc: Gao Liming <liming.gao@intel.com>
Cc: Zhu Yonghong <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Chasel Chiu <chasel.chiu@intel.com>
Reviewed-by: Jiewen Yao <Jiewen.yao@intel.com>
Reviewed-by: Zhu Yonghong <yonghong.zhu@intel.com>
2018-10-11 07:54:08 +08:00
1a3bee2082 MdeModulePkg/NonDiscoverablePciDeviceDxe: expose unique B/D/F identifiers
Currently, the implementation of EFI_PCI_IO_PROTOCOL::GetLocation()
in NonDiscoverablePciDeviceDxe returns the same set of dummy values
for each instance of the NON_DISCOVERABLE_DEVICE protocol that it
attaches itself to. However, this turns out to be causing problems
in cases where software (such as the ARM Compliance Test Suite [ACS])
attempts to use these values to uniquely identify controllers, since
the collisions create ambiguity in this regard.

So let's modify GetLocation() to return an arbitrary bus/device tuple
on segment 0xff instead. This is guaranteed not to clash with other
non-discoverable PCI devices, and highly unlikely to clash with real
PCIe devices.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Tested-by: Marcin Wojtas <mw@semihalf.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-10-10 11:14:29 +02:00
2730470f9d MdeModulePkg: Update Brotli DecompressLib to the latest v1.0.6
https://bugzilla.tianocore.org/show_bug.cgi?id=1201
Update Brotli to the latest version 1.0.6
https://github.com/google/brotli

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-10-09 22:28:16 +08:00
dd4f667e70 BaseTools: Update Brotli Compress to the latest one 1.0.6
https://bugzilla.tianocore.org/show_bug.cgi?id=1201
Update Brotli to the latest version 1.0.6
https://github.com/google/brotli
Verify VS2017, GCC5 build.
Verify Decompression boot functionality.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-10-09 22:27:32 +08:00
78af0984b4 MdePkg: Add PciRoot/PcieRoot text for ACPI Expanded Device Path
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1228

According to UEFI spec,for ACPI Expanded Device Path
when HID=PNP0A03 or CID=PNP0A03 and HID != PNP0A08,
the device path node can be displayed as: PciRoot(UID|UIDSTR)
When HID=PNP0A08 or CID=PNP0A08, the device path node can be
displayed as: PcieRoot(UID|UIDSTR). But current code miss the
code logic.

This commit is to do the enhancement.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Dandan Bi <dandan.bi@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-10-09 22:20:19 +08:00
fb4bea551e MdePkg: Correct condition check for AcpiExp text format
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1226

According to UEFI Spec, for ACPI Expanded Device Path,
when HIDSTR=empty, CIDSTR=empty, UID STR!=empty,
the ACPI Expanded Device Path node can be displayed as
AcpiExp(HID,CID,UIDSTR) format.
And if UID is 0 and UIDSTR is empty, then use AcpiEx format.

This patch is to correct the condition check to follow UEFI
Spec when convert the device path node to the AcpiExp text
format.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Dandan Bi <dandan.bi@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-10-09 22:20:19 +08:00
05fe752589 MdePkg: Correct the string order of ACPI Expanded Device Path
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1227

According to UEFI Spec, ACPI Expanded Device Path can be display
AcpiEx(HID|HIDSTR,(CID|CIDSTR,UID|UIDSTR)), but current code display
UID|UIDSTR before CID|CIDSTR.
This patch is to fix this issue.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Dandan Bi <dandan.bi@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-10-09 22:20:18 +08:00
959be180e1 MdePkg: Correct the string expression of UTF8 vendor device path
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1225

According to UEFI spec, the string expression of UTF8 vendor
device node should be displayed as: VenUtf8(). Current code
display it as: VenUft8() by mistake when convert device
path node to text.

This commit is to fix this bug.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Dandan Bi <dandan.bi@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-10-09 22:20:18 +08:00
7f8aabef15 MdeModulePkg/HiiDB: Fix incorrect structure convention for checkbox
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1224

When covert IFR binary to EFI_IFR_CHECKBOX structure,
Current code has following incorrect code logic:
IfrCheckBox = (EFI_IFR_CHECKBOX *) (IfrOpHdr + 1);
The correct one should be:
IfrCheckBox = (EFI_IFR_CHECKBOX *) IfrOpHdr;

1. The bug is in function "UpdateDefaultSettingInFormPackage()"
which is to update the default setting of some HII Questions
in the IFR binary data. So it only has impact when platform
overrides default setting in HII VarStore through DynamicHii
PCD setting in Platform DSC file. If platform doesn't
override default setting, it has no impact.

2. The implementation updates the "Flags" filed in the
EFI_IFR_CHECKBOX structure to update the default
setting of checkbox.
If using "IfrCheckBox = (EFI_IFR_CHECKBOX *) (IfrOpHdr + 1);"
when wants to update the " Flags" filed in checkbox,
but in fact it will update the opcode binary
data(opcode binary length) behind checkbox binary.
And then it will cause Browser can't parse the IFR
binary data correctly. And then the possible symptom
is that some HII Question and forms may be not parsed
and then cannot be shown.

This patch is to fix this bug.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Star Zeng <star.zeng@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Dandan Bi <dandan.bi@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-09 22:20:08 +08:00
50431b9cc7 BaseTools LzmaCompress: Fix GCC warning misleading-indentation
GCC 6 or above reports the warning misleading-indentation.
This patch fixes it.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Star Zeng <star.zeng@intel.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1234
[lersek@redhat.com: reference the BZ that got reported meanwhile]
2018-10-09 11:47:25 +02:00
39bbbc8759 IntelFrameworkModulePkg Lzma: Update LZMA SDK version to 18.05
https://bugzilla.tianocore.org/show_bug.cgi?id=1006
New formal release in https://www.7-zip.org/sdk.html is 18.05.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-10-09 09:31:11 +08:00
f0737de8d9 MdeModulePkg Lzma: Update LZMA SDK version to 18.05
https://bugzilla.tianocore.org/show_bug.cgi?id=1006
New formal release in https://www.7-zip.org/sdk.html is 18.05.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-10-09 09:31:10 +08:00
5ec5a236d1 BaseTools Lzma: Update LZMA SDK version to 18.05
https://bugzilla.tianocore.org/show_bug.cgi?id=1006
New formal release in https://www.7-zip.org/sdk.html is 18.05.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-10-09 09:31:09 +08:00
a7ab1c315c ShellPkg/TftpDynamicCommand: Correct comments to align with the input parameter.
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1230

Cc: Carsey Jaben <jaben.carsey@intel.com>
Cc: Fu Siyuan <siyuan.fu@intel.com>
Cc: Ye Ting <ting.ye@intel.com>
Cc: Bi Dandan <dandan.bi@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Wu Jiaxin <jiaxin.wu@intel.com>
Reviewed-by: Dandan Bi <dandan.bi@intel.com>
Reviewed-by: Fu Siyuan <siyuan.fu@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2018-10-09 08:43:43 +08:00
e8601e972f NetworkPkg/UefiPxeBcDxe: Correct comments to align with the input parameter.
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1230

Cc: Fu Siyuan <siyuan.fu@intel.com>
Cc: Ye Ting <ting.ye@intel.com>
Cc: Bi Dandan <dandan.bi@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Wu Jiaxin <jiaxin.wu@intel.com>
Reviewed-by: Dandan Bi <dandan.bi@intel.com>
Reviewed-by: Fu Siyuan <siyuan.fu@intel.com>
2018-10-09 08:43:35 +08:00
fab334b278 MdeModulePkg/Tcp4Dxe: Remove the trailing white space in one line.
Cc: Fu Siyuan <siyuan.fu@intel.com>
Cc: Ye Ting <ting.ye@intel.com>
Cc: Bi Dandan <dandan.bi@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Wu Jiaxin <jiaxin.wu@intel.com>
Reviewed-by: Dandan Bi <dandan.bi@intel.com>
Reviewed-by: Fu Siyuan <siyuan.fu@intel.com>
2018-10-09 08:43:28 +08:00
d20ae95a13 OvmfPkg/PlatformPei: clear CPU caches
This is for conformance with the TCG "Platform Reset Attack Mitigation
Specification". Because clearing the CPU caches at boot doesn't impact
performance significantly, do it unconditionally, for simplicity's
sake.

Flush the cache on all logical processors, thanks to
EFI_PEI_MP_SERVICES_PPI and CacheMaintenanceLib.

Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Cc: Anthony Perard <anthony.perard@citrix.com>
Cc: Julien Grall <julien.grall@linaro.org>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Tested-by: Anthony PERARD <anthony.perard@citrix.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Regression-tested-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Michael D Kinney <michael.d.kinney@intel.com>
[lersek@redhat.com: remove bogus Message-Id line from commit msg]
2018-10-05 22:02:55 +02:00
c0b1f749ef ShellPkg: Create a homefilesystem environment variable
Create a homefilesystem environment variable whose value is the file
system on which the executing shell is located. For example: "FS14:".

This eliminates the need for people to have to try and find the "boot"
file system in their startup script.  After this change they can simply
execute %homefilesystem% to set the cwd to the root of the file system
where the shell is located.

A future enhancement could be to add "homefilesystem" to the list of
predefined, read-only variables listed in the EfiShellSetEnv function of
file ShellProtocol.c

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jim Dailey <jim_dailey@dell.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2018-10-03 11:09:47 -07:00
c526dcd40f BaseTools: refactor the error for PCD value is negative or exceed max
refactor the error handling for the PCD value that is negative or it
exceed the max value.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-09-30 16:05:49 +08:00
5b02be4d9a UefiCpuPkg/PiSmmCpuDxeSmm: [CVE-2017-5753] Fix bounds check bypass
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=1194

Speculative execution is used by processor to avoid having to wait for
data to arrive from memory, or for previous operations to finish, the
processor may speculate as to what will be executed.

If the speculation is incorrect, the speculatively executed instructions
might leave hints such as which memory locations have been brought into
cache. Malicious actors can use the bounds check bypass method (code
gadgets with controlled external inputs) to infer data values that have
been used in speculative operations to reveal secrets which should not
otherwise be accessed.

It is possible for SMI handler(s) to call EFI_SMM_CPU_PROTOCOL service
ReadSaveState() and use the content in the 'CommBuffer' (controlled
external inputs) as the 'CpuIndex'. So this commit will insert AsmLfence
API to mitigate the bounds check bypass issue within SmmReadSaveState().

For SmmReadSaveState():

The 'CpuIndex' will be passed into function ReadSaveStateRegister(). And
then in to ReadSaveStateRegisterByIndex().

With the call:
ReadSaveStateRegisterByIndex (
  CpuIndex,
  SMM_SAVE_STATE_REGISTER_IOMISC_INDEX,
  sizeof(IoMisc.Uint32),
  &IoMisc.Uint32
  );

The 'IoMisc' can be a cross boundary access during speculative execution.
Later, 'IoMisc' is used as the index to access buffers 'mSmmCpuIoWidth'
and 'mSmmCpuIoType'. One can observe which part of the content within
those buffers was brought into cache to possibly reveal the value of
'IoMisc'.

Hence, this commit adds a AsmLfence() after the check of 'CpuIndex'
within function SmmReadSaveState() to prevent the speculative execution.

A more detailed explanation of the purpose of commit is under the
'Bounds check bypass mitigation' section of the below link:
https://software.intel.com/security-software-guidance/insights/host-firmware-speculative-execution-side-channel-mitigation

And the document at:
https://software.intel.com/security-software-guidance/api-app/sites/default/files/337879-analyzing-potential-bounds-Check-bypass-vulnerabilities.pdf

Cc: Michael D Kinney <michael.d.kinney@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Jiewen Yao <jiewen.yao@intel.com>
Reviewed-by: Eric Dong <eric.dong@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Regression-tested-by: Laszlo Ersek <lersek@redhat.com>
2018-09-30 13:06:42 +08:00
e83d841fdc MdeModulePkg/Variable: [CVE-2017-5753] Fix bounds check bypass
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=1194

Speculative execution is used by processor to avoid having to wait for
data to arrive from memory, or for previous operations to finish, the
processor may speculate as to what will be executed.

If the speculation is incorrect, the speculatively executed instructions
might leave hints such as which memory locations have been brought into
cache. Malicious actors can use the bounds check bypass method (code
gadgets with controlled external inputs) to infer data values that have
been used in speculative operations to reveal secrets which should not
otherwise be accessed.

This commit will focus on the SMI handler(s) registered within the
Variable\RuntimeDxe driver and insert AsmLfence API to mitigate the
bounds check bypass issue.

For SMI handler SmmVariableHandler():

Under "case SMM_VARIABLE_FUNCTION_GET_VARIABLE:",
'SmmVariableHeader->NameSize' can be a potential cross boundary access of
the 'CommBuffer' (controlled external input) during speculative execution.

This cross boundary access is later used as the index to access array
'SmmVariableHeader->Name' by code:
"SmmVariableHeader->Name[SmmVariableHeader->NameSize/sizeof (CHAR16) - 1]"
One can observe which part of the content within array was brought into
cache to possibly reveal the value of 'SmmVariableHeader->NameSize'.

Hence, this commit adds a AsmLfence() after the boundary/range checks of
'CommBuffer' to prevent the speculative execution.

And there are 2 similar cases under
"case SMM_VARIABLE_FUNCTION_SET_VARIABLE:" and
"case SMM_VARIABLE_FUNCTION_VAR_CHECK_VARIABLE_PROPERTY_GET:" as well.
This commits also handles them.

Also, under "case SMM_VARIABLE_FUNCTION_SET_VARIABLE:",
'(UINT8 *)SmmVariableHeader->Name + SmmVariableHeader->NameSize' points to
the 'CommBuffer' (with some offset) and then passed as parameter 'Data' to
function VariableServiceSetVariable().

Within function VariableServiceSetVariable(), there is a sanity check for
EFI_VARIABLE_AUTHENTICATION_2 descriptor for the data pointed by 'Data'.
If this check is speculatively bypassed, potential cross-boundary data
access for 'Data' is possible to be revealed via the below function calls
sequence during speculative execution:

AuthVariableLibProcessVariable()
ProcessVarWithPk() or ProcessVarWithKek()

Within function ProcessVarWithPk() or ProcessVarWithKek(), for the code
"PayloadSize = DataSize - AUTHINFO2_SIZE (Data);", 'AUTHINFO2_SIZE (Data)'
can be a cross boundary access during speculative execution.

Then, 'PayloadSize' is possible to be revealed by the function call
sequence:

AuthServiceInternalUpdateVariableWithTimeStamp()
mAuthVarLibContextIn->UpdateVariable()
VariableExLibUpdateVariable()
UpdateVariable()
CopyMem()

Hence, this commit adds a AsmLfence() after the sanity check for
EFI_VARIABLE_AUTHENTICATION_2 descriptor upon 'Data' within function
VariableServiceSetVariable() to prevent the speculative execution.

Also, please note that the change made within function
VariableServiceSetVariable() will affect DXE as well. However, since we
only focuses on the SMM codes, the commit will introduce a new module
internal function called VariableLoadFence() to handle this. This internal
function will have 2 implementations (1 for SMM, 1 for DXE). For the SMM
implementation, it is a wrapper to call the AsmLfence() API; for the DXE
implementation, it is empty.

A more detailed explanation of the purpose of commit is under the
'Bounds check bypass mitigation' section of the below link:
https://software.intel.com/security-software-guidance/insights/host-firmware-speculative-execution-side-channel-mitigation

And the document at:
https://software.intel.com/security-software-guidance/api-app/sites/default/files/337879-analyzing-potential-bounds-Check-bypass-vulnerabilities.pdf

Cc: Jiewen Yao <jiewen.yao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Regression-tested-by: Laszlo Ersek <lersek@redhat.com>
2018-09-30 13:06:42 +08:00
ee65b84e76 MdeModulePkg/SmmLockBox: [CVE-2017-5753] Fix bounds check bypass
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=1194

Speculative execution is used by processor to avoid having to wait for
data to arrive from memory, or for previous operations to finish, the
processor may speculate as to what will be executed.

If the speculation is incorrect, the speculatively executed instructions
might leave hints such as which memory locations have been brought into
cache. Malicious actors can use the bounds check bypass method (code
gadgets with controlled external inputs) to infer data values that have
been used in speculative operations to reveal secrets which should not
otherwise be accessed.

This commit will focus on the SMI handler(s) registered within the
SmmLockBox driver and insert AsmLfence API to mitigate the
bounds check bypass issue.

For SMI handler SmmLockBoxHandler():

Under "case EFI_SMM_LOCK_BOX_COMMAND_SAVE:", the 'CommBuffer' (controlled
external inputs) is passed to function SmmLockBoxSave().

'TempLockBoxParameterSave.Length' can be a potential cross boundary access
of the 'CommBuffer' during speculative execution. This cross boundary
access is later passed as parameter 'Length' into function SaveLockBox().

Within function SaveLockBox(), the value of 'Length' can be inferred by
code:
"CopyMem ((VOID *)(UINTN)SmramBuffer, (VOID *)(UINTN)Buffer, Length);".
One can observe which part of the content within 'Buffer' was brought into
cache to possibly reveal the value of 'Length'.

Hence, this commit adds a AsmLfence() after the boundary/range checks of
'CommBuffer' to prevent the speculative execution.

And there is a similar case under "case EFI_SMM_LOCK_BOX_COMMAND_UPDATE:"
function SmmLockBoxUpdate() as well. This commits also handles it.

A more detailed explanation of the purpose of commit is under the
'Bounds check bypass mitigation' section of the below link:
https://software.intel.com/security-software-guidance/insights/host-firmware-speculative-execution-side-channel-mitigation

And the document at:
https://software.intel.com/security-software-guidance/api-app/sites/default/files/337879-analyzing-potential-bounds-Check-bypass-vulnerabilities.pdf

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Jiewen Yao <jiewen.yao@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Regression-tested-by: Laszlo Ersek <lersek@redhat.com>
2018-09-30 13:06:42 +08:00
cb54cd2463 MdeModulePkg/FaultTolerantWrite:[CVE-2017-5753]Fix bounds check bypass
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=1194

Speculative execution is used by processor to avoid having to wait for
data to arrive from memory, or for previous operations to finish, the
processor may speculate as to what will be executed.

If the speculation is incorrect, the speculatively executed instructions
might leave hints such as which memory locations have been brought into
cache. Malicious actors can use the bounds check bypass method (code
gadgets with controlled external inputs) to infer data values that have
been used in speculative operations to reveal secrets which should not
otherwise be accessed.

This commit will focus on the SMI handler(s) registered within the
FaultTolerantWriteDxe driver and insert AsmLfence API to mitigate the
bounds check bypass issue.

For SMI handler SmmFaultTolerantWriteHandler():

Under "case FTW_FUNCTION_WRITE:", 'SmmFtwWriteHeader->Length' can be a
potential cross boundary access of the 'CommBuffer' (controlled external
inputs) during speculative execution. This cross boundary access is later
passed as parameter 'Length' into function FtwWrite().

Within function FtwWrite(), the value of 'Length' can be inferred by code:
"CopyMem (MyBuffer + Offset, Buffer, Length);". One can observe which part
of the content within 'Buffer' was brought into cache to possibly reveal
the value of 'Length'.

Hence, this commit adds a AsmLfence() after the boundary/range checks of
'CommBuffer' to prevent the speculative execution.

A more detailed explanation of the purpose of commit is under the
'Bounds check bypass mitigation' section of the below link:
https://software.intel.com/security-software-guidance/insights/host-firmware-speculative-execution-side-channel-mitigation

And the document at:
https://software.intel.com/security-software-guidance/api-app/sites/default/files/337879-analyzing-potential-bounds-Check-bypass-vulnerabilities.pdf

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Jiewen Yao <jiewen.yao@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Regression-tested-by: Laszlo Ersek <lersek@redhat.com>
2018-09-30 13:06:42 +08:00
2ecd829972 MdePkg/BaseLib: Add new AsmLfence API
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=1193

This commit will add a new BaseLib API AsmLfence(). This API will perform
a serializing operation on all load-from-memory instructions that were
issued prior to the call of this function. Please note that this API is
only available on IA-32 and x64.

The purpose of adding this API is to mitigate of the [CVE-2017-5753]
Bounds Check Bypass issue when untrusted data are being processed within
SMM. More details can be referred at the 'Bounds check bypass mitigation'
section at the below link:

https://software.intel.com/security-software-guidance/insights/host-firmware-speculative-execution-side-channel-mitigation

Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Cc: Leif Lindholm <leif.lindholm@linaro.org>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Jiewen Yao <jiewen.yao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Regression-tested-by: Laszlo Ersek <lersek@redhat.com>
2018-09-30 13:06:42 +08:00
8285f42e58 MdePkg/BaseLib: Remove an unused PCD
The PCD below is unused, so it has been removed from inf.
gEfiMdePkgTokenSpaceGuid.PcdDebugPropertyMask

Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
2018-09-30 09:21:05 +08:00
aa9986651b ShellPkg/UefiShellDebug1CommandsLib: Remove unused PCDs
The PCDs below are unused, so they have been removed from inf.
gEfiShellPkgTokenSpaceGuid.PcdShellFileOperationSize
gEfiMdePkgTokenSpaceGuid.PcdMaximumUnicodeStringLength

Cc: Jaben Carsey <jaben.carsey@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2018-09-30 09:16:04 +08:00
42a7c2871a ShellPkg/UefiHandleParsingLib: Remove an unused PCD
The PCD below is unused, so it has been removed from inf.
gEfiShellPkgTokenSpaceGuid.PcdShellPrintBufferSize

Cc: Jaben Carsey <jaben.carsey@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2018-09-30 09:16:03 +08:00
604cb49868 ShellPkg/DpDynamicCommand: Remove unused PCDs
The PCDs below are unused, so they have been removed from inf.
gEfiMdePkgTokenSpaceGuid.PcdUefiLibMaxPrintBufferSize in DpApp.inf
gEfiMdePkgTokenSpaceGuid.PcdUefiLibMaxPrintBufferSize in
DpDynamicCommand.inf

Cc: Jaben Carsey <jaben.carsey@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2018-09-30 09:16:01 +08:00
a9dfe53f56 ShellPkg/Shell: Remove unused PCDs
The PCDs below are unused, so they have been removed from inf.
gEfiShellPkgTokenSpaceGuid.PcdShellLibAutoInitialize
gEfiShellPkgTokenSpaceGuid.PcdShellMapNameLength

Cc: Jaben Carsey <jaben.carsey@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2018-09-30 09:16:00 +08:00
cc73a7ec00 UefiCpuPkg/Include/Register/Msr/GoldmontPlusMsr.h: Change to DOS format file.
Follow EDKII coding style, change file format to dos style.

BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1213

Cc: Dandan Bi <dandan.bi@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Dandan Bi <dandan.bi@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
2018-09-30 08:01:56 +08:00
83c10d2e8c SecurityPkg/Tcg2Dxe: Remove unused PCDs
The PCDs below are unused, so they have been removed from inf.
gEfiSecurityPkgTokenSpaceGuid.PcdTpmBaseAddress
gEfiMdeModulePkgTokenSpaceGuid.PcdAcpiDefaultOemId
gEfiMdeModulePkgTokenSpaceGuid.PcdAcpiDefaultOemTableId
gEfiMdeModulePkgTokenSpaceGuid.PcdAcpiDefaultOemRevision
gEfiMdeModulePkgTokenSpaceGuid.PcdAcpiDefaultCreatorId
gEfiMdeModulePkgTokenSpaceGuid.PcdAcpiDefaultCreatorRevision

Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Chao Zhang <chao.b.zhang@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Chao Zhang <chao.b.zhang@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
2018-09-30 07:39:01 +08:00
3e11c27f67 SecurityPkg/Tcg2ConfigPei: Remove an unused PCD
The PCD below is unused, so it has been removed from inf.
gEfiSecurityPkgTokenSpaceGuid.PcdTpmBaseAddress

Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Chao Zhang <chao.b.zhang@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Chao Zhang <chao.b.zhang@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
2018-09-30 07:39:01 +08:00
32fd20bb24 MdeModulePkg/FirmwarePerformanceDataTableDxe: Remove an unused PCD
The PCD below is unused, so it has been removed from inf.
gEfiMdeModulePkgTokenSpaceGuid.PcdExtFpdtBootRecordPadSize

Cc: Star Zeng <star.zeng@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-09-29 17:49:03 +08:00
0e800fde04 MdeModulePkg/DxeCapsuleLibFmp: Remove unused PCDs
The PCDs below are unused, so they have been removed from inf.
gEfiMdeModulePkgTokenSpaceGuid.PcdCapsuleMax
gEfiMdeModulePkgTokenSpaceGuid.PcdSystemRebootAfterCapsuleProcessFlag
gEfiMdeModulePkgTokenSpaceGuid.PcdStatusCodeSubClassCapsule
gEfiMdeModulePkgTokenSpaceGuid.PcdCapsuleStatusCodeProcessCapsulesBegin
gEfiMdeModulePkgTokenSpaceGuid.PcdCapsuleStatusCodeProcessCapsulesEnd
gEfiMdeModulePkgTokenSpaceGuid.PcdCapsuleStatusCodeUpdatingFirmware
gEfiMdeModulePkgTokenSpaceGuid.PcdCapsuleStatusCodeUpdateFirmwareSuccess
gEfiMdeModulePkgTokenSpaceGuid.PcdCapsuleStatusCodeUpdateFirmwareFailed
gEfiMdeModulePkgTokenSpaceGuid.PcdCapsuleStatusCodeResettingSystem
The library Pcdlib is not linked, so "#include <Library/PcdLib.h>"
is deleted.

Cc: Star Zeng <star.zeng@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-09-29 17:49:02 +08:00
f6e68d3cda UefiCpuPkg/CpuCommonFeaturesLib: Remove an unused PCD
The PCD below is unused, so it has been removed from inf.
gUefiCpuPkgTokenSpaceGuid.PcdCpuFeaturesSupport

Cc: Eric Dong <eric.dong@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Eric Dong <eric.dong@intel.com>
2018-09-29 16:00:39 +08:00
130e629284 NetworkPkg/HttpUtilitiesDxe: fix read memory access overflow.
The input param String of AsciiStrStr() requires a pointer to
 Null-terminated string, however in HttpUtilitiesParse(),
 the Buffersize before AllocateZeroPool() is equal to the size
 of TCP header, after the CopyMem(), it might not end with
 Null-terminator. It might cause memory access overflow.

Cc: Fu Siyuan <siyuan.fu@intel.com>
Cc: Wu Jiaxin <jiaxin.wu@intel.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1204
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Songpeng Li <songpeng.li@intel.com>
Reviewed-by: Fu Siyuan <siyuan.fu@intel.com>
2018-09-29 10:51:37 +08:00
2239ea71b6 NetworkPkg/HttpDxe: fix read memory access overflow in HTTPBoot.
The input param String of AsciiStrStr() requires a pointer to
 Null-terminated string, however in HttpTcpReceiveHeader(),
 the Buffersize before AllocateZeroPool() is equal to the size
 of TCP header, after the CopyMem(), it might not end with
 Null-terminator. It might cause memory access overflow.

Cc: Fu Siyuan <siyuan.fu@intel.com>
Cc: Wu Jiaxin <jiaxin.wu@intel.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1204
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Songpeng Li <songpeng.li@intel.com>
Reviewed-by: Fu Siyuan <siyuan.fu@intel.com>
2018-09-29 10:51:27 +08:00
b9cee524e6 MdeModulePkg/RegularExpressionDxe: modify inf to pass vs 2012 build
Add /wd4703 for passing VS 2012 build.This equals /wd4701 in VS2015.

Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Dongao Guo <dongao.guo@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-09-28 12:48:43 +08:00
d75c07bc68 IntelFsp2Pkg/GenCfgOpt.py: remove tailing space from output file
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1212

UPD header files generated by GenCfgOpt.py may have
tailing white space from some of the UPD description in DSC,
especially when python script automatically switching lines
for long description string. This patch will always remove
tailing white space for UPD header files.

Test: Verified the patch can remove tailing space in
      output header files when UPD DSC contains intentional
      tailing white space.

Cc: Jiewen Yao <Jiewen.yao@intel.com>
Cc: Gao Liming <liming.gao@intel.com>
Cc: Zhu Yonghong <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Chasel Chiu <chasel.chiu@intel.com>
Reviewed-by: Jiewen Yao <Jiewen.yao@intel.com>
2018-09-28 11:58:49 +08:00
1d7eed4182 IntelFsp2Pkg/GenCfgOpt.py: Support UPD offset auto assignment
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1211

For reducing maintenance effort, the UPD offset can be
automatic assigned by GenCfgOpt.py following by alignment
requirements.

The usage model as below:
. If UPD offset in DSC file are all '*', GenCfgOpt.py will
  assign offset for all UPD automatically. In this case no
  need to manually hardcode offset to all UPD in DSC.

. If UPD offset in DSC file are all not '*', GenCfgOpt.py
  will use hardcoded offset directly (original usage model)

. Tool does not support mixing scenario so UPD offset in DSC
  should be all '*' or all hardcoded but not mixed.

In auto mode UPD offset will be assigned following natural
alignment (size aligned) rule and the whole structure size
will align to either 32bits or 64bits depends on maximal UPD
size in the structure.

Test: Verified by both UPD offset hardcoded or '*' in DSC and
      generated UPD header files are correct.

Cc: Jiewen Yao <Jiewen.yao@intel.com>
Cc: Gao Liming <liming.gao@intel.com>
Cc: Zhu Yonghong <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Chasel Chiu <chasel.chiu@intel.com>
Reviewed-by: Jiewen Yao <Jiewen.yao@intel.com>
2018-09-28 11:58:49 +08:00
ccacc4d273 IntelFsp2Pkg/GenCfgOpt.py: support FixedAtBuild PCD
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1210

FixedAtBuild PCD is suggested to be used instead
of FeatureFlag PCD so extend this tool to support.
Also skipped PCDs which commented out by '#'.

Test: Verified with FixedAtBuild PCD for including or
excluding lines in generated UPD header files successfully.

Cc: Jiewen Yao <Jiewen.yao@intel.com>
Cc: Gao Liming <liming.gao@intel.com>
Cc: Zhu Yonghong <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Chasel Chiu <chasel.chiu@intel.com>
Reviewed-by: Jiewen Yao <Jiewen.yao@intel.com>
2018-09-28 11:58:49 +08:00
4063d37c6a MdeModulePkg Variable: Fix comment typo 'end' to 'start'
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1214
This patch only updates comment and function description, so has
no functionality impact.

This patch fixes comment typo 'end' to 'start' in GetStartPointer.

GetStartPointer for PEI and DXE has aligned function description,
but GetEndPointer does not.
This patch also aligns GetEndPointer's function description for
PEI and DXE.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Jian J Wang <jian.j.wang@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-09-28 11:13:34 +08:00
631db6b24d IntelFsp2WrapperPkg/BaseFspWrapperPlatformLibSample: Remove PCDs
The PCDs below are unused, so they have been removed from inf.
gIntelFsp2PkgTokenSpaceGuid.PcdTemporaryRamBase
gIntelFsp2PkgTokenSpaceGuid.PcdTemporaryRamSize

Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Chasel Chiu <chasel.chiu@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Chasel Chiu <chasel.chiu@intel.com>
2018-09-28 10:55:10 +08:00
8a9678c7a4 IntelFsp2WrapperPkg/FspWrapperNotifyDxe: Remove an unused PCD
The PCD below is unused, so it has been removed from inf.
gIntelFsp2WrapperTokenSpaceGuid.PcdFspsBaseAddress

Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Chasel Chiu <chasel.chiu@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Chasel Chiu <chasel.chiu@intel.com>
2018-09-28 10:55:01 +08:00
dab8d71dc6 IntelFsp2Pkg/BaseFspSwitchStackLib: Remove unused PCDs
The PCDs below are unused, so they have been removed from inf.
gIntelFsp2PkgTokenSpaceGuid.PcdFspMaxPatchEntry
gIntelFsp2PkgTokenSpaceGuid.PcdFspMaxPerfEntry

Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Chasel Chiu <chasel.chiu@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Chasel Chiu <chasel.chiu@intel.com>
2018-09-28 10:54:52 +08:00
06e88b8bfb IntelFsp2Pkg/BaseFspPlatformLib: Remove unused PCDs
The PCDs below are unused, so they have been removed from inf.
gIntelFsp2PkgTokenSpaceGuid.PcdGlobalDataPointerAddress
gIntelFsp2PkgTokenSpaceGuid.PcdTemporaryRamBase
gIntelFsp2PkgTokenSpaceGuid.PcdTemporaryRamSize
gIntelFsp2PkgTokenSpaceGuid.PcdFspTemporaryRamSize
gIntelFsp2PkgTokenSpaceGuid.PcdFspMaxPatchEntry
gIntelFsp2PkgTokenSpaceGuid.PcdFspMaxPerfEntry

Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Chasel Chiu <chasel.chiu@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Chasel Chiu <chasel.chiu@intel.com>
2018-09-28 10:54:42 +08:00
80d49b5714 IntelFsp2Pkg/BaseFspCommonLib: Remove unused PCDs
The PCDs below are unused, so they have been removed from inf.
gIntelFsp2PkgTokenSpaceGuid.PcdFspMaxPatchEntry
gIntelFsp2PkgTokenSpaceGuid.PcdFspMaxPerfEntry
gIntelFsp2PkgTokenSpaceGuid.PcdTemporaryRamBase
gIntelFsp2PkgTokenSpaceGuid.PcdTemporaryRamSize

Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Chasel Chiu <chasel.chiu@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Chasel Chiu <chasel.chiu@intel.com>
2018-09-28 10:54:33 +08:00
1d94f67b24 IntelFsp2Pkg/FspSecCore: Remove unused PCDs
The PCDs below are unused, so they have been removed from inf.
gEfiMdePkgTokenSpaceGuid.PcdPciExpressBaseAddress
gIntelFsp2PkgTokenSpaceGuid.PcdGlobalDataPointerAddress
gIntelFsp2PkgTokenSpaceGuid.PcdFspMaxPatchEntry
gIntelFsp2PkgTokenSpaceGuid.PcdFspMaxPerfEntry
gEfiMdePkgTokenSpaceGuid.PcdPciExpressBaseAddress
gIntelFsp2PkgTokenSpaceGuid.PcdGlobalDataPointerAddress
gIntelFsp2PkgTokenSpaceGuid.PcdTemporaryRamBase
gIntelFsp2PkgTokenSpaceGuid.PcdTemporaryRamSize
gIntelFsp2PkgTokenSpaceGuid.PcdFspTemporaryRamSize
gEfiMdePkgTokenSpaceGuid.PcdPciExpressBaseAddress
gIntelFsp2PkgTokenSpaceGuid.PcdFspMaxPatchEntry
gIntelFsp2PkgTokenSpaceGuid.PcdFspMaxPerfEntry

Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Chasel Chiu <chasel.chiu@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Chasel Chiu <chasel.chiu@intel.com>
2018-09-28 10:54:22 +08:00
ed5de31189 IntelFsp2(Wrapper)Pkg: Revert from e8208100 to 737f812b
Commit formats had issues so reverted 9 commits
from IntelFsp2Pkg and IntelFsp2WrapperPkg.
Will re-submit them with correct formats.

Cc: Jiewen Yao <Jiewen.yao@intel.com>
Cc: Star Zeng <star.zeng@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Chasel Chiu <chasel.chiu@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-09-28 10:50:43 +08:00
61d3f1000e ShellPkg/TftpDynamicCommand: Fix the potentially uninitialized local variable used.
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1217

Local variable 'Mtftp4Token' might be uninitialized when error happen. This patch is to
resolve the issue.

Cc: Ye Ting <ting.ye@intel.com>
Cc: Fu Siyuan <siyuan.fu@intel.com>
Cc: Carsey Jaben <jaben.carsey@intel.com>
Cc: Zeng Star <star.zeng@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Wu Jiaxin <jiaxin.wu@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-09-28 10:46:28 +08:00
cf320090fa EmulatorPkg/EmulatorPkg.dsc: override PCD default to avoid boot failure
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1215

This issue is caused by commit 5267926134.

The reason is this commit changed the condition of building page table
in DxeIpl. The code before it will only build page table for the sake
of PcdSetNxForStack. This commit added PcdImageProtectionPolicy and
PcdDxeNxMemoryProtectionPolicy into the logic. But the default value of
PcdImageProtectionPolicy is 02, which means the DxeIpl should build page
table. Due to the fact that EmulatorPkg doesn't support page table at all,
this will cause exception on Windows OS.

This patch solves this issue by setting PcdImageProtectionPolicy to 0
explicitly in EmulatorPkg.dsc.

Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Andrew Fish <afish@apple.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Star Zeng <star.zeng@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jian J Wang <jian.j.wang@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-09-28 10:19:35 +08:00
2b55b922e9 Nt32Pkg/Nt32Pkg.dsc: override PCD default to avoid boot failure
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1215

This issue is caused by commit 5267926134.

The reason is this commit changed the condition of building page table
in DxeIpl. The code before it will only build page table for the sake
of PcdSetNxForStack. This commit added PcdImageProtectionPolicy and
PcdDxeNxMemoryProtectionPolicy into the logic. But the default value of
PcdImageProtectionPolicy is 02, which means the DxeIpl should build page
table. Due to the fact that Nt32Pkg doesn't support page table at all,
this will cause exception on Windows OS.

This patch solves this issue by setting PcdImageProtectionPolicy to 0
explicitly in Nt32Pkg.dsc.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Hao Wu <hao.a.wu@intel.com>
Cc: Star Zeng <star.zeng@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jian J Wang <jian.j.wang@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-09-28 10:19:35 +08:00
737f812b00 GenCfgOpt.py: remove tailing space from output file
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1212

UPD header files generated by GenCfgOpt.py may have
tailing white space from some of the UPD description in DSC,
especially when python script automatically switching lines
for long description string. This patch will always remove
tailing white space for UPD header files.

Test: Verified the patch can remove tailing space in
      output header files when UPD DSC contains intentional
      tailing white space.

Cc: Jiewen Yao <Jiewen.yao@intel.com>
Cc: Gao Liming <liming.gao@intel.com>
Cc: Zhu Yonghong <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Chasel Chiu <chasel.chiu@intel.com>
Reviewed-by: Jiewen Yao <Jiewen.yao@intel.com>
2018-09-28 08:44:46 +08:00
29933058fc IntelFsp2Pkg/GenCfgOpt.py: Support UPD offset auto assignment
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1211

For reducing maintenance effort, the UPD offset can be
automatic assigned by GenCfgOpt.py following by alignment
requirements.

The usage model as below:
. If UPD offset in DSC file are all '*', GenCfgOpt.py will
  assign offset for all UPD automatically. In this case no
  need to manually hardcode offset to all UPD in DSC.

. If UPD offset in DSC file are all not '*', GenCfgOpt.py
  will use hardcoded offset directly (original usage model)

. Tool does not support mixing scenario so UPD offset in DSC
  should be all '*' or all hardcoded but not mixed.

In auto mode UPD offset will be assigned following natural
alignment (size aligned) rule and the whole structure size
will align to either 32bits or 64bits depends on maximal UPD
size in the structure.

Test: Verified by both UPD offset hardcoded or '*' in DSC and
      generated UPD header files are correct.

Cc: Jiewen Yao <Jiewen.yao@intel.com>
Cc: Gao Liming <liming.gao@intel.com>
Cc: Zhu Yonghong <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Chasel Chiu <chasel.chiu@intel.com>
Reviewed-by: Jiewen Yao <Jiewen.yao@intel.com>
2018-09-28 08:39:25 +08:00
bb88882b75 IntelFsp2Pkg/GenCfgOpt.py: support FixedAtBuild PCD
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1210

FixedAtBuild PCD is suggested to be used instead
of FeatureFlag PCD so extend this tool to support.
Also skipped PCDs which commented out by '#'.

Test: Verified with FixedAtBuild PCD for including or
excluding lines in generated UPD header files successfully.

Cc: Jiewen Yao <Jiewen.yao@intel.com>
Cc: Gao Liming <liming.gao@intel.com>
Cc: Zhu Yonghong <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Chasel Chiu <chasel.chiu@intel.com>
2018-09-28 07:58:48 +08:00
6532fdec11 [PATCH v3 06/16] IntelFsp2WrapperPkg/BaseFspWrapperPlatformLibSample:
Remove PCDs

The PCDs below are unused, so they have been removed from inf.
gIntelFsp2PkgTokenSpaceGuid.PcdTemporaryRamBase
gIntelFsp2PkgTokenSpaceGuid.PcdTemporaryRamSize

Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Chasel Chiu <chasel.chiu@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Chasel Chiu <chasel.chiu@intel.com>
2018-09-27 15:42:57 +08:00
70493a8d54 [PATCH v3 05/16] IntelFsp2WrapperPkg/FspWrapperNotifyDxe: Remove an
unused PCD

The PCD below is unused, so it has been removed from inf.
gIntelFsp2WrapperTokenSpaceGuid.PcdFspsBaseAddress

Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Chasel Chiu <chasel.chiu@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Chasel Chiu <chasel.chiu@intel.com>
2018-09-27 15:42:57 +08:00
fbb44beba0 [PATCH v3 04/16] IntelFsp2Pkg/BaseFspSwitchStackLib: Remove unused
PCDs

The PCDs below are unused, so they have been removed from inf.
gIntelFsp2PkgTokenSpaceGuid.PcdFspMaxPatchEntry
gIntelFsp2PkgTokenSpaceGuid.PcdFspMaxPerfEntry

Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Chasel Chiu <chasel.chiu@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Chasel Chiu <chasel.chiu@intel.com>
2018-09-27 15:42:56 +08:00
54611eacfe [PATCH v3 03/16] IntelFsp2Pkg/BaseFspPlatformLib: Remove unused PCDs
The PCDs below are unused, so they have been removed from inf.
gIntelFsp2PkgTokenSpaceGuid.PcdGlobalDataPointerAddress
gIntelFsp2PkgTokenSpaceGuid.PcdTemporaryRamBase
gIntelFsp2PkgTokenSpaceGuid.PcdTemporaryRamSize
gIntelFsp2PkgTokenSpaceGuid.PcdFspTemporaryRamSize
gIntelFsp2PkgTokenSpaceGuid.PcdFspMaxPatchEntry
gIntelFsp2PkgTokenSpaceGuid.PcdFspMaxPerfEntry

Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Chasel Chiu <chasel.chiu@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Chasel Chiu <chasel.chiu@intel.com>
2018-09-27 15:42:56 +08:00
47042f60e4 [PATCH v3 02/16] IntelFsp2Pkg/BaseFspCommonLib: Remove unused PCDs
The PCDs below are unused, so they have been removed from inf.

gIntelFsp2PkgTokenSpaceGuid.PcdFspMaxPatchEntry
gIntelFsp2PkgTokenSpaceGuid.PcdFspMaxPerfEntry
gIntelFsp2PkgTokenSpaceGuid.PcdTemporaryRamBase
gIntelFsp2PkgTokenSpaceGuid.PcdTemporaryRamSize

Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Chasel Chiu <chasel.chiu@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Chasel Chiu <chasel.chiu@intel.com>
2018-09-27 15:42:56 +08:00
e8208100c9 [PATCH v3 01/16] IntelFsp2Pkg/FspSecCore: Remove unused PCDs
The PCDs below are unused, so they have been removed from inf.

gEfiMdePkgTokenSpaceGuid.PcdPciExpressBaseAddress
gIntelFsp2PkgTokenSpaceGuid.PcdGlobalDataPointerAddress
gIntelFsp2PkgTokenSpaceGuid.PcdFspMaxPatchEntry
gIntelFsp2PkgTokenSpaceGuid.PcdFspMaxPerfEntry
gEfiMdePkgTokenSpaceGuid.PcdPciExpressBaseAddress
gIntelFsp2PkgTokenSpaceGuid.PcdGlobalDataPointerAddress
gIntelFsp2PkgTokenSpaceGuid.PcdTemporaryRamBase
gIntelFsp2PkgTokenSpaceGuid.PcdTemporaryRamSize
gIntelFsp2PkgTokenSpaceGuid.PcdFspTemporaryRamSize
gEfiMdePkgTokenSpaceGuid.PcdPciExpressBaseAddress
gIntelFsp2PkgTokenSpaceGuid.PcdFspMaxPatchEntry
gIntelFsp2PkgTokenSpaceGuid.PcdFspMaxPerfEntry

Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Chasel Chiu <chasel.chiu@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Chasel Chiu <chasel.chiu@intel.com>
2018-09-27 15:42:56 +08:00
4c214f8229 UefiCpuPkg PiSmmCpuDxeSmm: Use new EfiLocateFirstAcpiTable()
https://bugzilla.tianocore.org/show_bug.cgi?id=967
Request to add a library function for GetAcpiTable() in order
to get ACPI table using signature as input.

After evaluation, we found there are many duplicated code to
find ACPI table by signature in different modules.

This patch updates PiSmmCpuDxeSmm to use new
EfiLocateFirstAcpiTable() and remove the duplicated code.

Cc: Younas khan <pmdyounaskhan786@gmail.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
2018-09-27 14:25:36 +08:00
cb6fe7bb8b ShellPkg DpDynamicCommand: Use new EfiLocateFirstAcpiTable()
https://bugzilla.tianocore.org/show_bug.cgi?id=967
Request to add a library function for GetAcpiTable() in order
to get ACPI table using signature as input.

After evaluation, we found there are many duplicated code to
find ACPI table by signature in different modules.

This patch updates DpDynamicCommand to use new
EfiLocateFirstAcpiTable() and remove the duplicated code.

Cc: Younas khan <pmdyounaskhan786@gmail.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Dandan Bi <dandan.bi@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-09-27 14:25:35 +08:00
44c385effd PcAtChipsetPkg PcRtc: Use new EfiLocateFirstAcpiTable()
https://bugzilla.tianocore.org/show_bug.cgi?id=967
Request to add a library function for GetAcpiTable() in order
to get ACPI table using signature as input.

After evaluation, we found there are many duplicated code to
find ACPI table by signature in different modules.

This patch updates PcatRealTimeClockRuntimeDxe to use new
EfiLocateFirstAcpiTable() and remove the duplicated code.

Cc: Younas khan <pmdyounaskhan786@gmail.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-09-27 14:25:35 +08:00
2ad0581b1c MdeModulePkg S3SaveStateDxe: Use new EfiLocateFirstAcpiTable()
https://bugzilla.tianocore.org/show_bug.cgi?id=967
Request to add a library function for GetAcpiTable() in order
to get ACPI table using signature as input.

After evaluation, we found there are many duplicated code to
find ACPI table by signature in different modules.

This patch updates S3SaveStateDxe to use new
EfiLocateFirstAcpiTable() and remove the duplicated code.

Cc: Younas khan <pmdyounaskhan786@gmail.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Jian J Wang <jian.j.wang@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Jiewen Yao <jiewen.yao@intel.com>
2018-09-27 14:25:34 +08:00
74bdd03ead IntelSiliconPkg IntelVTdDxe: Use new EfiLocateFirstAcpiTable()
https://bugzilla.tianocore.org/show_bug.cgi?id=967
Request to add a library function for GetAcpiTable() in order
to get ACPI table using signature as input.

After evaluation, we found there are many duplicated code to
find ACPI table by signature in different modules.

This patch updates IntelVTdDxe to use new
EfiLocateFirstAcpiTable() and remove the duplicated code.

Cc: Younas khan <pmdyounaskhan786@gmail.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Jiewen Yao <jiewen.yao@intel.com>
2018-09-27 14:25:33 +08:00
c1a00d0942 IntelFrameworkPkg FrameworkUefiLib: Add new EfiLocateXXXAcpiTable APIs
https://bugzilla.tianocore.org/show_bug.cgi?id=967
Request to add a library function for GetAcpiTable() in order
to get ACPI table using signature as input.

After evaluation, we found there are many duplicated code to
find ACPI table by signature in different modules.

This patch adds new EfiLocateXXXAcpiTable() APIs in UefiLib
for the request and also the following patch to remove the
duplicated code.

V2:
ASSERT(FALSE) and return NULL instead of real implementation for the
new APIs as the IntelFrameworkPkg is a to-be-deprecated packages.

Cc: Younas khan <pmdyounaskhan786@gmail.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Jiewen Yao <jiewen.yao@intel.com>
2018-09-27 14:25:32 +08:00
af5e952159 MdePkg UefiLib: Add new EfiLocateXXXAcpiTable() APIs
https://bugzilla.tianocore.org/show_bug.cgi?id=967
Request to add a library function for GetAcpiTable() in order
to get ACPI table using signature as input.

After evaluation, we found there are many duplicated code to
find ACPI table by signature in different modules.

This patch adds new EfiLocateXXXAcpiTable() APIs in UefiLib
for the request and also the following patch to remove the
duplicated code.

Cc: Younas khan <pmdyounaskhan786@gmail.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-09-27 14:25:32 +08:00
24c55f5dcc NetworkPkg/UefiPxeBcDxe: Add the clarification compared to UefiPxeBcDxe in MdeModulePkg.
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1205

This patch is to add the driver usage/difference clarification
compared to UefiPxeBcDxe in MdeModulePkg.

Cc: Ye Ting <ting.ye@intel.com>
Cc: Fu Siyuan <siyuan.fu@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Wu Jiaxin <jiaxin.wu@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
2018-09-27 09:00:07 +08:00
0bcbdf9c74 NetworkPkg/IScsiDxe: Add the clarification compared to IScsiDxe in MdeModulePkg.
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1205

This patch is to add the driver usage/difference clarification
compared to IScsiDxe in MdeModulePkg.

Cc: Ye Ting <ting.ye@intel.com>
Cc: Fu Siyuan <siyuan.fu@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Wu Jiaxin <jiaxin.wu@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
2018-09-27 09:00:06 +08:00
897720daef NetworkPkg/TcpDxe: Add the clarification compared to Tcp4Dxe in MdeModulePkg.
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1205

This patch is to add the driver usage/difference clarification
compared to Tcp4Dxe in MdeModulePkg.

Cc: Ye Ting <ting.ye@intel.com>
Cc: Fu Siyuan <siyuan.fu@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Wu Jiaxin <jiaxin.wu@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
2018-09-27 09:00:06 +08:00
f120d15e6d MdeModulePkg/UefiPxeBcDxe: Add the clarification compared to UefiPxeBcDxe in NetworkPkg.
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1205

This patch is to add the driver usage/difference clarification
compared to UefiPxeBcDxe in NetworkPkg.

Cc: Ye Ting <ting.ye@intel.com>
Cc: Fu Siyuan <siyuan.fu@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Wu Jiaxin <jiaxin.wu@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
2018-09-27 09:00:05 +08:00
d619dab698 MdeModulePkg/IScsiDxe: Add the clarification compared to IScsiDxe in NetworkPkg.
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1205

This patch is to add the driver usage/difference clarification
compared to IScsiDxe in NetworkPkg.

Cc: Ye Ting <ting.ye@intel.com>
Cc: Fu Siyuan <siyuan.fu@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Wu Jiaxin <jiaxin.wu@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
2018-09-27 09:00:04 +08:00
18e205754d MdeModulePkg/Tcp4Dxe: Add the clarification compared to TcpDxe in NetworkPkg.
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1205

This patch is to add the driver usage/difference clarification
compared to TcpDxe in NetworkPkg.

Cc: Ye Ting <ting.ye@intel.com>
Cc: Fu Siyuan <siyuan.fu@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Wu Jiaxin <jiaxin.wu@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
2018-09-27 09:00:03 +08:00
160b082e64 NetworkPkg/UefiPxeBcDxe: Use the specified MTFTP windowsize.
*v2: Since the new PCD (PcdPxeTftpWindowSize) was renamed/defined in
NetworkPkg instead of MdeModulePkg, this new version is to update the
consuming PXE driver.

Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=886

This patch is to use the specified MTFTP windowsize to benefit the PXE
download performance.

Cc: Ye Ting <ting.ye@intel.com>
Cc: Fu Siyuan <siyuan.fu@intel.com>
Cc: Shao Ming <ming.shao@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Wu Jiaxin <jiaxin.wu@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Fu Siyuan <siyuan.fu@intel.com>
Reviewed-by: Ye Ting <ting.ye@intel.com>
2018-09-27 09:00:02 +08:00
2fff697f1e NetworkPkg: Define one PCD for PXE to specify MTFTP windowsize.
*v2: Rename and redefine the PCD in NetworkPkg instead of MdeModulePkg.

Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=886

This patch is to define one new PCD for PXE driver to specify MTFTP windowsize so as
to improve the PXE download performance. The default value is set to 4.

Cc: Ye Ting <ting.ye@intel.com>
Cc: Fu Siyuan <siyuan.fu@intel.com>
Cc: Shao Ming <ming.shao@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Wu Jiaxin <jiaxin.wu@intel.com>
Reviewed-by: Fu Siyuan <siyuan.fu@intel.com>
Reviewed-by: Ye Ting <ting.ye@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
2018-09-27 09:00:02 +08:00
5c6fdb5e83 ShellPkg/TftpDynamicCommand: Add one option for tftp command to specify windowsize.
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=886

This patch is to define one new option for TFTP shell command to specify the
windowsize option as defined in RFC 7440. Valid range is between 1 and 64,
default value is 1.

Note that: RFC 7440 does not mention max window size value, but for the
stability reason, the value is limited to 64.

Cc: Ye Ting <ting.ye@intel.com>
Cc: Fu Siyuan <siyuan.fu@intel.com>
Cc: Carsey Jaben <jaben.carsey@intel.com>
Cc: Shao Ming <ming.shao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Wu Jiaxin <jiaxin.wu@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Fu Siyuan <siyuan.fu@intel.com>
Reviewed-by: Ye Ting <ting.ye@intel.com>
2018-09-27 09:00:01 +08:00
f3427f12a4 NetworkPkg/Mtftp6Dxe: Support windowsize in read request operation.
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=886

This patch is to support the TFTP windowsize option described in RFC 7440.
The feature allows the client and server to negotiate a window size of
consecutive blocks to send as an alternative for replacing the single-block
lockstep schema.

Currently, the windowsize for write request operation is not supported since
there is no real use cases.

Cc: Ye Ting <ting.ye@intel.com>
Cc: Fu Siyuan <siyuan.fu@intel.com>
Cc: Shao Ming <ming.shao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Wu Jiaxin <jiaxin.wu@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Fu Siyuan <siyuan.fu@intel.com>
Reviewed-by: Ye Ting <ting.ye@intel.com>
2018-09-27 09:00:00 +08:00
6c047cfab1 MdeModulePke/Mtftp4Dxe: Support windowsize in read request operation.
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=886

This patch is to support the TFTP windowsize option described in RFC 7440.
The feature allows the client and server to negotiate a window size of
consecutive blocks to send as an alternative for replacing the single-block
lockstep schema.

Currently, the windowsize for write request operation is not supported since
there is no real use cases.

Cc: Ye Ting <ting.ye@intel.com>
Cc: Fu Siyuan <siyuan.fu@intel.com>
Cc: Shao Ming <ming.shao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Wu Jiaxin <jiaxin.wu@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Fu Siyuan <siyuan.fu@intel.com>
Reviewed-by: Ye Ting <ting.ye@intel.com>
2018-09-27 08:59:59 +08:00
6a147d6dae BaseTool: Support different PCDs that refers to the same EFI variable.
If Structure PCD and Normal Pcd refer to the
same EFI variable, do EFI variable merge, otherwise, do
EFI variable combination.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-09-26 22:50:40 +08:00
57ee97c01c BaseTool: Replace dict with OrderedDict.
Replace dict with OrderedDict for PCD so that
the pcd list has same order.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-09-26 22:50:36 +08:00
f27e800a99 BaseTools: Fix the wrong reference _GetSkuIds() in AutoGen code
DscBuildData has been updated to define SkuIds instead of _GetSkuIds().
The consumer code should refer to SkuIds.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2018-09-26 22:15:42 +08:00
8a94eb9283 MdePkg/BaseSynchronizationLib: fix XADD operands in GCC IA32/X64 assembly
Currently, "gcc-4.8.5-28.el7_5.1.x86_64" generates the following code for
me, from the XADD inline assembly added to "X64/GccInline.c" in commit
17634d026f:

> 0000000000004383 <InternalSyncIncrement>:
> UINT32
> EFIAPI
> InternalSyncIncrement (
>   IN      volatile UINT32    *Value
>   )
> {
>     4383:       55                      push   %rbp
>     4384:       48 89 e5                mov    %rsp,%rbp
>     4387:       48 83 ec 10             sub    $0x10,%rsp
>     438b:       48 89 4d 10             mov    %rcx,0x10(%rbp)
>   UINT32  Result;
>
>   __asm__ __volatile__ (
>     438f:       48 8b 55 10             mov    0x10(%rbp),%rdx
>     4393:       48 8b 45 10             mov    0x10(%rbp),%rax
>     4397:       b8 01 00 00 00          mov    $0x1,%eax
>     439c:       f0 0f c1 00             lock xadd %eax,(%rax)
>     43a0:       ff c0                   inc    %eax
>     43a2:       89 45 fc                mov    %eax,-0x4(%rbp)
>     : "m"  (*Value)           // %2
>     : "memory",
>       "cc"
>     );
>
>   return Result;
>     43a5:       8b 45 fc                mov    -0x4(%rbp),%eax
> }
>     43a8:       c9                      leaveq
>     43a9:       c3                      retq
>

The MOV $0X1,%EAX instruction corrupts the address of Value in %RAX before
we reach the XADD instruction. In fact, it makes no sense for XADD to use
%EAX as source operand and (%RAX) as destination operand at the same time.

The XADD instruction's destination operand is a read-write operand. The
GCC documentation states:

> The ordinary output operands must be write-only; GCC will assume that
> the values in these operands before the instruction are dead and need
> not be generated. Extended asm supports input-output or read-write
> operands. Use the constraint character `+' to indicate such an operand
> and list it with the output operands. You should only use read-write
> operands when the constraints for the operand (or the operand in which
> only some of the bits are to be changed) allow a register.

(The above is intentionally quoted from the oldest GCC release that edk2
supports, namely gcc-4.4:
<https://gcc.gnu.org/onlinedocs/gcc-4.4.7/gcc/Extended-Asm.html>.)

Fix the operand list accordingly.

With the patch applied, I get:

> 0000000000004383 <InternalSyncIncrement>:
> UINT32
> EFIAPI
> InternalSyncIncrement (
>   IN      volatile UINT32    *Value
>   )
> {
>     4383:       55                      push   %rbp
>     4384:       48 89 e5                mov    %rsp,%rbp
>     4387:       48 83 ec 10             sub    $0x10,%rsp
>     438b:       48 89 4d 10             mov    %rcx,0x10(%rbp)
>   UINT32  Result;
>
>   __asm__ __volatile__ (
>     438f:       48 8b 55 10             mov    0x10(%rbp),%rdx
>     4393:       48 8b 45 10             mov    0x10(%rbp),%rax
>     4397:       b8 01 00 00 00          mov    $0x1,%eax
>     439c:       f0 0f c1 02             lock xadd %eax,(%rdx)
>     43a0:       ff c0                   inc    %eax
>     43a2:       89 45 fc                mov    %eax,-0x4(%rbp)
>     :                         // no inputs that aren't also outputs
>     : "memory",
>       "cc"
>     );
>
>   return Result;
>     43a5:       8b 45 fc                mov    -0x4(%rbp),%eax
> }
>     43a8:       c9                      leaveq
>     43a9:       c3                      retq

Note that some other bugs remain in
"BaseSynchronizationLib/*/GccInline.c"; those should be addressed later,
under <https://bugzilla.tianocore.org/show_bug.cgi?id=1208>.

Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Michael Kinney <michael.d.kinney@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1207
Fixes: 17634d026f
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-09-26 14:02:07 +02:00
53002b7ecf UefiCpuPkg/Include/Register/Msr/XeonPhiMsr.h: Change structure definition.
V3 changes include:
  1. Keep the ReservedX not change if bit info not changed for this field.

V2 changes include:
  1. Use X in ReservedX fields from totally new value if MSR structure definition changed.
     For example, if in current structure, the max reserved variable is Reserved2, in new
     definition, reserved variable is begin with Reserved3.

V1 Changes:
Changes includes:
  1. Update MSR structure definition, change some reserved fields to useful fields:
     1. MSR_XEON_PHI_PKG_CST_CONFIG_CONTROL_REGISTER
     2. MSR_XEON_PHI_SMM_MCA_CAP_REGISTER
  2. For MSR_XEON_PHI_PMG_IO_CAPTURE_BASE_REGISTER structure, it expand the field range.
     Old definition like below:
       typedef union {
         ///
         /// Individual bit fields
         ///
         struct {
           ///
           /// [Bits 15:0] LVL_2 Base Address (R/W).
           ///
           UINT32  Lvl2Base:16;
           ///
           /// [Bits 18:16] C-state Range (R/W)  Specifies the encoding value of the
           /// maximum C-State code name to be included when IO read to MWAIT
           /// redirection is enabled by MSR_PKG_CST_CONFIG_CONTROL[bit10]: 100b - C4
           /// is the max C-State to include 110b - C6 is the max C-State to include.
           ///
           UINT32  CStateRange:3;
           UINT32  Reserved1:13;
           UINT32  Reserved2:32;
         } Bits;
         ///
         /// All bit fields as a 32-bit value
         ///
         UINT32  Uint32;
         ///
         /// All bit fields as a 64-bit value
         ///
         UINT64  Uint64;
       } MSR_XEON_PHI_PMG_IO_CAPTURE_BASE_REGISTER;
    This patch make below changes for this data structure, it expand "CStateRange" field width.
      old one:
        UINT32  CStateRange:3;
        UINT32  Reserved1:13;
      new one:
        UINT32  CStateRange:7;
        UINT32  Reserved1:9;

Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
2018-09-26 15:17:15 +08:00
4382394aa9 UefiCpuPkg/Include/Register/ArchitecturalMsr.h: Change structure definition.
V3 changes include:
  1. Keep ReservedX not change if bit info not changed for this field.

V2 changes include:
  1. Use X in ReservedX fields from totally new value if MSR structure definition changed.
     For example, if in current structure, the max reserved variable is Reserved2, in new
     definition, reserved variable is begin with Reserved3.

V1 Changes includes:
  1. Change fields which is reserved in old version: MSR_IA32_RTIT_CTL_REGISTER

Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
2018-09-26 15:17:15 +08:00
140d713175 UefiCpuPkg/Include/Register/Msr/GoldmontMsr.h: Add new MSR name and keep old one.
Changes includes:
  1. Change MSR name:
    1. MSR_GOLDMONT_SGXOWNER0 => MSR_GOLDMONT_SGXOWNEREPOCH0
    2. MSR_GOLDMONT_SGXOWNER1 => MSR_GOLDMONT_SGXOWNEREPOCH1
  2. Keep old MSR definition (MSR_GOLDMONT_SGXOWNER0/1) for compatibility.
    1. Define old MSR like below style:
           #define MSR_GOLDMONT_SGXOWNER0   MSR_GOLDMONT_SGXOWNEREPOCH0

Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
2018-09-26 15:17:15 +08:00
8b344785a3 UefiCpuPkg/Include/Register/Msr/SkylakeMsr.h: Add new MSR name and keep old one.
Changes includes:
  1. Change MSR name:
    1. MSR_SKYLAKE_SGXOWNER0 => MSR_SKYLAKE_SGXOWNEREPOCH0
    2. MSR_SKYLAKE_SGXOWNER1 => MSR_SKYLAKE_SGXOWNEREPOCH1
  2. Keep old MSR definition(MSR_SKYLAKE_SGXOWNER0/1) for compatibility
    1. Use below coding style to define old MSR
         #define MSR_SKYLAKE_SGXOWNER0  MSR_SKYLAKE_SGXOWNEREPOCH0

Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
2018-09-26 15:17:14 +08:00
f49bbeda3e UefiCpuPkg/Include/Register/Msr/CoreMsr.h: Remove old MSR.
Changes includes:
  1. Remove old MSR which not existed in 2018-05 version spec:
     1. MSR_CORE_ROB_CR_BKUPTMPDR6

Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
2018-09-26 15:17:14 +08:00
7ce15b89bb UefiCpuPkg/Include/Register/Msr/P6Msr.h: Remove old MSR.
Changes includes:
  1. Remove MSR which not existed in 2018-05 version spec: MSR_P6_ROB_CR_BKUPTMPDR6.

Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
2018-09-26 15:17:14 +08:00
a71920ae14 UefiCpuPkg/Include/Register/Msr/Core2Msr.h: Remove old MSR.
Changes includes:
  1. Remove old MSR which not existed in 2018-05 version spec:
     1. MSR_CORE2_BBL_CR_CTL3

Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
2018-09-26 15:17:13 +08:00
d05b288ae9 UefiCpuPkg/Include/Register/ArchitecturalMsr.h: Add new MSR.
Changes includes:
  1. Add new MSRs: MSR_IA32_L2_QOS_CFG/MSR_IA32_CSTAR.

Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
2018-09-26 15:17:13 +08:00
3add020579 UefiCpuPkg/Include/Register/Msr/SkylakeMsr.h: Add new MSRs.
Changes includes:
  1. Add new MSR definitions.
  2. Add support platform info.

Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
2018-09-26 15:17:12 +08:00
dfb208511e UefiCpuPkg/Include/Register/Msr/XeonPhiMsr.h: Add new MSR.
Changes includes:
  1. Add new MSR definition:
     1. MSR_XEON_PHI_PPIN_CTL
     2. MSR_XEON_PHI_PPIN
     3. MSR_XEON_PHI_MISC_FEATURE_ENABLES
     4. MSR_XEON_PHI_MSRUNCORE_RATIO_LIMIT
  2. Add DisplayModule == 0x85 supports.

Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
2018-09-26 15:17:12 +08:00
c4b073632d UefiCpuPkg/Include/Register/Msr/*.h: Add new MSR.
Changes includes:
  1. Add new MSR: MSR_*_MSRUNCORE_RATIO_LIMIT

Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
2018-09-26 15:17:12 +08:00
47f75d2169 UefiCpuPkg/Include/Register/Msr/SilvermontMsr.h: Add new MSR.
Changes includes:
  1. Add new MSR: MSR_SILVERMONT_PLATFORM_INFO

Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
2018-09-26 15:17:11 +08:00
b715c37eaa UefiCpuPkg/Include/Register/Msr/GoldmontPlusMsr.h: Add new MSR file for goldmont plus microarchitecture.
Changes includes:
  1. Add new MSR file which used for goldmont plus microarchitecture.

Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
2018-09-26 15:17:11 +08:00
ba1a2d1102 UefiCpuPkg/Include/Register/Msr: Update reference spec info.
Latest SDM has moved MSR related content from volume 3 chapter 35 to volume 4
chapter 2. Current MSR's comments need to be updated to reference the new
chapter info.

Changes includes:
  1. Update referenced chapter info from some MSRs.
  2. Update referenced SDM version info.

Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
2018-09-26 15:17:10 +08:00
447b08b3d2 UefiCpuPkg/MtrrLib: Revert "Skip MSR access when the pair is invalid"
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1187

The patch reverts 9c8c4478cf
"UefiCpuPkg/MtrrLib: Skip Base MSR access when the pair is invalid".

Microsoft Windows will report an error in event manager if MTRR
usage is different across hibernate even when the difference is
in an non valid MTRR pair. This seems like a bug in Windows but
for compatibility and servicing reasons we think a change in UEFI
would wise.
A Windows change has already been submitted for the next iteration
(2019 time frame).

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Eric Dong <eric.dong@intel.com>
Cc: Sean Brogan <sean.brogan@microsoft.com>
2018-09-26 13:09:15 +08:00
69b4046504 MdeModulePkg/PciHostBridge: Move declaration of mIoMmu to header file
The change doesn't have functionality impact.
It just renames the mIoMmuProtocol to mIoMmu and moves the
declaration from PciRootBridgeIo.c to PciHostBridge.h.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Suggested-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
2018-09-26 12:12:53 +08:00
b8bfb92b4e MdeModulePkg/PciHostBridge: Add RESOURCE_VALID() to simplify code
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Garrett Kirkendall <garrett.kirkendall@amd.com>
2018-09-26 12:12:51 +08:00
bff6584d1f MdeModulePkg/PciHostBridge: Fix a bug that prevents PMEM access
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1196

RootBridgeIoCheckParameter() verifies that the requested MMIO access
can fit in any of the MEM/PMEM 32/64 ranges. But today's logic
somehow only checks the requested access against MEM 32/64 ranges.

It should also check the requested access against PMEM 32/64 ranges.

The patch fixes this issue.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Garrett Kirkendall <garrett.kirkendall@amd.com>
2018-09-26 12:12:50 +08:00
666d05a0ad MdeModulePkg/PciHostBridge: Enhance boundary check in Io/Mem.Read/Write
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Garrett Kirkendall <garrett.kirkendall@amd.com>
2018-09-26 12:12:49 +08:00
2939283f2d UefiCpuPkg/CpuMpPei: fix vs2012 build error
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1166

Visual Studio 2012 will complain uninitialized variable, StackBase,
in the CpuPaging.c. This patch adds code to init it to zero and
ASSERT check against 0. This is enough since uninit case will only
happen during retrieving stack memory via gEfiHobMemoryAllocStackGuid.
But this HOB will always be created in advance.

Cc: Dandan Bi <dandan.bi@intel.com>
Cc: Hao A Wu <hao.a.wu@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jian J Wang <jian.j.wang@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Eric Dong <eric.dong@intel.com>
2018-09-26 10:20:33 +08:00
5267926134 MdeModulePkg/DxeIpl: support more NX related PCDs
BZ#1116: https://bugzilla.tianocore.org/show_bug.cgi?id=1116

Currently IA32_EFER.NXE is only set against PcdSetNxForStack. This
confuses developers because following two other PCDs also need NXE
to be set, but actually not.

    PcdDxeNxMemoryProtectionPolicy
    PcdImageProtectionPolicy

This patch solves this issue by adding logic to enable IA32_EFER.NXE
if any of those PCDs have anything enabled.

Cc: Star Zeng <star.zeng@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jian J Wang <jian.j.wang@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
2018-09-26 08:55:10 +08:00
b888c57a05 MdeModulePkg/MdeModulePkg.dec/.uni: clarify PCDs usage
BZ#1116: https://bugzilla.tianocore.org/show_bug.cgi?id=1116

The usage of following PCDs described in MdeModulePkg.dec don't match
the implementation exactly. This patch updates related description in
both .dec and .uni files to avoid confusion in platform configuration.

  PcdSetNxForStack
  PcdImageProtectionPolicy
  PcdDxeNxMemoryProtectionPolicy

The main change is at the statement on how to handle the FALSE or 0
setting value in those PCDs. Current statement says the implementation
should unset or disable related features but in fact the related code
just do nothing (leave it to AS-IS). That means the result might be
disabled, or might be not. It depends on other features or platform
policy.

For example, if one don't want to enforce NX onto stack memory, he/she
needs to set PcdSetNxForStack to FALSE as well as to clear BIT4 of
PcdDxeNxMemoryProtectionPolicy.

Cc: Star Zeng <star.zeng@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jian J Wang <jian.j.wang@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
2018-09-26 08:55:09 +08:00
67998a547e UefiCpuPkg SecCore:Add a GUID removed previously
The Guid gPeiSecPerformancePpiGuid removed previously
is added into SecCore.inf.
https://bugzilla.tianocore.org/show_bug.cgi?id=1203

Cc: Eric Dong <eric.dong@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Eric Dong <eric.dong@intel.com>
2018-09-26 08:39:22 +08:00
e5cd809087 MdeModulePkg: XhciDxe: Prevent illegal memory access in XhcSetHsee
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1206

Newly added XhcSetHsee() routine reads 4 bytes into
a UINT16 variable causing issues on PCIE and NonDiscoverable
Xhci controllers. Fix that.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Fei1 Wang <fei1.wang@intel.com>
Cc: Star Zeng <star.zeng@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Marcin Wojtas <mw@semihalf.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-09-26 08:33:52 +08:00
3cb0a311cb MdePkg: Removing ipf which is no longer supported from edk2.
Removing rules for Ipf sources file:
* Remove the source file which path with "ipf" and also listed in
  [Sources.IPF] section of INF file.
* Remove the source file which listed in [Components.IPF] section
  of DSC file and not listed in any other [Components] section.
* Remove the embedded Ipf code for MDE_CPU_IPF.

Removing rules for Inf file:
* Remove IPF from VALID_ARCHITECTURES comments.
* Remove DXE_SAL_DRIVER from LIBRARY_CLASS in [Defines] section.
* Remove the INF which only listed in [Components.IPF] section in DSC.
* Remove statements from [BuildOptions] that provide IPF specific flags.
* Remove any IPF sepcific sections.

Removing rules for Dec file:
* Remove [Includes.IPF] section from Dec.

Removing rules for Dsc file:
* Remove IPF from SUPPORTED_ARCHITECTURES in [Defines] section of DSC.
* Remove any IPF specific sections.
* Remove statements from [BuildOptions] that provide IPF specific flags.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Chen A Chen <chen.a.chen@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-09-25 23:48:21 +08:00
ba6037f833 EmulatorPkg: Removing ipf which is no longer supported from edk2.
Removing rules for Ipf sources file:
* Remove the source file which path with "ipf" and also listed in
  [Sources.IPF] section of INF file.
* Remove the source file which listed in [Components.IPF] section
  of DSC file and not listed in any other [Components] section.
* Remove the embedded Ipf code for MDE_CPU_IPF.

Removing rules for Inf file:
* Remove IPF from VALID_ARCHITECTURES comments.
* Remove DXE_SAL_DRIVER from LIBRARY_CLASS in [Defines] section.
* Remove the INF which only listed in [Components.IPF] section in DSC.
* Remove statements from [BuildOptions] that provide IPF specific flags.
* Remove any IPF sepcific sections.

Removing rules for Dec file:
* Remove [Includes.IPF] section from Dec.

Removing rules for Dsc file:
* Remove IPF from SUPPORTED_ARCHITECTURES in [Defines] section of DSC.
* Remove any IPF specific sections.
* Remove statements from [BuildOptions] that provide IPF specific flags.

Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Andrew Fish <afish@apple.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Chen A Chen <chen.a.chen@intel.com>
Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2018-09-25 23:46:00 +08:00
bc7c34afb1 EdkCompatibilityPkg: Removing ipf from edk2.
Removing rules for Ipf sources file:
* Remove the source file which path with "ipf" and also listed in
  [Sources.IPF] section of INF file.
* Remove the source file which listed in [Components.IPF] section
  of DSC file and not listed in any other [Components] section.
* Remove the embedded Ipf code for MDE_CPU_IPF.

Removing rules for Inf file:
* Remove IPF from VALID_ARCHITECTURES comments.
* Remove DXE_SAL_DRIVER from LIBRARY_CLASS in [Defines] section.
* Remove the INF which only listed in [Components.IPF] section in DSC.
* Remove statements from [BuildOptions] that provide IPF specific flags.
* Remove any IPF sepcific sections.

Removing rules for Dec file:
* Remove [Includes.IPF] section from Dec.

Removing rules for Dsc file:
* Remove IPF from SUPPORTED_ARCHITECTURES in [Defines] section of DSC.
* Remove any IPF specific sections.
* Remove statements from [BuildOptions] that provide IPF specific flags.

The following rules are specially proposed by package owner:
* Remove CommonIpf.dsc file.
* Update Common.dsc file, to remove the section with IPF key.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Chen A Chen <chen.a.chen@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2018-09-25 23:43:46 +08:00
94d67262d8 CryptoPkg: Removing ipf which is no longer supported from edk2.
Removing rules for Ipf sources file:
* Remove the source file which path with "ipf" and also listed in
  [Sources.IPF] section of INF file.
* Remove the source file which listed in [Components.IPF] section
  of DSC file and not listed in any other [Components] section.
* Remove the embedded Ipf code for MDE_CPU_IPF.

Removing rules for Inf file:
* Remove IPF from VALID_ARCHITECTURES comments.
* Remove DXE_SAL_DRIVER from LIBRARY_CLASS in [Defines] section.
* Remove the INF which only listed in [Components.IPF] section in DSC.
* Remove statements from [BuildOptions] that provide IPF specific flags.
* Remove any IPF sepcific sections.

Removing rules for Dec file:
* Remove [Includes.IPF] section from Dec.

Removing rules for Dsc file:
* Remove IPF from SUPPORTED_ARCHITECTURES in [Defines] section of DSC.
* Remove any IPF specific sections.
* Remove statements from [BuildOptions] that provide IPF specific flags.

The following rules are specially proposed by package owner:
* Remove whole "CryptRuntimeDxe" folder which was designed for IPF.
* Remove whole "Include/Protocol" folder
* Update .Dec and .Dsc file accordingly.

Cc: Qin Long <qin.long@intel.com>
Cc: Ting Ye <ting.ye@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Chen A Chen <chen.a.chen@intel.com>
Reviewed-by: Ye Ting <ting.ye@intel.com>
Reviewed-by: Long Qin <qin.long@intel.com>
2018-09-25 23:40:41 +08:00
3888487499 SignedCapsulePkg: Remove PalLib in dsc which was missed at 52664c5
The PalLib is IPF specific and will be removed from MdePkg.
So this patch removes PalLib in SignedCapsulePkg.dsc which was missed
at 52664c5252.

Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Chen A Chen <chen.a.chen@intel.com>
Reviewed-by: Jiewen Yao <jiewen.yao@intel.com>
2018-09-25 23:40:40 +08:00
39b6f6ce1f MdeModulePkg: Remove PalLib in dsc which was missed at de00522
The PalLib is IPF specific and will be removed from MdePkg.
So this patch removes PalLib in MdeModulePkg.dsc which was missed
at de005223b7.

Cc: Star Zeng <star.zeng@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Chen A Chen <chen.a.chen@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-09-25 23:40:40 +08:00
5090a3dc5e IntelFrameworkModulePkg: Remove PalLib in dsc
Remove PalLib in dsc which was missed at 64bbf1d.
The PalLib is IPFspecific and will be removed from MdePkg.
So this patch removes PalLib in IntelFrameworkModulePkg.dsc
which was missed at 64bbf1dee2.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Chen A Chen <chen.a.chen@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-09-25 23:40:39 +08:00
17634d026f MdePkg/SynchronizationLib: fix Interlocked[De|In]crement return value
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1197

Today's InterlockedIncrement()/InterlockedDecrement() guarantees to
perform atomic increment/decrement but doesn't guarantee the return
value equals to the new value.

The patch fixes the behavior to use "XADD" instruction to guarantee
the return value equals to the new value.

The patch calls intrinsic functions for MSVC tool chain, calls the
NASM implementation for INTEL tool chain and calls GCC inline
assembly implementation (GccInline.c) for GCC tool chain.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
2018-09-25 10:02:53 +08:00
ca3e4f8ab8 MdePkg UefiPciLibPciRootBridgeIo: Remove redundant dependency
PiDxe.h is not used PciSegmentLib.h.
So <PiDxe.h> is deleted.
https://bugzilla.tianocore.org/show_bug.cgi?id=1184

v2:Update the bugzilla link.

Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-09-25 09:49:18 +08:00
32eb6739b9 MdePkg UefiPciSegmentLibPciRootBridgeIo: Remove redundant dependency
PiDxe.h is not used PciSegmentLib.h.
So "#include <PiDxe.h>" is deleted.
https://bugzilla.tianocore.org/show_bug.cgi?id=1183

Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-09-25 09:49:14 +08:00
d5b28edd63 MdePkg: Add a inf path in MdePkg.dsc
DxeRuntimeDebugLibSerialPort.inf is missing in MdePkg.dsc.
So add the path into it.
https://bugzilla.tianocore.org/show_bug.cgi?id=1177

Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-09-25 09:49:11 +08:00
0fd04efd01 BaseTools: Latter full value should overwrite the former field value.
For structure Pcd, the latter full assign value in commandLine should
override the former field assign value. For example in commandLine,
build --pcd Token.pcd.field="haha" --pcd Token.pcd=H"{0x01,0x02}",
the former field value "haha" will be ignored and overwrite by the latter
full value "{0x01,0x02}".

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: ZhiqiangX Zhao <zhiqiangx.zhao@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-09-25 09:43:51 +08:00
b602265d55 MdeModulePkg RegularExpressionDxe: Update Oniguruma to 6.9.0
https://bugzilla.tianocore.org/show_bug.cgi?id=1200
Update Oniguruma to the latest version v6.9.0.
Oniguruma https://github.com/kkos/oniguruma
Verify VS2017, GCC5 build.
Verify RegularExpressionProtocol GetInfo() and Match() function.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Dongao Guo <dongao.guo@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Cinnamon Shia <cinnamon.shia@hpe.com>
2018-09-25 08:52:51 +08:00
a364928195 UefiCpuPkg PiSmmCpuDxeSmm: Update SmiEntry function run the same position
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1191

Before commit e21e355e2c, jmp _SmiHandler
is commented. And below code, ASM_PFX(CpuSmmDebugEntry) is moved into rax,
then call it. But, this code doesn't work in XCODE5 tool chain. Because XCODE5
doesn't generated the absolute address in the EFI image. So, rax stores the
relative address. Once this logic is moved to another place, it will not work.
;   jmp     _SmiHandler                 ; instruction is not needed
...
    mov     rax, ASM_PFX(CpuSmmDebugEntry)
    call    rax

Commit e21e355e2c is to support XCODE5.
One tricky way is selected to fix it. Although SmiEntry logic is copied to
another place and run, but here jmp _SmiHandler is enabled to jmp the original
code place, then call ASM_PFX(CpuSmmDebugEntry) with the relative address.
    mov     rax, strict qword 0         ;   mov     rax, _SmiHandler
_SmiHandlerAbsAddr:
    jmp     rax
...
    call    ASM_PFX(CpuSmmDebugEntry)

Now, BZ 1191 raises the issue that SmiHandler should run in the copied address,
can't run in the common address. So, jmp _SmiHandler is required to be removed,
the code is kept to run in copied address. And, the relative address is
requried to be fixed up to the absolute address. The necessary changes should
not affect the behavior of platforms that already consume PiSmmCpuDxeSmm.
OVMF SMM boot to shell with VS2017, GCC5 and XCODE5 tool chain has been verified.
...
    mov     rax, strict qword 0         ;   call    ASM_PFX(CpuSmmDebugEntry)
CpuSmmDebugEntryAbsAddr:
    call    rax

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Eric Dong <eric.dong@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Jiewen Yao <jiewen.yao@intel.com>
2018-09-25 08:25:41 +08:00
60eb6c6d2e MdePkg/BasePeCoffLib: remove PE/COFF header workaround for ELILO on IPF
Now that Itanium support has been dropped, we can remove the various
occurrences of the ELILO on Itanium PE/COFF header workaround.

Link: https://bugzilla.tianocore.org/show_bug.cgi?id=816
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Jiewen Yao <jiewen.yao@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-09-24 16:56:44 +02:00
05cfd73f01 MdeModulePkg: remove PE/COFF header workaround for ELILO on IPF
Now that Itanium support has been dropped, we can remove the various
occurrences of the ELILO on Itanium PE/COFF header workaround.

Link: https://bugzilla.tianocore.org/show_bug.cgi?id=816
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Jiewen Yao <jiewen.yao@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-09-24 16:56:32 +02:00
997731e796 UefiCpuPkg: Remove redundant library classes, Ppis and GUIDs
Some redundant library classes Ppis and GUIDs
have been removed in inf, .c and .h files.

v2:
1.Remove ReadOnlyVariable2.h in S3Resume.c which should be
  deleted in last version in which gEfiPeiReadOnlyVariable2PpiGuid
  was removed.
2.Remove the library class BaseLib in CpuPageTable.c
  which is included elsewhere.
3.Add library classes in SecCore.inf which are removed
  at last version.
  They are DebugAgentLib and CpuExceptionHandlerLib.
4.Add two Ppis in SecCore.inf which are removed
  at last version.
  They are gEfiSecPlatformInformationPpiGuid and
  gEfiSecPlatformInformation2PpiGuid.

https://bugzilla.tianocore.org/show_bug.cgi?id=1043
https://bugzilla.tianocore.org/show_bug.cgi?id=1013
https://bugzilla.tianocore.org/show_bug.cgi?id=1032
https://bugzilla.tianocore.org/show_bug.cgi?id=1016

Cc: Eric Dong <eric.dong@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Eric Dong <eric.dong@intel.com>
2018-09-21 14:42:53 +08:00
a8beaf8f90 SecurityPkg/TcgStorageOpalLib: Fixed correct user password not works issue.
After admin password reach the TryLimit value, code logic will direct return
error password result no matter which password been inputted. So even correct
user password will return TryLimit error.

Now update code logic to also check user password. Only when both user/admin
password reach the TryLimit count, code will return exceed TryLimit error.

Change-Id: I17fbd1425b77150eb933a5002a1f87a0f0032e9b
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Hao Wu <hao.a.wu@intel.com>
2018-09-21 14:42:53 +08:00
8c06d18bc1 MdeModulePkg CapsuleApp:Remove two redundant Guids
Remove two redundant Guids which are not used.
They are gEfiCertTypeRsa2048Sha256Guid and
gEfiCertPkcs7Guid.This is an improved version of
https://bugzilla.tianocore.org/show_bug.cgi?id=1062

Cc: Star Zeng <star.zeng@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-09-21 11:34:42 +08:00
4cd9d7fc6f MdeModulePkg/SdMmcPciHcDxe: Execute card detect only for RemovableSlot
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1182

Some devices can be non removable (such as eMMC) and checking
Present State Register on host controller may falsely return
an information that device is not present. Execute this
check conditionally on the SloType field value.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Marcin Wojtas <mw@semihalf.com>
Reviewed-by: Hao Wu <hao.a.wu@intel.com>
2018-09-21 10:47:48 +08:00
064d301fb0 MdeModulePkg/SdMmcPciHcDxe: Fix SdMmcHcReset to set only necesery bits
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1185

SdMmcHcReset used to set all bits of Software Reset Register to 1
including reserved ones, which on some controllers may result in
timeout.

Now only first bit is set, which means "Software Reset for All".

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Marcin Wojtas <mw@semihalf.com>
Reviewed-by: Hao Wu <hao.a.wu@intel.com>
2018-09-21 10:47:47 +08:00
9062ab4787 IntelFsp2Pkg: Fix typo in SplitFspBin
Signed-off-by: Patrick Georgi <pgeorgi@google.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Reviewed-by: Nate DeSimone <nathaniel.l.desimone@intel.com>
Reviewed-by: Chasel Chiu <chasel.chiu@intel.com>
2018-09-21 07:38:09 +08:00
719fd85c73 BaseTools: refactor to cache InfBuildData data
use Common.caching and auto cache properties and functions of InfBuildData

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-09-20 22:18:08 +08:00
4def57d98c BaseTools: refactor to not overcreate ModuleAutoGen objects
currently created for 3 different purposes and saved once.
this makes it created once and saved and then referenced.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-09-20 22:18:08 +08:00
6f73a03665 BaseTools: Don't save unused workspace data
FlexibleFieldName was never used not set.
  DefinitionPosition (file and line number) are recalculated
and never used outside the function.  remove the saving of the
data.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-09-20 22:18:07 +08:00
b0a2645162 BaseTools: refactor Build Database objects
1) use namedtuple instead of custom class when apropriate
2) rename collections.OrderedDict to OrderedDict since we import it already

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-09-20 22:18:07 +08:00
71cac3f791 BaseTools: Workspace classes refactor properties
1) use decorators
2) also change some private functions to public when all callers are
external
3) change external callers to use functions instead of directly
accessing private data.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-09-20 22:18:06 +08:00
6c204ed4f2 BaseTools: refactor class properties
use decorators and auto cache those that were cached manually
remove properties never used

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-09-20 22:18:06 +08:00
4c92c81d61 BaseTools: AutoGen - refactor class properties
use function decorators

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-09-20 22:18:06 +08:00
e6c62ab768 BaseTools: AutoGen refactor WorkspaceAutoGen class
Update the WorkspaceAutoGen class to use caching decorators and remove
the no longer needed private variables.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-09-20 22:18:05 +08:00
e6eae3b4c7 BaseTools: Refactor PlatformAutoGen
use decorators for property and automatic caching
remove circular dependency between some APIs

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Bob Feng <Bob.c.Feng@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-09-20 22:18:05 +08:00
f199664ce7 SecurityPkg: remove PE/COFF header workaround for ELILO on IPF
Now that Itanium support has been dropped, we can remove the various
occurrences of the ELILO on Itanium PE/COFF header workaround.

Link: https://bugzilla.tianocore.org/show_bug.cgi?id=816
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Yao Jiewen <jiewen.yao@intel.com>
Reviewed-by: Zhang Chao B <chao.b.zhang@intel.com>
2018-09-20 22:01:50 +08:00
ae57950fc8 BaseTools: Fix a bug for Unused PCDs section display in the report
Fix a regression issue caused by ac4578af36, when there doesn't exist
not used PCD, it also display the not used Pcd section in the report.

Fixes: https://bugzilla.tianocore.org/show_bug.cgi?id=1170
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-09-18 09:41:26 +08:00
3eb747c16b BaseTools: Regression bug Linux script used windows format
regression by 15e2022825
and 9f3594782d
Linux execute script must use '\n' not '\r\n' for end of line

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-09-18 09:40:30 +08:00
1737b7e3b4 MdeModulePkg/Ip4Dxe: Refine the coding style.
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1180
Remove the trailing white spaces.

Cc: Ye Ting <ting.ye@intel.com>
Cc: Fu Siyuan <siyuan.fu@intel.com>
Cc: Bi Dandan <dandan.bi@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Wu Jiaxin <jiaxin.wu@intel.com>
Reviewed-by: Fu Siyuan <siyuan.fu@intel.com>
Reviewed-by: Dandan Bi <dandan.bi@intel.com>
2018-09-18 09:30:57 +08:00
cde5a72d36 NetworkPkg: UefiPxeBcDxe: Add EXCLUSIVE attribute when opening SNP protocol installed by PXE.
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1152

v2: Sync the same logic to Ipv6 and update code comments.

The PXE driver installs a SNP and open this SNP with attribute BY_DRIVER
to avoid it being opened by MNP driver, this SNP is also expected not to
be opened by other drivers with EXCLUSIVE attribute. In some cases, other
drivers may happen to do this by error, and thus cause a system crash.
This patch adds EXCLUSIVE attribute when opening SNP in PXE driver, and
will reject all OpenProtocol requests by EXCLUSIVE.

Cc: Subramanian, Sriram <sriram-s@hpe.com>
Cc: Ye Ting <ting.ye@intel.com>
Cc: Fu Siyuan <siyuan.fu@intel.com>
Cc: Wu Jiaxin <jiaxin.wu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Wang Fan <fan.wang@intel.com>
Reviewed-by: Sriram Subramanian <sriram-s@hpe.com>
Reviewed-by: Fu Siyuan <siyuan.fu@intel.com>
Reviewed-by: Wu Jiaxin <jiaxin.wu@intel.com>
2018-09-17 14:32:35 +08:00
16d8465756 UefiCpuPkg/PiSmmCpuDxeSmm: move InitSmmS3Cr3() into else block
BZ#: https://bugzilla.tianocore.org/show_bug.cgi?id=1165

InitSmmS3Cr3 () will update SmmS3ResumeState so moving the calling of
it into else block to keep the logic consistency.

Cc: Star Zeng <star.zeng@intel.com>
Cc: Benjamin You <benjamin.you@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jian J Wang <jian.j.wang@intel.com>
Reviewed-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
2018-09-17 10:45:53 +08:00
a95c9cfd18 UefiCpuPkg/PiSmmCpuDxeSmm: add message for S3 config error
BZ#: https://bugzilla.tianocore.org/show_bug.cgi?id=1165

HOB gEfiAcpiVariableGuid is a must have data for S3 resume if
PcdAcpiS3Enable is set to TRUE. Current code in CpuS3.c doesn't
embody this strong binding between them. An error message and
CpuDeadLoop are added in this patch to warn platform developer
about it.

Cc: Star Zeng <star.zeng@intel.com>
Cc: Benjamin You <benjamin.you@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jian J Wang <jian.j.wang@intel.com>
Reviewed-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
2018-09-17 10:45:52 +08:00
b06dfd40bb IntelSiliconPkg IntelVTdDxe: Check HeaderType if func 0 is implemented
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1169

Current code checks HeaderType of Function 0 even Function 0 is not
implemented. HeaderType value will be 0xFF if Function 0 is not
implemented, then MaxFunction will be set to PCI_MAX_FUNC + 1.

The code can be optimized to only check HeaderType if Function 0 is
implemented.

Test done:
With this patch, the result is same with the result after the patch at
https://lists.01.org/pipermail/edk2-devel/2018-September/029623.html.

Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Rangasai V Chaganty <rangasai.v.chaganty@intel.com>
Cc: Tomson Chang <tomson.chang@intel.com>
Cc: Jenny Huang <jenny.huang@intel.com>
Cc: Amy Chan <amy.chan@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Jiewen Yao <jiewen.yao@intel.com>
2018-09-17 09:17:58 +08:00
e69d7e99e7 IntelSiliconPkg IntelVTdDxe: Optimize when func 0 is not implemented
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1169

PCI spec:
They are also required to always implement function 0 in the device.
Implementing other functions is optional and may be assigned in any
order (i.e., a two-function device must respond to function 0 but
can choose any of the other possible function numbers (1-7) for the
second function).

This patch updates ScanPciBus() to not scan other functions if
function 0 is not implemented.

Test done:
Added debug code below in the second loop of ScanPciBus(),
compared the debug logs with and without this patch, many
non-0 unimplemented functions are skipped correctly.

  DEBUG ((
    DEBUG_INFO,
    "%a() B%02xD%02xF%02x VendorId: %04x DeviceId: %04x\n",
    __FUNCTION__,
    Bus,
    Device,
    Function,
    VendorID,
    DeviceID
    ));

Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Rangasai V Chaganty <rangasai.v.chaganty@intel.com>
Cc: Tomson Chang <tomson.chang@intel.com>
Cc: Jenny Huang <jenny.huang@intel.com>
Cc: Amy Chan <amy.chan@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Jiewen Yao <jiewen.yao@intel.com>
2018-09-17 09:17:42 +08:00
cfd10276ce MdePkg: Update the comments in dec
Correct the directory of a header file in comments.
https://bugzilla.tianocore.org/show_bug.cgi?id=1172

Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-09-14 14:35:15 +08:00
85e5d3cf6b BaseTools: Check GUID C structure format
GUID C format must conform to {8,4,4,{2,2,2,2,2,2,2,2}}

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
2018-09-14 13:41:07 +08:00
f316a26013 MdeModulePkg: Avoid key notification called more than once
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=996

Issue:
In current code logic, when a key is pressed, it will search
the whole NotifyList to find whether a notification has been
registered with the keystroke. if yes, it will en-queue the
key for notification execution later. And now if different
notification functions have been registered with the same key,
then the key will be en-queued more than once. Then it will
cause the notification executed more than once.

This patch is to enhance the code logic to fix this issue.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Star Zeng <star.zeng@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Dandan Bi <dandan.bi@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-09-14 10:18:31 +08:00
f92b8ee3b7 IntelFrameworkModulePkg: Avoid key notification called more than once
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=996

Issue:
In current code logic, when a key is pressed, it will search
the whole NotifyList to find whether a notification has been
registered with the keystroke. if yes, it will en-queue the
key for notification execution later. And now if different
notification functions have been registered with the same key,
then the key will be en-queued more than once. Then it will
cause the notification executed more than once.

This patch is to enhance the code logic to fix this issue.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Dandan Bi <dandan.bi@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-09-14 10:18:31 +08:00
9c8e9e76bd EmbeddedPkg/VirtualKeyboard: Avoid notification called more than once
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=996

Issue:
In current code logic, when a key is pressed, it will search
the whole NotifyList to find whether a notification has been
registered with the keystroke. if yes, it will en-queue the
key for notification execution later. And now if different
notification functions have been registered with the same key,
then the key will be en-queued more than once. Then it will
cause the notification executed more than once.

This patch is to enhance the code logic to fix this issue.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Cc: Leif Lindholm <leif.lindholm@linaro.org>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Dandan Bi <dandan.bi@intel.com>
Reviewed-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-09-14 10:18:31 +08:00
52664c5252 SignedCapsulePkg: Removing ipf which is no longer supported from edk2.
Removing rules for Ipf sources file:
* Remove the source file which path with "ipf" and also listed in
  [Sources.IPF] section of INF file.
* Remove the source file which listed in [Components.IPF] section
  of DSC file and not listed in any other [Components] section.
* Remove the embedded Ipf code for MDE_CPU_IPF.

Removing rules for Inf file:
* Remove IPF from VALID_ARCHITECTURES comments.
* Remove DXE_SAL_DRIVER from LIBRARY_CLASS in [Defines] section.
* Remove the INF which only listed in [Components.IPF] section in DSC.
* Remove statements from [BuildOptions] that provide IPF specific flags.
* Remove any IPF sepcific sections.

Removing rules for Dec file:
* Remove [Includes.IPF] section from Dec.

Removing rules for Dsc file:
* Remove IPF from SUPPORTED_ARCHITECTURES in [Defines] section of DSC.
* Remove any IPF specific sections.
* Remove statements from [BuildOptions] that provide IPF specific flags.

Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Chen A Chen <chen.a.chen@intel.com>
Reviewed-by: <Jiewen.yao@intel.com>
2018-09-13 13:12:03 +08:00
29cc93f773 PcAtChipsetPkg: Removing ipf which is no longer supported from edk2.
Removing rules for Ipf sources file:
* Remove the source file which path with "ipf" and also listed in
  [Sources.IPF] section of INF file.
* Remove the source file which listed in [Components.IPF] section
  of DSC file and not listed in any other [Components] section.
* Remove the embedded Ipf code for MDE_CPU_IPF.

Removing rules for Inf file:
* Remove IPF from VALID_ARCHITECTURES comments.
* Remove DXE_SAL_DRIVER from LIBRARY_CLASS in [Defines] section.
* Remove the INF which only listed in [Components.IPF] section in DSC.
* Remove statements from [BuildOptions] that provide IPF specific flags.
* Remove any IPF sepcific sections.

Removing rules for Dec file:
* Remove [Includes.IPF] section from Dec.

Removing rules for Dsc file:
* Remove IPF from SUPPORTED_ARCHITECTURES in [Defines] section of DSC.
* Remove any IPF specific sections.
* Remove statements from [BuildOptions] that provide IPF specific flags.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Chen A Chen <chen.a.chen@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-09-13 13:12:03 +08:00
6861b0fe72 IntelSiliconPkg: Removing ipf which is no longer supported from edk2.
Removing rules for Ipf sources file:
* Remove the source file which path with "ipf" and also listed in
  [Sources.IPF] section of INF file.
* Remove the source file which listed in [Components.IPF] section
  of DSC file and not listed in any other [Components] section.
* Remove the embedded Ipf code for MDE_CPU_IPF.

Removing rules for Inf file:
* Remove IPF from VALID_ARCHITECTURES comments.
* Remove DXE_SAL_DRIVER from LIBRARY_CLASS in [Defines] section.
* Remove the INF which only listed in [Components.IPF] section in DSC.
* Remove statements from [BuildOptions] that provide IPF specific flags.
* Remove any IPF sepcific sections.

Removing rules for Dec file:
* Remove [Includes.IPF] section from Dec.

Removing rules for Dsc file:
* Remove IPF from SUPPORTED_ARCHITECTURES in [Defines] section of DSC.
* Remove any IPF specific sections.
* Remove statements from [BuildOptions] that provide IPF specific flags.

Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Rangasai V Chaganty <rangasai.v.chaganty@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Chen A Chen <chen.a.chen@intel.com>
Reviewed-by: <Jiewen.yao@intel.com>
2018-09-13 13:12:02 +08:00
7c41b8135d BaseTools: Align the boolean type PCD value's display in the report
This patch align the boolean type PCD value's display in the build
report. Original it may display 0x0, also may use 0 for the same PCD.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-09-13 09:28:55 +08:00
eb2afe08ce BaseTools\GenFds: remove extra content
remove uncalled functions
remove extra blank lines
remove commented out code

Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-09-13 09:27:28 +08:00
e9d21340c6 MdeModulePkg/Library/DxeHttpLib: Handle the blank value in HTTP header.
This patch is to resolve the lock-up issue if the value of HTTP header
is blank.  The issue is recorded @
https://bugzilla.tianocore.org/show_bug.cgi?id=1102.

Cc: Stephen Benjamin <stephen@redhat.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Ye Ting <ting.ye@intel.com>
Cc: Fu Siyuan <siyuan.fu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Wu Jiaxin <jiaxin.wu@intel.com>
Tested-by: Stephen Benjamin <stephen@redhat.com>
Reviewed-by: Fu Siyuan <siyuan.fu@intel.com>
2018-09-13 09:09:51 +08:00
ecbd055d4e MdeModulePkg/Ip4Dxe: Sync the direct route entry setting.
v2: use "IP & Netmask" directly instead of defining an additional variable.

This patch is to sync the direct route entry setting in both the default
and Instance route table {Subnet, Mask, NextHope} (
https://bugzilla.tianocore.org/show_bug.cgi?id=1143).

Cc: Ye Ting <ting.ye@intel.com>
Cc: Fu Siyuan <siyuan.fu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Wu Jiaxin <jiaxin.wu@intel.com>
Reviewed-by: Ye Ting <ting.ye@intel.com>
Reviewed-by: Fu Siyuan <siyuan.fu@intel.com>
2018-09-13 09:09:08 +08:00
4423f0bc61 BaseTools: Fix the RaiseError variable issue caused by 855698fb69
The bug is that it cause the RaiseError always be set to TRUE even we
call the function with FALSE parameter.

Cc: Hess Chen <hesheng.chen@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Hess Chen <hesheng.chen@intel.com>
2018-09-12 13:20:46 +08:00
4c0d19e5bf BaseTools: Support multi thread build Basetool on Windows
Add NmakeSubdirs.py to replace NmakeSubdirs.bat in VS Makefile. This script will
invoke nmake in multi thread mode. It can save more than half time of BaseTools
C clean build.
GCC make supports multiple thread in make phase. So, GNUmakefile doesn't need apply
this script.

single task or job=1:
    just single thread and invoke subprocess,subprocess will use
    system.stdout to print output.
multi task:
    thread number is logic cpu count.All subprocess output will pass to
    python script by PIPE and then script print it to system.stdout.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Dongao Guo<dongao.guo@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Test-by: Liming Gao <liming.gao@intel.com>
2018-09-12 13:02:11 +08:00
73dbd6afab MdeModulePkg XhciDxe: Set HSEE Bit if SERR# Enable Bit is set
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1167

When the HSEE in the USBCMD bit is a ‘1’ and the HSE bit in the
USBSTS register is a ‘1’, the xHC shall assert out-of-band error
signaling to the host and assert the SERR# pin.
To prevent masking any potential issues with SERR, this patch is
to set USBCMD Host System Error Enable(HSEE) Bit if PCICMD SERR#
Enable Bit is set.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Jian J Wang <jian.j.wang@intel.com>
Cc: Fei1 Wang <fei1.wang@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-09-12 12:57:16 +08:00
84a52d4d03 BaseTools: SKU inheritance.
If the SkuB's parent SkuA is not in SKUID_IDENTIFIER, then
make SkuB inherit from SkuA as if the SKUID_INDENTIFIER
is ALL.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: ZhiqiangX Zhao <zhiqiangx.zhao@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-09-12 11:25:23 +08:00
4c6fda33c8 BaseTools: Correct DXE_PCD_DATABASE_INIT.
Add the handle of PCD_DATABASE_INIT and PCD_DATABASE_UNINIT
for Boolean type pcd.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: ZhiqiangX Zhao <zhiqiangx.zhao@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-09-12 11:25:22 +08:00
ced8685838 BaseTools: Check PcdNvStoreDefaultValueBuffer.
Build tool should report warning if a platform
defines [DefaultStores] but forgets to defined
PcdNvStoreDefaultValueBuffer as PcdsDynamicExVpd in dsc file.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: ZhiqiangX Zhao <zhiqiangx.zhao@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-09-12 11:25:22 +08:00
cfed8a37ec BaseTools: Involve Dec default value to calculate Maxsize
Involve Dec default value to calculate Maxsize for structure PCD

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Jaben Carsey <jaben.carsey@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-09-12 11:25:21 +08:00
71127ce883 BaseTools: Structure Pcd value override incorrect.
This patch is going to fix the issue that
The Pcd field value is override incorrectly when there
is no Pcd overall value assignment in Dsc file.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-09-12 11:25:20 +08:00
b62cbfb787 BaseTools: Report error for incorrect hex value format
The case is user use 0x1} as a hex value for Pcd, it directly cause
tool report traceback info. This patch add more error info.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-09-12 10:57:00 +08:00
5c9ac43f47 MdeModulePkg/ConPlatform: Support short-form USB device path
Today's implementation does an exact device path match to check
whether the device path of a console is in ConIn/ConOut/ErrOut.
But that doesn't work for the USB keyboard.
Because when a platform have multiple USB port, ConIn needs to
carry all device paths corresponding to each port.
Even worse, today's BDS core logic removes the device path from
ConIn/ConOut/ErrOut when the connection to that device path fails.
So if user switches the USB keyboard from one port to another across
boot, the USB keyboard doesn't work in the second boot.

ConPlatform driver solved this problem by adding the
IsHotPlugDevice() function. So that for USB keyboard, ConPlatform
doesn't care whether its device path is in ConIn or not.
But the rule is too loose, and now causes platform BDS cannot control
whether to enable USB keyboard as an active console.

The patch changes ConPlatform to support USB short-form device path
when checking whether the device path of a console is in
ConIn/ConOut/ErrOut.

The logic to always accept USB/PCCARD device as active console is
removed.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Hao A Wu <hao.a.wu@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-09-12 10:12:05 +08:00
1ad635b283 BaseTools/GenFds: delete unused file
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-09-11 21:12:56 +08:00
8e2018f944 UefiCpuPkg/CpuDxe: fix an incorrect bit-wise operation
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1164

The left operand is 64-bit but right operand could be 32-bit.
A typecast is a must because of '~' op before it.

Cc: Hao A Wu <hao.a.wu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jian J Wang <jian.j.wang@intel.com>
Reviewed-by: Hao Wu <hao.a.wu@intel.com>
2018-09-10 17:08:22 +08:00
51d0100b3c UefiCpuPkg/CpuDxe: fix ECC reported issues
BZ#: https://bugzilla.tianocore.org/show_bug.cgi?id=1160

There're two parameters which have different name in comment and prototype.

Cc: Dandan Bi <dandan.bi@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jian J Wang <jian.j.wang@intel.com>
Reviewed-by: Dandan Bi <dandan.bi@intel.com>
2018-09-10 17:08:21 +08:00
f4eaaf1a6d Emulator/Win: Fix build failure using VS2015x86 or old WinSDK
When build with WinSDK <= Win10 TH2, the terminal over CMD.exe
doesn't work. Because Win10 later than TH2 starts to support VT
terminal.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Hao A Wu <hao.a.wu@intel.com>
2018-09-10 10:29:35 +08:00
289cb872ed EmulatorPkg: Update package level Readme.md
Since the emulator under Windows is enabled, the patch changes
README to include the information of emulator under Windows.
It also changes README to Readme.md for better looking.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Hao A Wu <hao.a.wu@intel.com>
2018-09-10 10:29:34 +08:00
34c3405cb7 UefiCpuPkg/PeiCpuException: Fix coding style issue
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Dandan Bi <dandan.bi@intel.com>
2018-09-10 10:22:27 +08:00
3a34753312 SecurityPkg: HashLib: Change dos format
Change file format to DOS

Cc: Bi Dandan <dandan.bi@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhang Chao B <chao.b.zhang@intel.com>
Reviewed-by: Bi Dandan <dandan.bi@intel.com>
2018-09-10 09:43:15 +08:00
0a0d5296e4 UefiCpuPkg/CpuMpPei: support stack guard feature
This feature is the same as Stack Guard enabled in driver CpuDxe but
applies to PEI phase. Due to the specialty in PEI module dispatching,
this driver is changed to do the actual initialization in notify
callback of event gEfiPeiMemoryDiscoveredPpiGuid. This can let the
stack guard apply to as most PEI drivers as possible.

To let Stack Guard work, some simple page table management code are
introduced to setup Guard page at base of stack for each processor.

Cc: Eric Dong <eric.dong@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Star Zeng <star.zeng@intel.com>
Cc: "Ware, Ryan R" <ryan.r.ware@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jian J Wang <jian.j.wang@intel.com>
Regression-tested-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Eric Dong <eric.dong@intel.com>
2018-09-10 09:28:28 +08:00
e09b6b5953 UefiCpuPkg/MpInitLib: fix register restore issue in AP wakeup
The conflict issues are introduced by Stack Guard feature enabled for
PEI.

The first is CR0 which should be restored after CR3 and CR4.
Another is TR which should not be passed from BSP to AP during init
phase.

Cc: Eric Dong <eric.dong@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Star Zeng <star.zeng@intel.com>
Cc: "Ware, Ryan R" <ryan.r.ware@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jian J Wang <jian.j.wang@intel.com>
Regression-tested-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Eric Dong <eric.dong@intel.com>
2018-09-10 09:28:27 +08:00
fc0e7fd5e8 UefiCpuPkg/CpuExceptionHandlerLib: support stack switch for PEI exceptions
Stack Guard needs to setup stack switch capability to allow exception
handler to be called with good stack if stack overflow is detected.
This patch update InitializeCpuExceptionHandlersEx() to allow pass
extra initialization data used to setup exception stack switch for
specified exceptions.

Cc: Eric Dong <eric.dong@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Star Zeng <star.zeng@intel.com>
Cc: "Ware, Ryan R" <ryan.r.ware@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jian J Wang <jian.j.wang@intel.com>
Regression-tested-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Eric Dong <eric.dong@intel.com>
2018-09-10 09:28:26 +08:00
2af2988f3a MdeModulePkg/DxeIpl: disable paging before creating new page table
PEI Stack Guard needs to enable paging before DxeIpl. This might cause
#GP in the transition from 32-bit PEI to 64-bit DXE due to the code
trying to write CR3 register with PML4 page table while the processor
is enabled with PAE paging.

Simply disabling paging before updating CR3 can solve this conflict.
There's no such issue for 64-bit PEI so this change applies only to
32-bit code.

Cc: Star Zeng <star.zeng@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: "Ware, Ryan R" <ryan.r.ware@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jian J Wang <jian.j.wang@intel.com>
Regression-tested-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Eric Dong <eric.dong@intel.com>
2018-09-10 09:28:26 +08:00
4b2dc555d8 ShellPkg: Remove trailing white space
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1158

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Jaben Carsey <jaben.carsey@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Fu Siyuan <siyuan.fu@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-09-10 08:39:34 +08:00
40a7b235e4 MdeModulePkg PeiCore: Always use PeiImageRead() function to load PEI image
In V2, Remove GetImageReadFunction(), directly use PeiImageRead().

The copy PeiImageReadForShadow function doesn't improve the boot performance.
This patch removes this copy logic to simplify the code logic.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-09-07 21:39:26 +08:00
98257f9820 ShellPkg Shell: Remove an unused global variable
The unused global variable InvalidChars is removed.
It is only used in the function IsValidCommandName which
was removed previously.
https://bugzilla.tianocore.org/show_bug.cgi?id=1066

Cc: Jaben Carsey <jaben.carsey@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-09-07 13:10:14 +02:00
c4709260f6 EmbeddedPkg/CoherentDmaLib: Add missing checks to DmaMap
UEFI Sct validates Dma mapping. For CoherentDmaLib it always failed
because there were no required checks present in DmaMap.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Vladimir Olovyannikov <vladimir.olovyannikov@broadcom.com>
Reviewed-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
2018-09-07 12:23:32 +02:00
c783da6599 EmbeddedPkg/CoherentDmaLib: Fix typo in DmaAlignedBuffer
The only valid memory types for DmaAlignedBuffer should be
EfiBootServicesData and EfiRuntimeServicesData. However due to the typo,
there is no way to allocate runtime pages, and INVALID_PARAMETER is
always returned. Fix the typo.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Vladimir Olovyannikov <vladimir.olovyannikov@broadcom.com>
Reviewed-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
2018-09-07 12:21:28 +02:00
4d62189347 Maintainer.txt: Add Ray to be co-maintainer of EmulatorPkg
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
Cc: Andrew Fish <afish@apple.com>
2018-09-07 12:33:16 +08:00
7761cee050 BaseTools/GenFds: remove function without callers
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-09-07 11:22:48 +08:00
7bf1eb6ef8 BaseTools: refactor to remove duplicate functions
Update GenFdsGlobalVariable GetAlignment to support G.
replace use of local function in Region with updated shared function.

Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob C Feng <bob.c.feng@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-09-07 08:44:57 +08:00
b48ec0e8ab MdeModulePkg/EhciDxe: factor out EhcIsDebugPortInUse()
The EhcReset(), EhcGetRootHubPortStatus() and EhcDriverBindingStart()
functions need to see whether the host controller (or a specific port on
the host controller) can be accessed, dependent on the controller having
(or the specific port being) an in-use debug port. Because the condition
isn't simple, extract it to a separate function.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Star Zeng <star.zeng@intel.com>
Suggested-by: Star Zeng <star.zeng@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-09-06 14:07:50 +02:00
22cf747fcf ShellPkg Shell: Remove redundant functions
The redundant functions which are never called have been
removed. They are InternalShellProtocolDebugPrintMessage,
UpdateFileName,RemoveFileTag and IsValidCommandName.
https://bugzilla.tianocore.org/show_bug.cgi?id=1066

Cc: Jaben Carsey <jaben.carsey@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2018-09-06 17:47:52 +08:00
de005223b7 MdeModulePkg: Removing ipf which is no longer supported from edk2.
Removing rules for Ipf sources file:
* Remove the source file which path with "ipf" and also listed in
  [Sources.IPF] section of INF file.
* Remove the source file which listed in [Components.IPF] section
  of DSC file and not listed in any other [Components] section.
* Remove the embedded Ipf code for MDE_CPU_IPF.

Removing rules for Inf file:
* Remove IPF from VALID_ARCHITECTURES comments.
* Remove DXE_SAL_DRIVER from LIBRARY_CLASS in [Defines] section.
* Remove the INF which only listed in [Components.IPF] section in DSC.
* Remove statements from [BuildOptions] that provide IPF specific flags.
* Remove any IPF sepcific sections.

Removing rules for Dec file:
* Remove [Includes.IPF] section from Dec.

Removing rules for Dsc file:
* Remove IPF from SUPPORTED_ARCHITECTURES in [Defines] section of DSC.
* Remove any IPF specific sections.
* Remove statements from [BuildOptions] that provide IPF specific flags.

Cc: Star Zeng <star.zeng@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Chen A Chen <chen.a.chen@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-09-06 15:25:16 +08:00
8d27b54bae OptionRomPkg: Removing ipf which is no longer supported from edk2.
Removing rules for Ipf sources file:
* Remove the source file which path with "ipf" and also listed in
  [Sources.IPF] section of INF file.
* Remove the source file which listed in [Components.IPF] section
  of DSC file and not listed in any other [Components] section.
* Remove the embedded Ipf code for MDE_CPU_IPF.

Removing rules for Inf file:
* Remove IPF from VALID_ARCHITECTURES comments.
* Remove DXE_SAL_DRIVER from LIBRARY_CLASS in [Defines] section.
* Remove the INF which only listed in [Components.IPF] section in DSC.
* Remove statements from [BuildOptions] that provide IPF specific flags.
* Remove any IPF sepcific sections.

Removing rules for Dec file:
* Remove [Includes.IPF] section from Dec.

Removing rules for Dsc file:
* Remove IPF from SUPPORTED_ARCHITECTURES in [Defines] section of DSC.
* Remove any IPF specific sections.
* Remove statements from [BuildOptions] that provide IPF specific flags.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Chen A Chen <chen.a.chen@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-09-06 14:44:30 +08:00
c8ff1df3a2 FatPkg: Removing ipf which is no longer supported from edk2.
Removing rules for Ipf sources file:
* Remove the source file which path with "ipf" and also listed in
  [Sources.IPF] section of INF file.
* Remove the source file which listed in [Components.IPF] section
  of DSC file and not listed in any other [Components] section.
* Remove the embedded Ipf code for MDE_CPU_IPF.

Removing rules for Inf file:
* Remove IPF from VALID_ARCHITECTURES comments.
* Remove DXE_SAL_DRIVER from LIBRARY_CLASS in [Defines] section.
* Remove the INF which only listed in [Components.IPF] section in DSC.
* Remove statements from [BuildOptions] that provide IPF specific flags.
* Remove any IPF sepcific sections.

Removing rules for Dec file:
* Remove [Includes.IPF] section from Dec.

Removing rules for Dsc file:
* Remove IPF from SUPPORTED_ARCHITECTURES in [Defines] section of DSC.
* Remove any IPF specific sections.
* Remove statements from [BuildOptions] that provide IPF specific flags.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Chen A Chen <chen.a.chen@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-09-06 14:44:30 +08:00
8baa3fc168 FatBinPkg: Removing ipf which is no longer supported from edk2.
Removing rules for Ipf sources file:
* Remove the source file which path with "ipf" and also listed in
  [Sources.IPF] section of INF file.
* Remove the source file which listed in [Components.IPF] section
  of DSC file and not listed in any other [Components] section.
* Remove the embedded Ipf code for MDE_CPU_IPF.

Removing rules for Inf file:
* Remove IPF from VALID_ARCHITECTURES comments.
* Remove DXE_SAL_DRIVER from LIBRARY_CLASS in [Defines] section.
* Remove the INF which only listed in [Components.IPF] section in DSC.
* Remove statements from [BuildOptions] that provide IPF specific flags.
* Remove any IPF sepcific sections.

Removing rules for Dec file:
* Remove [Includes.IPF] section from Dec.

Removing rules for Dsc file:
* Remove IPF from SUPPORTED_ARCHITECTURES in [Defines] section of DSC.
* Remove any IPF specific sections.
* Remove statements from [BuildOptions] that provide IPF specific flags.

Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Chen A Chen <chen.a.chen@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-09-06 14:44:29 +08:00
6623f8fe66 EmulatorPkg/PlatformBmLib: Fix GCC build failure
Some local variables are initialized but never used.
GCC complains about that. The patch fixes this issue.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Dandan Bi <dandan.bi@intel.com>
2018-09-06 14:43:17 +08:00
6460513aa6 BaseTools/PatchCheck.py: Fix error when run with Python3
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1145

Currently run PatchCheck.py with Python3 will meet
following error:
.....
File "PatchCheck.py", line 554, in run_git
return Result[0].decode('utf-8', 'ignore') if Result[0] and
Result[0].find("fatal")!=0 else None
TypeError: a bytes-like object is required, not 'str'

This issue was introduce by commit:5ac4548cdf654.

This patch is to convert the str object of "fatal" to
byte object to fix this failure.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Jaben Carsey <jaben.carsey@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Dandan Bi <dandan.bi@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2018-09-06 13:41:35 +08:00
4a76d9b9a3 MdeModulePkg: Remove trailing white space
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1144

Cc: Dandan Bi <dandan.bi@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Jian J Wang <jian.j.wang@intel.com>
Cc: Siyuan Fu <siyuan.fu@intel.com>
Cc: Jiaxin Wu <jiaxin.wu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Siyuan Fu <siyuan.fu@intel.com>
Reviewed-by: Dandan Bi <dandan.bi@intel.com>
2018-09-06 09:17:03 +08:00
fa6e2804de BaseTools: Report more clear error message for PCD used in expression
Only the FeatureFlag type or FixedAtBuild type can be used in the
expression.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-09-06 09:12:56 +08:00
ac4578af36 BaseTools: Fix a bug about list the PCD in "not used" section
Defined a pcd in Ovmf.dec and used that pcd in AcpiPlatformDxe.inf,
then assign a value to that pcd from DSC, then build Ovmf platform
successfully. But this Pcd was wrongly listed into not used section
in the report.txt file.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-09-06 09:11:45 +08:00
e23d9c3ed8 UefiCpuPkg/MpInitLib: Fix ECC issues.
Fix trailing white spaces and invalid line ending issue.

Cc: Dandan Bi <dandan.bi@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Dandan Bi <dandan.bi@intel.com>
2018-09-06 08:04:29 +08:00
c3d5d800d7 MdeModulePkg/EhciDxe: fix host controller reset condition in BindingStart
Commit 09943f5ecc ("MdeModulePkg: Skip to manage usb debug port in EDKII
EHCI driver if it's used by usb debug port driver", 2012-04-28) made the
host controller reset in EhcDriverBindingStart() conditional. The intent
was to avoid the reset when
- one of the USB ports on the host controller was a Debug Port, AND
- the Debug Port was in use.

This translates to the following positive condition: reset the controller
only if:
- no USB port on the host controller is a Debug Port, OR
- the Debug Port is not in use.

Commit 09943f5ecc failed to implement the first subcondition, and thus
the result is too strict now (for resetting the host controller). Relax it
to the correct condition.

This issue was found by Steven Shi on QEMU. QEMU's EHCI device model does
not have a Debug Port. In repeated disconnect / connect cycles, the
controller is never reset in EhcDriverBindingStart(), therefore the
devices on the controller are never re-enumerated after a disconnect.

Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: Jian J Wang <jian.j.wang@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Star Zeng <star.zeng@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Reported-by: Steven Shi <steven.shi@intel.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1129
Fixes: 09943f5ecc
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
Tested-by: Steven Shi <steven.shi@intel.com>
2018-09-05 14:00:12 +02:00
8a204b2aca MdeModulePkg PeiCore: Fix VS2012 build failure
fwvol.c(1572) : warning C4701: potentially uninitialized
local variable 'Status' used

The build failure is caused by
0e042d0ad7 for
https://bugzilla.tianocore.org/show_bug.cgi?id=1131

This patch initializes Status to fix the build failure.

Cc: Dandan Bi <dandan.bi@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Jian J Wang <jian.j.wang@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Dandan Bi <dandan.bi@intel.com>
2018-09-05 13:17:10 +08:00
865d7f7b01 BaseTools: Extend the keyword "!include"/"!if" to case-insensitive
Extend the keyword "!include", "!if", etc to case-insensitive.
Current DSC parser already support it, while FDF parser only support
the lower case, so this patch add the support for FDF parser.

Fixes: https://bugzilla.tianocore.org/show_bug.cgi?id=1111
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-09-05 09:04:13 +08:00
04722cfa30 MdeModulePkg/Setup: Fix incorrect size used in AllocateCopyPool
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=1115

When the type of HiiValue is EFI_IFR_TYPE_BUFFER,
its question type is EFI_IFR_ORDERED_LIST_OP.
And the buffer size allocated for Statement->BufferValue
of orderedList is "Statement->StorageWidth"
in IfrParse.c.

So here when backup the buffer value and copy the size of
"Statement->StorageWidth + sizeof(CHAR16)" is incorrect.

This patch is to fix this issue.

Cc: Eric Dong <eric.dong@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Dandan Bi <dandan.bi@intel.com>
Reviewed-by: Eric Dong <eric.dong@intel.com>
2018-09-04 09:14:46 +08:00
e3b9ab433a MdeModulePkg DxeCore: Handle multiple FV images in one FV file
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1131

PI spec and BaseTools support to generate multiple FV images
in one FV file.
This patch is to update DxeCore to handle the case.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-09-03 17:04:58 +08:00
0e042d0ad7 MdeModulePkg PeiCore: Handle multiple FV images in one FV file
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1131

PI spec and BaseTools support to generate multiple FV images
in one FV file.
This patch is to update PeiCore to handle the case.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-09-03 17:04:57 +08:00
374168ae65 UefiCpuPkg/CpuExceptionHandlerLib: Avoid calling PEI services from AP
When an exception happens in AP, system hangs at
GetPeiServicesTablePointer(), complaining the PeiServices retrieved
from memory before IDT is NULL.

Due to the following commit:
c563077a38
* UefiCpuPkg/MpInitLib: Avoid calling PEI services from AP
the IDT used by AP no longer preserve PeiServices pointer in the
very beginning.
But the implementation of PeiExceptionHandlerLib still assumes
the PeiServices pointer is there, so the assertion happens.

The patch fixes the exception handler library to not call
PEI services from AP.

The patch duplicates the #0 exception stub header in an allocated
pool but with extra 4-byte/8-byte to store the exception handler
data which was originally stored in HOB.
When AP exception happens, the code gets the exception handler data
from the exception handler for #0.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Jian J Wang <jian.j.wang@intel.com>
Reviewed-by: Fan Jeff <vanjeff_919@hotmail.com>
2018-09-03 14:02:26 +08:00
87a9dd0d15 CpuExceptionHandlerLib: Add comments to make code more readable
Today's implementation of handling HOOK_BEFORE and HOOK_AFTER is
a bit complex. More comments is better.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Fan Jeff <vanjeff_919@hotmail.com>
Reviewed-by: Jian J Wang <jian.j.wang@intel.com>
2018-09-03 14:02:24 +08:00
4cf022f2f1 BaseTools: Fixed the PcdValue trailing zero issue.
1. Not append trailing zero for PcdValue
2. make sure the point to Variable Name in PCD
DataBase 2 bytes aligned.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-09-03 10:14:50 +08:00
7c19378762 BaseTools: Dynamic Pcd value override from command line.
Fixed the pcd value override issue when Dynamic Pcd is from
command line but is not list in Dsc file.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-09-03 10:14:47 +08:00
24bd035c90 BaseTools: Check pcd DefaultValue and SkuId EBNF.
1. When assign dynamic hii pcd value in dsc file,
missed the DefaultValue, build should be fail.
2. Check the EBNF of SkuId.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: ZhiqiangX Zhao <zhiqiangx.zhao@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-09-03 10:14:44 +08:00
b8bcaafb39 ShellPkg: Update Ifconfig command to accept 32bit subnet mask.
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Fu Siyuan <siyuan.fu@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Ye Ting <ting.ye@intel.com>
Reviewed-by: Wu Jiaxin <jiaxin.wu@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2018-09-03 08:42:59 +08:00
12ae56cf28 MdeModulePkg/Network: Add 32bit subnet mask support for IP4 PXE boot.
This patch updates IP4 stack to support 32bit subnet mask in PXE boot process.
When 32bit subnet mask is used, the IP4 driver couldn't use the subnet mask to determine
whether destination IP address is on-link or not, so it will always try to send all the
packets to the destination IP address directly first, if failed it will continue
to try the default gateway.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Fu Siyuan <siyuan.fu@intel.com>
Reviewed-by: Ye Ting <ting.ye@intel.com>
Reviewed-by: Wu Jiaxin <jiaxin.wu@intel.com>
2018-09-03 08:36:55 +08:00
1e57188216 EmulatorPkg: IoThunk->Close() is called too early, may causing hang
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1121

To produce a EFI_BLOCK_IO_PROTOCOL instance in Emulator platform,
EmulatorPkg defines the EMU_IO_THUNK_PROTOCOL. OS dependent layer
needs to produce this protocol implementation and a generic OS
independent layer consumes this protocol to produce
EFI_BLOCK_IO_PROTOCOL.

EMU_IO_THUNK_PROTOCOL can also be used to abstract the OS dependent
IO operation for other UEFI protocols, e.g.: GOP, SimpleFileSystem
and etc.

It contains two interfaces Open() and Close(). Open() creates the
specific IO instances, e.g. for Block IO access, File System access,
Screen access, etc. Close() destroys the specific IO instances.

Later on the Emulator generic module (e.g.: EmuBlockIoDxe) calls
Open() to create the IO instance in DriverBindingStart() and calls
Close() in DriverBindingStop().
But today's implementation of DriverBindingStop() contains a bug
that it calls Close() before uninstalling the EFI_BLOCK_IO_PROTOCOL.

It's a mistake in code. Take EFI_BLOCK_IO for example,
the uninstallation may cause the upper layer driver that consumes
EFI_BLOCK_IO call BlockIo.Reset(), which consequently calls
EmuBlockIo.Reset(). But the EmuBlockIo instance is already destroyed
by Close() that happens before uninstallation.

So a proper implementation is to call Close() after uninstallation
succeeds.

REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1121

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Andrew Fish <afish@apple.com>
Reviewed-by: Hao Wu <hao.a.wu@intel.com>
2018-08-31 10:40:20 +08:00
e01238946c EmulatorPkg: Use MdeModulePkg/Bds module
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1120

Logo is enabled by adding a separate core driver LogoDxe.
UiApp and BootManagerMenuApp are added to provide two UIs.

LoadFileOnFv2 is added to auto-install LoadFile protocol for
applications in FV so the boot options for applications can be
auto-created from LoadFile.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Hao Wu <hao.a.wu@intel.com>
Cc: Andrew Fish <afish@apple.com>
2018-08-31 10:40:18 +08:00
0195dd509f EmulatorPkg/Win: Add VS2017 project file
Developer can build the Win Host in VS2017 and launch to debug it.
Platform 'x64' is to build 64bit EmulatorWin.
Platform 'Win32' is to build 32bit EmulatorWin.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Hao Wu <hao.a.wu@intel.com>
Cc: Andrew Fish <afish@apple.com>
2018-08-31 10:40:16 +08:00
54ee6176ed EmulatorPkg/AutoScanPei: Report the correct CPU address size
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1119

Today's implementation reports CPU address size as 36 through CPU
HOB. But when WinHost is running at 64bit, the system memory might
be allocated above 2^36.

It causes system asserts when DxeCore code tries to find the
corresponding GCD entry for a given valid address.

The patch uses 57 as the CPU address size which is maximum linear
address size when 5-level paging is enabled in host OS.

Using 64 seems more proper and a one-time change even 6-level
paging might be invented. But it causes CoreInitializeGcdServices()
assertion on following code:
  Entry->EndAddress = LShiftU64 (1, SizeOfMemorySpace) - 1;
Because LShiftU64 expects SizeOfMemorySpace < 64.

So to be practical, I didn't report 64 and change
CoreInitializeGcdServices().

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Hao Wu <hao.a.wu@intel.com>
Cc: Andrew Fish <afish@apple.com>
2018-08-31 10:40:15 +08:00
f89c018f3d EmulatorPkg/Win: Enable 64bit (SEC,PEI,DXE all run at 64bit)
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Hao Wu <hao.a.wu@intel.com>
Cc: Andrew Fish <afish@apple.com>
2018-08-31 10:40:13 +08:00
e3c96c391c EmulatorPkg/Win: ReadKeyStrokeEx() always returns correct KeyState
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1118

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Hao Wu <hao.a.wu@intel.com>
Cc: Andrew Fish <afish@apple.com>
2018-08-31 10:40:11 +08:00
d773459e2f EmulatorPkg/Win: Use FrameBufferBltLib for BLT operation
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Hao Wu <hao.a.wu@intel.com>
Cc: Andrew Fish <afish@apple.com>
2018-08-31 10:40:10 +08:00
ce8c1d92f5 EmulatorPkg/EmuGopDxe: Clear screen to black in GOP.SetMode
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1118

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Hao Wu <hao.a.wu@intel.com>
Cc: Andrew Fish <afish@apple.com>
2018-08-31 10:40:08 +08:00
158b558711 EmulatorPkg/EmuGopDxe: Fix TxtInEx.SetState SCT conformance failure
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1118

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Hao Wu <hao.a.wu@intel.com>
Cc: Andrew Fish <afish@apple.com>
2018-08-31 10:40:07 +08:00
209d095968 BaseTools: Clarify a DSC parsing error about PCDs
This error needs the information about which DEC files were searched.

Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Lee Hamel <lee.m.hamel@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-08-31 08:53:28 +08:00
a77e5bcac5 BaseTools: minimize assignment processing
Reverse the checking and only assign once to each variable.

Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-08-30 21:08:49 +08:00
f5f4667dae BaseTools: Refactor to remove functionally equivalent functions
IsSupportedArch and IsBinaryModule return the same value under the same
curcimstances.  Remove newer one with fewer callers and send them to the
other function.

Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-08-30 21:05:18 +08:00
0fece18d6d BaseTools: include variable namespace GUIDs of HII PCDs in Guid.xref
[PcdsDynamicHii]
gFooTokenSpaceGuid.PcdBar|L"Variable"|gVarNameSpaceGuid|0x0|FALSE|NV,BS

This patch add the variable namespace GUIDs in "Guid.xref" that are
used with dynamic HII PCDs.

Fixes: https://bugzilla.tianocore.org/show_bug.cgi?id=452
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-08-30 20:59:18 +08:00
36faa23c46 MdeModulePkg EhciPei: Remove a redundant function
The function UsbHcUnlinkMemBlock that is never called
and its related comments have been removed.
It is missed in the patch series according to the log in
https://bugzilla.tianocore.org/show_bug.cgi?id=1062

v2:Update the title.

Cc: Star Zeng <star.zeng@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-08-30 14:44:03 +08:00
7475ac5157 ShellPkg/SmbiosView: Update SmbiosView for SMBIOS3.2.0
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1099

Update SmbiosView to parse the new definitions which
are introduced in SMBIOS3.2.0

V2:
1. Add structure length check before dump the fileds in
Type 9 and Type 17 in case some fileds are not organized
and reported by drivers.
2. Dump the InterfaceTypeSpecificData in Type 42.

V3:
1. Correct the structure length in Type17.
2. Remove the redundant check "if (PeerGroupCount > 0)" in Type 9.
3. Use the Uint16 filed instead of Bits field in union
MEMORY_DEVICE_OPERATING_MODE_CAPABILITY to dump data.

Cc: Jaben Carsey <jaben.carsey@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Star Zeng <star.zeng@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Dandan Bi <dandan.bi@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-08-30 10:06:10 +08:00
79e4f2a56a EmulatorPkg: formalize line endings
The patch is the result of running
"BaseTools/Scripts/FormatDosFiles.py EmulatorPkg/"

No functionality impact.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Hao A Wu <hao.a.wu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
2018-08-30 09:26:54 +08:00
a07533fab1 Maintainers.txt: Update maintainer of MdeModulePkg
Add Jian J Wang <jian.j.wang@intel.com> and
remove Eric Dong <eric.dong@intel.com>.
Eric is focusing on UefiCpuPkg.

Cc: Jian J Wang <jian.j.wang@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Andrew Fish <afish@apple.com>
Cc: Leif Lindholm <leif.lindholm@linaro.org>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Jian J Wang <jian.j.wang@intel.com>
Reviewed-by: Eric Dong <eric.dong@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
2018-08-30 08:59:10 +08:00
94c0455937 BaseTools: Create and use a shared value for 'MSFT' from DataType
I see lots of 'MSFT' throughout code and this can reduce them.

Cc: Bob Feng <Bob.c.Feng@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-08-30 08:42:09 +08:00
7c7c8190d3 UefiCpuPkg/CpuDxe: change level of DEBUG message
BZ#1127: https://bugzilla.tianocore.org/show_bug.cgi?id=1127

It's reported the debug message in CpuDxe driver is quite annoying in
boot and shell, and slow down the boot process. To solve this issue,
this patch changes the DEBUG_INFO to DEBUG_VERBOSE. On a typical Intel
real platform, at least 16s boot time can be saved.

Cc: Eric Dong <eric.dong@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jian J Wang <jian.j.wang@intel.com>
Reviewed-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
2018-08-30 07:43:56 +08:00
09afd9a42a UefiCpuPkg/PiSmmCpuDxeSmm: implement non-stop mode for SMM
Since SMM profile feature has already implemented non-stop mode if #PF
occurred, this patch just makes use of the existing implementation to
accommodate heap guard and NULL pointer detection feature.

Cc: Eric Dong <eric.dong@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jian J Wang <jian.j.wang@intel.com>
Reviewed-by: Eric Dong <eric.dong@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
2018-08-30 07:22:30 +08:00
dcc026217f UefiCpuPkg/CpuDxe: implement non-stop mode for uefi
Same as SMM profile feature, a special #PF is used to set page attribute
to 'present' and a special #DB handler to reset it back to 'not-present',
right after the instruction causing #PF got executed.

Since the new #PF handler won't enter into dead-loop, the instruction
which caused the #PF will get chance to re-execute with accessible pages.

The exception message will still be printed out on debug console so that
the developer/QA can find that there's potential heap overflow or null
pointer access occurred.

Cc: Eric Dong <eric.dong@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jian J Wang <jian.j.wang@intel.com>
Reviewed-by: Eric Dong <eric.dong@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
2018-08-30 07:22:30 +08:00
16b918bbaf UefiCpuPkg/CpuExceptionHandlerLib: Setup single step in #PF handler
Once the #PF handler has set the page to be 'present', there should
be a way to reset it to 'not-present'. 'TF' bit in EFLAGS can be used
for this purpose. 'TF' bit will be set in interrupted function context
so that it can be triggered once the cpu control returns back to the
instruction causing #PF and re-execute it.

This is an necessary step to implement non-stop mode for Heap Guard
and NULL Pointer Detection feature.

Cc: Eric Dong <eric.dong@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jian J Wang <jian.j.wang@intel.com>
Reviewed-by: Eric Dong <eric.dong@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
2018-08-30 07:22:29 +08:00
8f2613628a MdeModulePkg/MdeModulePkg.dec: add new settings for PCDs
The change is part of requirement tracked by #BZ1095

    https://bugzilla.tianocore.org/show_bug.cgi?id=1095

Background:
Heap Guard and NULL Pointer Detection are very useful features to detect
code flaw in EDK II. If an issue is detected, #PF exception will be
triggered and the BIOS will enter into dead loop, which is the default
behavior of exception handling. From QA perspective, this default behavior
will block them to collect all tests result in reasonable time.

The solution is to introduce non-stop mode to Heap Guard and NULL Pointer
Detection features. This needs to update CpuDxe, PiSmmCpuDxeSmm and
CpuExceptionHandlerLib to allow the code to continue execution after #PF.
The mechanism behind it is the same as SMM Profile feature, in which a
special #PF handler is registered to set the page causing #PF to be
'present' and setup single steop trap, then return the control back to
the instruction accessing that page. Once the instruction is re-executed,
a #DB is triggered and a special handler for it will be called to reset
the page back to 'not-present'.

The non-stop mode is controlled by BIT6 of following PCDs

  gEfiMdeModulePkgTokenSpaceGuid.PcdHeapGuardPropertyMask
  gEfiMdeModulePkgTokenSpaceGuid.PcdNullPointerDetectionPropertyMask

BIT6 of PcdHeapGuardPropertyMask is used to enable/disable non-stop mode
of Heap Guard feature. It applies to both UEFI and SMM heap guard, if
any of them is enabled.

BIT6 of PcdNullPointerDetectionPropertyMask is used to enable/disable
non-stop mode of NULL Pointer Detection feature. It applies to both
UEFI and SMM NULL Pointer Detection, if any of them is enabled.

The default setting is 'disable', meaning the boot will stop at #PF
exception.

Cc: Eric Dong <eric.dong@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jian J Wang <jian.j.wang@intel.com>
Reviewed-by: Eric Dong <eric.dong@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
2018-08-30 07:22:29 +08:00
497a5fb1d8 MdePkg: Add the missing spec version information for header files
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-08-29 14:00:32 +08:00
0fa0e56ee0 BaseTools: AutoGen.py remove unused import
AutoGen does not use anything defined in BuildClassObject

Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-08-29 08:45:29 +08:00
f25cd80e4d BaseTools: Fix one expression bug to support ~ operate
current use (0x41>=~0x0&0x41|0x0) as Pcd value cause build failure
because the ~ is not correctly recognized.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-08-28 14:33:47 +08:00
f2cc33d849 BaseTools: Add check only VOID* type Pcd need the maxsize info
Add check for the datum type keyword "VOID*", only the VOID* type
Pcd need the additional maxsize info.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-08-28 14:00:06 +08:00
f965b772fc MdeModulePkg/PiSmmCore: Check valid memory range.
Call BS.AllocatePages in DXE driver and call SMM FreePages with the address of the buffer allocated in the DXE driver. SMM FreePages success and add a non-SMRAM range into SMM heap list. This is not an expected behavior. SMM FreePages should return error for this case and not free the pages.

BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1098

Change-Id: Ie5ffa1ac62c558aa418a8a3d7d0e8158b846e13b
Cc: Star Zeng <star.zeng@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-08-28 10:24:25 +08:00
17da1b9108 MdeModulePkg: Update SMBIOS PCDs to 3.2.0
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1099

Cc: Liming Gao <liming.gao@intel.com>
Cc: Dandan Bi <dandan.bi@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Dandan Bi <dandan.bi@intel.com>
2018-08-28 10:13:33 +08:00
cfcca3c2de MdePkg SmBios.h: Add SMBIOS 3.2.0 definitions
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1099

Add SMBIOS 3.2.0 definitions according to
www.dmtf.org/sites/default/files/standards/documents/DSP0134_3.2.0.pdf.

Processor Information (Type 4):
- SMBIOSCR00163: add socket LGA2066
- SMBIOSCR00173: add Intel Core i9
- SMBIOSCR00176: add new processor sockets
Port Connector Information (Type 8):
- SMBIOSCR00168: add USB Type-C
System Slots (Type 9):
- SMBIOSCR00164: add "unavailable" to current usage field
- SMBIOSCR00167: add support for PCIe bifurcation
Memory Device (Type 17):
- SMBIOSCR00162: add support for NVDIMMs
- SMBIOSCR00166: extend support for NVDIMMs and add support for logical memory type
- SMBIOSCR00172: rename "Configured Memory Clock Speed" to "Configured Memory Speed"
- SMBIOSCR00174: add new memory technology value (Intel Persistent Memory, 3D XPoint)
IPMI Device Information (Type 38):
- SMBIOSCR00171: add SSIF
Management Controller Host Interface (Type 42)
- SMBIOSCR00175: fix structure data parsing issue

V2: Add missing update to MISC_PORT_TYPE and SMBIOS_TABLE_TYPE9.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Dandan Bi <dandan.bi@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Dandan Bi <dandan.bi@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-08-28 10:13:32 +08:00
033949a810 EmulatorPkg/DSC: Remove FS mapping to EDK Shell bin directory
The EDK Shell is end of line so the FS mapping to old EDK
Shell bin directory is better to be removed.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Hao Wu <hao.a.wu@intel.com>
Cc: Andrew Fish <afish@apple.com>
2018-08-27 15:21:03 +08:00
3da7b0639f EmulatorPkg/EmuFileSystem: Fix a bug that causes Close() assertion
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1117

The root cause is when a file is opened through File.Open(), the
private data for the File is not allocated, so when later
when File.Close() is called, the signature check in CR() causes
the assertion.

The private data for the File is allocated properly when the file
is opened from FS.OpenVolume().

The patch also fixes a minor issue that wrongly assigns
revision number to File.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Hao Wu <hao.a.wu@intel.com>
Cc: Andrew Fish <afish@apple.com>
2018-08-27 15:21:02 +08:00
41fd56be34 EmulatorPkg/PlatformBds: Signal EndOfDxe in platform BDS
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1079

Without signal of EndOfDxe, the 3rd party code (.efi from non-flash
storage) cannot run. It's forbidden by
8be37a5cee
*MdeModulePkg/SecurityStubDxe: Defer 3rd party image before EndOfDxe

The patch enables running of SCT from internal shell.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Hao A Wu <hao.a.wu@intel.com>
Cc: Andrew Fish <afish@apple.com>
2018-08-27 15:21:01 +08:00
8f8196976e EmulatorPkg/Win: Add BlockIo support
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Hao A Wu <hao.a.wu@intel.com>
Cc: Andrew Fish <afish@apple.com>
2018-08-27 15:20:59 +08:00
56502bf1ad EmulatorPkg/Win: Add SimpleFileSystem support
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Hao A Wu <hao.a.wu@intel.com>
Cc: Andrew Fish <afish@apple.com>
2018-08-27 15:20:58 +08:00
07d8559c6b EmulatorPkg/Win: Add RTC support
Now firmware can display the time correctly.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Hao A Wu <hao.a.wu@intel.com>
Cc: Andrew Fish <afish@apple.com>
2018-08-27 15:20:56 +08:00
269c512393 EmulatorPkg/Win: Add timer and interrupt support
Now the firmware can boot to Shell and count down 5 seconds to
command prompt.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Hao A Wu <hao.a.wu@intel.com>
Cc: Andrew Fish <afish@apple.com>
2018-08-27 15:20:55 +08:00
7a465451be EmulatorPkg/Win: Add input/output support
The patch adds GOP and SimpleTextIn[Ex] support.
Now firmware can boot to UI and Shell but timer doesn't work.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Andrew Fish <afish@apple.com>
Reviewed-by: Hao A Wu <hao.a.wu@intel.com>
2018-08-27 15:20:54 +08:00
5601e90d5c EmulatorPkg/Win: Enable native OS console as firmware console
The patch implements Stdin/Stdout/Stderr access so that
the native OS console (cmd.exe) can be used as the firmware
console and debug message can also be routed to it.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Hao A Wu <hao.a.wu@intel.com>
Cc: Andrew Fish <afish@apple.com>
2018-08-27 15:20:52 +08:00
0d9b5c461a EmulatorPkg/Win: Enable source level debugging
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Andrew Fish <afish@apple.com>
Reviewed-by: Hao A Wu <hao.a.wu@intel.com>
2018-08-27 15:20:51 +08:00
3c859dfebb EmulatorPkg/Win: Add Windows host support
This is the initial patch to make it boot to early DXE
phase without display (only debug message).

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Hao A Wu <hao.a.wu@intel.com>
Cc: Andrew Fish <afish@apple.com>
2018-08-27 15:20:49 +08:00
47c04c382e EmulatorPkg/ThunkProtocolList: Fix VS build failure
VS compiler complains converting UINTN to UINT16 causes data lost.
Add typecast to fix the build failure.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Andrdw Fish <afish@apple.com>
Reviewed-by: Hao A Wu <hao.a.wu@intel.com>
2018-08-27 15:20:47 +08:00
983f5abb9a MdeModulePkg/PciBus: Restrict one VGA per HostBridge not Segment
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1109

Today's restriction of VGA device is to have only one VGA device
enabled per PCI segment. It's not correct because several segments
may share one IO / MMIO address space.
We should restrict to have one VGA per Host Bridge because each
Host Bridge has its only IO / MMIO address space.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-08-27 09:40:22 +08:00
06da1e310b MdeModulePkg/PciBus: Refine ActiveVGADeviceOnTheRootBridge
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1109
The patch doesn't change any behavior of this function.
It just renames the function to LocateVgaDevice() and renames
some parameters and local variables.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-08-27 09:40:21 +08:00
6861765935 EmbeddedPkg/AndroidFastbootApp: only use ENTER or SPACE to exit
Since hotkey 'f' is used to start AndroidFastbootApp. If user
press 'f' key too long, it may be recognized pressing 'f' key
multiple times. Then AndroidFastbootApp exists since it delcares
any key press could make it exit.

So only use ENTER or SPACE key to exit AndroidFastbootApp.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Haojian Zhuang <haojian.zhuang@linaro.org>
Reviewed-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
2018-08-24 15:44:18 +01:00
16973234fc IntelFrameworkModulePkg BdsDxe: Remove redundant functions
The redundant functions which are never called have been
removed. They are GetProducerString,ChangeVariableDevicePath,
EfiReallocatePool,Var_UpdateAllConsoleOption and BOpt_IsEfiApp.
https://bugzilla.tianocore.org/show_bug.cgi?id=1063

Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-08-23 22:46:18 -07:00
ebad1c5e0a IntelFrameworkModulePkg LegacyBiosDxe: Remove redundant LibraryClasses
The redundant LibraryClasses which are PeCoffLib and CacheMaintenanceLib
have been removed.
https://bugzilla.tianocore.org/show_bug.cgi?id=1063

Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-08-23 22:46:17 -07:00
0ef7320e27 IntelFrameworkModulePkg Snp16Dxe: Remove redundant functions
The redundant functions which are never called have been removed.
They are PxeUndiSetMcastAddr, PxeUndiInitiateDiags,PxeUndiGetState
PxeUndiForceInterrupt and PxeUndiSetPacketFilter.
https://bugzilla.tianocore.org/show_bug.cgi?id=1063

Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-08-23 22:46:16 -07:00
ef3b352c73 IntelFrameworkModulePkg IdeBusDxe: Remove redundant functions
The redundant functions which are never called have been removed.
They are AtaNonDataCommandInExt,IDEBusDriverConfigurationSetOptions,
GetResponse,IDEBusDriverConfigurationOptionsValid and
IDEBusDriverConfigurationForceDefaults.
https://bugzilla.tianocore.org/show_bug.cgi?id=1063

Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-08-23 22:46:16 -07:00
4562975865 IntelFrameworkModulePkg Ps2KeyboardDxe: Remove a redundant function
The redundant function DisableKeyboard which is never called
has been removed.
https://bugzilla.tianocore.org/show_bug.cgi?id=1063

Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-08-23 22:46:15 -07:00
1f3da070ac IntelFrameworkModulePkg IsaIoDxe: Remove a redundant protocol
Remove a redundant protocol which is gEfiGenericMemTestProtocolGuid.
https://bugzilla.tianocore.org/show_bug.cgi?id=1063

Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-08-23 22:46:15 -07:00
2b2959dd34 ArmPkg/PlatformBootManagerLib: fix hotkey of boot option
Fix the parameter in EfiBootManagerAddKeyOptionVariable (). Otherwise,
the hotkey of boot option won't be registered correctly.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Haojian Zhuang <haojian.zhuang@linaro.org>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
2018-08-23 16:41:47 +01:00
b701aae6a8 NetworkPkg: UefiPxeBcDxe: Remove the redundant code.
The function PxeBcFreeBootFileOption that is never called
have been removed.

Cc: Jiaxin Wu <jiaxin.wu@intel.com>
Cc: Siyuan Fu <siyuan.fu@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1064
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Songpeng Li <songpeng.li@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Jiaxin Wu <jiaxin.wu@intel.com>
2018-08-23 16:30:47 +08:00
8343c75024 NetworkPkg: TcpDxe: Remove the redundant code.
The function TcpPawsOK that is never called have been removed.

Cc: Jiaxin Wu <jiaxin.wu@intel.com>
Cc: Siyuan Fu <siyuan.fu@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1064
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Songpeng Li <songpeng.li@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Jiaxin Wu <jiaxin.wu@intel.com>
2018-08-23 16:30:46 +08:00
ad1e45ddb5 NetworkPkg: IpSecDxe: Remove the redundant code.
The functions that are never called have been removed.
They are Ikev2OnPacketAccepted, Ikev2SaSessionOnDeleting,
Ikev2ChildSaSessionLookupByMid, Ikev2ChildSaSessionOnDeleting
and Ikev2ChildExchangeRekeySpi.

Cc: Jiaxin Wu <jiaxin.wu@intel.com>
Cc: Siyuan Fu <siyuan.fu@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1064
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Songpeng Li <songpeng.li@intel.com>
Reviewed-by: Jiaxin Wu <jiaxin.wu@intel.com>
2018-08-23 16:30:46 +08:00
fafdb489a7 NetworkPkg: IScsiDxe: Remove the redundant code and definition.
The function IScsiFindTcbByITT that is never called have been removed.
The definition gEfiAcpi20TableGuid has been removed.

Cc: Jiaxin Wu <jiaxin.wu@intel.com>
Cc: Siyuan Fu <siyuan.fu@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1064
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Songpeng Li <songpeng.li@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Jiaxin Wu <jiaxin.wu@intel.com>
2018-08-23 16:30:45 +08:00
fcb1af1b69 BaseTools: Use hashlib instead of md5
Use from hashlib import md5 instead of import md5
due to md5 deprecated in python3

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-08-23 15:01:27 +08:00
87010d3d02 BaseTools: remove cmp due to deprecated in python3
remove cmp due to deprecated in python3

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-08-23 15:01:27 +08:00
abb8e6e97a BaseTools: Modify class OrderedListDict
class OrderedListDict(OrderedDict, defaultdict) will
encounter multiple bases have instance lay-out
conflict error on python3

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-08-23 15:01:27 +08:00
c1260801fd BaseTools: Update Makefile for ECC tool
V2: Add --target-name to specify the file name to create

Because Ecc.py was renamed to EccMain.py

Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-08-23 14:54:15 +08:00
aa4e0df1f0 BaseTools/VfrCompile: honor EXTRA_LDFLAGS
In commit 81502cee20 ("BaseTools/Source/C: take EXTRA_LDFLAGS from the
caller", 2018-08-16), I missed that "VfrCompile/GNUmakefile" does not use
BUILD_LFLAGS in the APPLICATION linking rule, unlike "app.makefile" does.
Instead, "VfrCompile/GNUmakefile" uses the (undefined) LFLAGS macro.
Therefore commit 81502cee20 did not cover the linking step of
VfrCompile.

Thankfully, the structure of the linking rules is the same, between
"app.makefile" and "VfrCompile/GNUmakefile". Rename the undefined LFLAGS
macro in "VfrCompile/GNUmakefile" to VFR_LFLAGS (for consistency with
VFR_CXXFLAGS), and set it to EXTRA_LDFLAGS.

As a result, we have:

             | compilation                    | linking
  -----------+--------------------------------+----------------------
  VfrCompile | VFR_CXXFLAGS =                 | VFR_LFLAGS =
             | BUILD_OPTFLAGS =               | EXTRA_LDFLAGS
             | '-O2' + EXTRA_OPTFLAGS         |
  -----------+--------------------------------+----------------------
  other apps | BUILD_CFLAGS/BUILD_CXXFLAGS =  | BUILD_LFLAGS =
             | [...] + BUILD_OPTFLAGS =       | [...] + EXTRA_LDFLAGS
             | [...] + '-O2' + EXTRA_OPTFLAGS |

This table shows
- that the VfrCompile compilation and linking flags are always a subset of
  the corresponding flags used by the other apps,
- and that the EXTRA flags are always at the end.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Ref: https://bugzilla.redhat.com/show_bug.cgi?id=1540244
Fixes: 81502cee20
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-08-22 09:32:46 -07:00
0442a9a9f4 UefiCpuPkg/PiSmmCpuDxeSmm: clear exec file mode bits on "PiSmmCpuDxeSmm.c"
Commit 241f914975 ("UefiCpuPkg/PiSmmCpuDxeSmm: Add support for PCD
PcdPteMemoryEncryptionAddressOrMask", 2017-03-01) unintentionally set the
executable file mode bits on "PiSmmCpuDxeSmm.c"; clear them now.

Cc: Eric Dong <eric.dong@intel.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1092
Fixes: 241f914975
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-08-22 10:33:49 +02:00
6896efdec2 NetworkPkg/TlsAuthConfigDxe: fix TlsCaCertificate attributes retrieval
Per spec, the GetVariable() runtime service is not required to populate
(*Attributes) on output when it fails with EFI_BUFFER_TOO_SMALL.

Therefore we have to fetch the full contents of the TlsCaCertificate
variable temporarily, just so we can (a) get the current attributes, and
(b) add EFI_VARIABLE_APPEND_WRITE to them for the subsequent SetVariable()
call.

Cc: Jiaxin Wu <jiaxin.wu@intel.com>
Cc: Siyuan Fu <siyuan.fu@intel.com>
Cc: Songpeng Li <songpeng.li@intel.com>
Reported-by: Songpeng Li <songpeng.li@intel.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1090
Fixes: b90c335fbb
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Songpeng Li <songpeng.li@intel.com>
Reviewed-by: Wu Jiaxin <jiaxin.wu@intel.com>
Reviewed-by: Fu Siyuan <siyuan.fu@intel.com>
2018-08-22 10:32:43 +02:00
d00759b212 IntelFrameworkPkg/FrameworkUefiLib: don't special-case EFI_FILE_MODE_CREATE
Synchronize EfiOpenFileByDevicePath() with the MdePkg/UefiLib instance, as
described in the previous patch.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Suggested-by: Ruiyu Ni <ruiyu.ni@intel.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1074
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-08-22 10:32:24 +02:00
5dbc768f1c MdePkg/UefiLib: don't special-case EFI_FILE_MODE_CREATE in OpenMode
While reviewing the patch that would land as 768b611136
("MdePkg/UefiLib: introduce EfiOpenFileByDevicePath()", 2018-08-16), Ray
pointed out that distinguishing EFI_FILE_MODE_CREATE was wasteful. Per
spec, if the file to create exists, then EFI_FILE_MODE_CREATE is ignored
by EFI_FILE_PROTOCOL.Open(), and the existent file is opened.

Therefore we don't need an attempt to "open-but-not-create" first, and a
fallback to "open-and-create-too" second -- that behavior is internal to
EFI_FILE_PROTOCOL.Open(). Remove the special-casing of
EFI_FILE_MODE_CREATE.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Suggested-by: Ruiyu Ni <ruiyu.ni@intel.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1074
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-08-22 10:31:47 +02:00
e0c93c9b76 MdeModulePkg/SmmCorePerfLib: Add DxeServicesLib library class in INF
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1100

SmmCorePerformanceLib consume the DxeServicesLib, but miss to
add the library class in INF file.
This patch is to fix this issue.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Star Zeng <star.zeng@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Dandan Bi <dandan.bi@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-08-22 10:29:28 +08:00
14a6beac3b IntelFsp2WrapperPkg: Add EFIAPI to AsmExecute32BitCode
AsmExecute32BitCode is assembly code and needs EFIAPI

Cc: Jiewen Yao <Jiewen.yao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Chasel Chiu <chasel.chiu@intel.com>
Reviewed-by: Jiewen Yao <Jiewen.yao@intel.com>
2018-08-22 09:41:25 +08:00
cc69390670 MdeModulePkg UiApp: Remove a redundant function and a variable
1.After the cleanup at "MdeModulePkg UiApp: Remove redundant functions",
  the function "IsResetReminderFeatureEnable()" and the variable
  "mFeaturerSwitch" become redundant so they have been removed.
2.Therefore, the "IF" expression has also been removed, whose judgment condition
  is "IsResetReminderFeatureEnable()".
https://bugzilla.tianocore.org/show_bug.cgi?id=1062

Cc: Star Zeng <star.zeng@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-08-21 16:29:11 +08:00
ae195de0a3 MdeModulePkg SetupBrowserDxe: Remove a redundant function
The function IsDevicePathExist that is never called
has been removed.
https://bugzilla.tianocore.org/show_bug.cgi?id=1062

Cc: Star Zeng <star.zeng@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
2018-08-21 16:29:11 +08:00
4750f754fc MdeModulePkg Tcp4Dxe: Remove redundant functions
The functions that are never called have been removed.
They are SockRcvdErr,SockGroup and TcpPawsOK.
https://bugzilla.tianocore.org/show_bug.cgi?id=1062

Cc: Star Zeng <star.zeng@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
2018-08-21 16:29:10 +08:00
d7bcd4c1b9 MdeModulePkg IScsiDxe: Remove a redundant function
The function IScsiFindTcbByITT that is never called
has been removed.
https://bugzilla.tianocore.org/show_bug.cgi?id=1062

Cc: Star Zeng <star.zeng@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
2018-08-21 16:29:09 +08:00
3028d73e5a MdeModulePkg Ip4Dxe: Remove a redundant function
The function Ip4Config2OnDhcp4Event that is never called
has been removed.
https://bugzilla.tianocore.org/show_bug.cgi?id=1062

Cc: Star Zeng <star.zeng@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
2018-08-21 16:29:08 +08:00
f787da363b MdeModulePkg HiiDatabaseDxe: Remove a redundant function
The function GetStorageWidth that is never called
has been removed.
https://bugzilla.tianocore.org/show_bug.cgi?id=1062

Cc: Star Zeng <star.zeng@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Eric Dong <eric.dong@intel.com>
2018-08-21 16:29:08 +08:00
a2a6f9f61d MdeModulePkg TerminalDxe: Remove a redundant function
The function UnicodeFiFoGetKeyCount that is never called
has been removed.
https://bugzilla.tianocore.org/show_bug.cgi?id=1062

Cc: Star Zeng <star.zeng@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
2018-08-21 16:29:07 +08:00
2394f31c55 MdeModulePkg SdBlockIoPei: Remove a redundant function
The function SdPeimUnlinkMemBlock that is never called
has been removed.
https://bugzilla.tianocore.org/show_bug.cgi?id=1062

v2: Remove the comment that mentioned 'SdPeimUnlinkMemBlock'
within function SdPeimFreeMemPool().

Cc: Star Zeng <star.zeng@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-08-21 16:29:06 +08:00
6e5676b380 MdeModulePkg EmmcBlockIoPei: Remove a redundant function
The function EmmcPeimUnlinkMemBlock that is never called
has been removed.
https://bugzilla.tianocore.org/show_bug.cgi?id=1062

v2: Remove the comment that mentioned 'EmmcPeimUnlinkMemBlock'
within function EmmcPeimFreeMemPool().

Cc: Star Zeng <star.zeng@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-08-21 16:29:06 +08:00
c575d091bb MdeModulePkg Ps2KeyboardDxe: Remove a redundant function
The function DisableKeyboard that is never called
has been removed.
https://bugzilla.tianocore.org/show_bug.cgi?id=1062

Cc: Star Zeng <star.zeng@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-08-21 16:29:05 +08:00
d637282efe MdeModulePkg PiSmmCore: Remove redundant functions
The functions that are never called have been removed.
They are IsImageInsideSmram,FindImageRecord,SmmRemoveImageRecord,
SmmMemoryAttributesTableConsistencyCheck,DumpSmmMemoryMapEntry,
SmmMemoryMapConsistencyCheckRange,SmmMemoryMapConsistencyCheck,
DumpSmmMemoryMap,ClearGuardMapBit,SetGuardMapBit,AdjustMemoryA,
AdjustMemoryS,IsHeadGuard and IsTailGuard.
FindImageRecord() is called by SmmRemoveImageRecord(); however,
nothing calls SmmRemoveImageRecord().
SmmMemoryMapConsistencyCheckRange() is called by
SmmMemoryMapConsistencyCheck(); however, nothing calls
SmmMemoryMapConsistencyCheck().
https://bugzilla.tianocore.org/show_bug.cgi?id=1062

v2:append the following to the commit message.
- FindImageRecord() is called by SmmRemoveImageRecord(); however,
nothing calls SmmRemoveImageRecord().
- SmmMemoryMapConsistencyCheckRange() is called by
SmmMemoryMapConsistencyCheck(); however, nothing calls
SmmMemoryMapConsistencyCheck().

Cc: Star Zeng <star.zeng@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Jian J Wang <jian.j.wang@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-08-21 16:29:04 +08:00
5bde6c2019 MdeModulePkg Core/Dxe: Remove redundant functions
The functions that are never called have been removed.
They are ClearGuardMapBit,SetGuardMapBit,IsHeadGuard,
IsTailGuard and CoreEfiNotAvailableYetArg0.
https://bugzilla.tianocore.org/show_bug.cgi?id=1062

Cc: Star Zeng <star.zeng@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Jian J Wang <jian.j.wang@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-08-21 16:29:04 +08:00
0ffa23c0f4 MdeModulePkg UsbBusPei: Remove redundant functions
The functions that are never called have been removed.
They are PeiHubSetHubFeature,IsPortConnectChange and
PeiUsbClearDeviceFeature.
https://bugzilla.tianocore.org/show_bug.cgi?id=1062

Cc: Star Zeng <star.zeng@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-08-21 16:29:03 +08:00
880cc3ebef MdeModulePkg UsbBusDxe: Remove redundant functions
The functions that are never called have been removed.
They are UsbHubCtrlSetHubFeature,UsbHubCtrlResetTT,UsbHcReset,
UsbHcAsyncIsochronousTransfer,UsbHcGetState,UsbHcSetState and
UsbHcIsochronousTransfer.
https://bugzilla.tianocore.org/show_bug.cgi?id=1062

Cc: Star Zeng <star.zeng@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-08-21 16:29:02 +08:00
2104e989cf MdeModulePkg UsbBotPei: Remove redundant functions
The functions that are never called have been removed.
They are PeiUsbSetConfiguration,PeiUsbSetDeviceAddress,
IsPortConnect,IsPortConnectChange,IsPortLowSpeedDeviceAttached
and PeiUsbGetDescriptor.
https://bugzilla.tianocore.org/show_bug.cgi?id=1062

Cc: Star Zeng <star.zeng@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-08-21 16:29:01 +08:00
78a6d34fa6 MdeModulePkg UfsPassThruDxe: Remove redundant functions
The functions that are never called have been removed.
They are UfsClearFlag and UfsFindAvailableSlotInTmrl.
https://bugzilla.tianocore.org/show_bug.cgi?id=1062

Cc: Star Zeng <star.zeng@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-08-21 16:29:01 +08:00
d270c1dea2 MdeModulePkg UfsBlockIoPei: Remove redundant functions
The functions that are never called have been removed.
They are UfsPeimUnlinkMemBlock,UfsPeimRequestSense,
UfsPeimInquiry,UfsRwAttributes,UfsClearFlag,UfsReadFlag
and UfsFindAvailableSlotInTmrl.
https://bugzilla.tianocore.org/show_bug.cgi?id=1062

v2: Remove the comment that mentioned 'UfsPeimUnlinkMemBlock'
within function UfsPeimFreeMemPool().

Cc: Star Zeng <star.zeng@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-08-21 16:29:00 +08:00
349b08d6ad MdeModulePkg XhciPei: Remove redundant functions
The functions that are never called have been removed.
They are XhcPeiReadDoorBellReg and UsbHcUnlinkMemBlock.
https://bugzilla.tianocore.org/show_bug.cgi?id=1062

Cc: Star Zeng <star.zeng@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-08-21 16:28:59 +08:00
2ed6c872f3 MdeModulePkg XhciDxe: Remove redundant functions
The functions that are never called have been removed.
They are XhcReadDoorBellReg and XhcWriteOpReg16.
https://bugzilla.tianocore.org/show_bug.cgi?id=1062

Cc: Star Zeng <star.zeng@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-08-21 16:28:59 +08:00
a829f08684 MdeModulePkg UhciPei: Remove redundant functions
The functions that are never called have been removed.
They are IoMmuFreeBuffer,DelinkMemoryBlock,IsMemoryBlockEmptied
,IsTDLinkPtrQHOrTD,GetCurrentFrameNumber,GetQHHorizontalLinkPtr
and GetQHHorizontalValidorInvalid.
https://bugzilla.tianocore.org/show_bug.cgi?id=1062

Cc: Star Zeng <star.zeng@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-08-21 16:28:58 +08:00
11237cf147 MdeModulePkg SdMmcPciHcDxe: Remove redundant functions
The functions that are never called have been removed.
They are SdCardGetCsd and SdCardGetScr.
https://bugzilla.tianocore.org/show_bug.cgi?id=1062

Cc: Star Zeng <star.zeng@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Hao Wu <hao.a.wu@intel.com>
2018-08-21 16:28:57 +08:00
66d3c76b8a MdeModulePkg PciBusDxe: Remove redundant functions
The functions that are never called have been removed.
They are UpStreamBridgesAttributes and GetHpcPciAddressFromRootBridge.
https://bugzilla.tianocore.org/show_bug.cgi?id=1062

Cc: Star Zeng <star.zeng@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-08-21 16:28:57 +08:00
442d4dde72 MdeModulePkg NvmExpressDxe: Remove redundant functions
The functions that are never called have been removed.
They are ReadNvmeAdminSubmissionQueueBaseAddress,
ReadNvmeAdminCompletionQueueBaseAddress and ReadNvmeAdminQueueAttributes
https://bugzilla.tianocore.org/show_bug.cgi?id=1062

Cc: Star Zeng <star.zeng@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Hao Wu <hao.a.wu@intel.com>
2018-08-21 16:28:56 +08:00
f210a43391 MdeModulePkg EhciDxe: Remove redundant functions
The functions that are never called have been removed.
They are EhcDumpRegs,EhcDisableAsyncSchd and EhcDisablePeriodSchd
https://bugzilla.tianocore.org/show_bug.cgi?id=1062

Cc: Star Zeng <star.zeng@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-08-21 16:28:55 +08:00
385fd1b846 MdeModulePkg AtaAtapiPassThru: Remove redundant functions
The functions that are never called have been removed.
They are AhciCheckDeviceStatus,AhciPortReset,DRDYReady,
DRDYReady2,WaitForBSYClear2 and AtaSoftReset.
https://bugzilla.tianocore.org/show_bug.cgi?id=1062

v2: DRDYReady, DRDYReady2, WaitForBSYClear2 and AtaSoftReset
are added to the commit message.

Cc: Star Zeng <star.zeng@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-08-21 16:28:54 +08:00
0629ae291a MdeModulePkg UiApp: Remove redundant functions
The functions that are never called have been removed.
They are EnableResetReminderFeature,
DisableResetReminderFeature and DisableResetRequired.
https://bugzilla.tianocore.org/show_bug.cgi?id=1062

Cc: Star Zeng <star.zeng@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Eric Dong <eric.dong@intel.com>
2018-08-21 16:28:54 +08:00
d82ebaa340 MdeModulePkg CapsuleApp: Remove a redundant function
The function DumpImageAuthentication that is never called
has been removed.
https://bugzilla.tianocore.org/show_bug.cgi?id=1062

Cc: Star Zeng <star.zeng@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-08-21 16:28:53 +08:00
0df5056012 UefiCpuPkg/SmmCpuFeaturesLib: [CVE-2017-5715] Stuff RSB before RSM
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1093

Return Stack Buffer (RSB) is used to predict the target of RET
instructions. When the RSB underflows, some processors may fall back to
using branch predictors. This might impact software using the retpoline
mitigation strategy on those processors.

This commit will add RSB stuffing logic before returning from SMM (the RSM
instruction) to avoid interfering with non-SMM usage of the retpoline
technique.

After the stuffing, RSB entries will contain a trap like:

@SpecTrap:
    pause
    lfence
    jmp     @SpecTrap

A more detailed explanation of the purpose of commit is under the
'Branch target injection mitigation' section of the below link:
https://software.intel.com/security-software-guidance/insights/host-firmware-speculative-execution-side-channel-mitigation

Please note that this commit requires further actions (BZ 1091) to remove
the duplicated 'StuffRsb.inc' files and merge them into one under a
UefiCpuPkg package-level directory (such as UefiCpuPkg/Include/).

REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1091

Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Eric Dong <eric.dong@intel.com>
2018-08-21 16:11:15 +08:00
02f7fd158e UefiCpuPkg/PiSmmCpuDxeSmm: [CVE-2017-5715] Stuff RSB before RSM
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1093

Return Stack Buffer (RSB) is used to predict the target of RET
instructions. When the RSB underflows, some processors may fall back to
using branch predictors. This might impact software using the retpoline
mitigation strategy on those processors.

This commit will add RSB stuffing logic before returning from SMM (the RSM
instruction) to avoid interfering with non-SMM usage of the retpoline
technique.

After the stuffing, RSB entries will contain a trap like:

@SpecTrap:
    pause
    lfence
    jmp     @SpecTrap

A more detailed explanation of the purpose of commit is under the
'Branch target injection mitigation' section of the below link:
https://software.intel.com/security-software-guidance/insights/host-firmware-speculative-execution-side-channel-mitigation

Please note that this commit requires further actions (BZ 1091) to remove
the duplicated 'StuffRsb.inc' files and merge them into one under a
UefiCpuPkg package-level directory (such as UefiCpuPkg/Include/).

REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1091

Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Hao Wu <hao.a.wu@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Regression-tested-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Eric Dong <eric.dong@intel.com>
2018-08-21 16:10:42 +08:00
6cf6fed024 BaseTools: Fix regression issue by b23414f654
V2: Renaming function DepexExpressionTokenList to  DepexExpressionDict
instead of changing the callers

Fix regression issue by b23414f654
AttributeError: 'ModuleAutoGen' object has no attribute
'DepexExpressionDict'

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-08-21 14:08:24 +08:00
131818ba5a MdeModulePkg: Remove redundant library classes and GUIDs
Some redundant library classes and GUIDs
have been removed in inf, .c and .h files.
https://bugzilla.tianocore.org/show_bug.cgi?id=1044
https://bugzilla.tianocore.org/show_bug.cgi?id=1045
https://bugzilla.tianocore.org/show_bug.cgi?id=1047
https://bugzilla.tianocore.org/show_bug.cgi?id=1049
https://bugzilla.tianocore.org/show_bug.cgi?id=1051
https://bugzilla.tianocore.org/show_bug.cgi?id=1052
https://bugzilla.tianocore.org/show_bug.cgi?id=1053
https://bugzilla.tianocore.org/show_bug.cgi?id=1054
https://bugzilla.tianocore.org/show_bug.cgi?id=1055
https://bugzilla.tianocore.org/show_bug.cgi?id=1056
https://bugzilla.tianocore.org/show_bug.cgi?id=1017
https://bugzilla.tianocore.org/show_bug.cgi?id=1035
https://bugzilla.tianocore.org/show_bug.cgi?id=1033
https://bugzilla.tianocore.org/show_bug.cgi?id=1012
https://bugzilla.tianocore.org/show_bug.cgi?id=1011
https://bugzilla.tianocore.org/show_bug.cgi?id=1062

v2:
1. Correct copyright year.
2. Fix alignment issue in AcpiPlatformDxe.inf.
3. Add DevicePathLib which is removed before in I2cHostDxe.inf.
4. Update NvmExpressPei.inf, NvmExpressPei.h, IScsiDxe.inf,
   IScsiDxe.inf, IScsiDxe.h, DxeMain.inf, DxeMain.h, PiSmmCore.inf
   and PiSmmCore.h for https://bugzilla.tianocore.org/show_bug.cgi?id=1062.

v3:
1. Add https://bugzilla.tianocore.org/show_bug.cgi?id=1062 to the
   link list.
2. Remove DevicePathLib from I2cHostDxe.inf.
3. Update NvmExpressPei.inf, NvmExpressPei.h, UfsPciHcDxe.inf,
   UfsPciHcDxe.h, I2cHostDxe.inf, AtaBusDxe.inf, IScsiDxe.inf,
   IScsiDxe.h, DxeMain.inf, DxeMain.h, PiSmmCore.inf and PiSmmCore.h
   for https://bugzilla.tianocore.org/show_bug.cgi?id=1062.
   (Forget to add UfsPciHcDxe.inf, UfsPciHcDxe.h, AtaBusDxe.inf,
   and I2cHostDxe.inf previously.)

Cc: Star Zeng <star.zeng@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-08-20 14:53:45 +08:00
004b811238 ShellPkg/edit: Fix heap access out-of-bounds
The issue was found when heap guard is turned on.
PrintLib somehow receives a non-null terminated string in var-arg.
When the PrintLib implementation reads the string it keeps
reading because no null-terminator is met, which triggers the page
fault set by the heap guard.

The issue is caused by a bug in FileBufferPrintLine().
When "edit" opens a binary file, in FileBufferPrintLine(),
the Line->Buffer may start with \x00 \x00, but the Line->Size is
larger than MainEditor.ScreenSize.Column, it causes the PrintLine is
set to an empty string by below call:
    StrnCpyS (
      PrintLine, BufLen/sizeof(CHAR16), Buffer,
      MIN(Limit, MainEditor.ScreenSize.Column)
      );
But since Limit (equals to Line->Size) is larger than
MainEditor.ScreenSize.Column, below for-loop doesn't successfully
set the whole PrintLine to all-empty-space.
    for (; Limit < MainEditor.ScreenSize.Column; Limit++) {
      PrintLine[Limit] = L' ';
    }
So after the for-loop, PrintLine is still an empty string.
Later in below call, the PrintLine2 is created based on PrintLine.
    ShellCopySearchAndReplace (
      PrintLine, PrintLine2,
      BufLen * 2, L"%", L"^%", FALSE, FALSE
      );
But due to the implementation of ShellCopySearchAndReplace(),
PrintLine2 is untouched and INVALID_PARAMETER is returned.
Finally an uninitialized string is passed to ShellPrintEx()
which causes the #PF exception.

The fix is to reset Limit to StrLen(PrintLine) before for-loop.
So that PrintLine can be converted from an empty string to a
string containing all spaces.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Jian Wang <jian.j.wang@intel.com>
2018-08-20 14:41:01 +08:00
90b3f171a7 SignedCapsulePkg: Remove redundant things in inf,.h and .c files
Two redundant library classes have been removed.
The redundant function "IsSystemFmp" has been removed.
https://bugzilla.tianocore.org/show_bug.cgi?id=1067

v2: Remove DxeServicesLib from SystemFirmwareUpdateDxe.inf.

Cc: Jiewen Yao <jiewen.yao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: shenglei <shenglei.zhang@intel.com>
Reviewed-by: Jiewen Yao <jiewen.yao@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-08-20 14:27:14 +08:00
43fe4c4052 BaseTools: remove unused setter functions
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-08-20 11:34:59 +08:00
fe6fdfe2bb BaseTools: remove unused code
the if statment just has pass statement.
invert if condition and just use do the else work.

Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-08-20 11:16:03 +08:00
0258ba6256 BaseTools: AutoGen refactor to iterate less
Don't iterate over new dictionaries with one item

Create the data and then add to dictionary.

Note: if you diff ignoring whitespace changes you
can more easily see the relevant changes.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-08-20 10:53:06 +08:00
830bf22fa5 BaseTools: AutoGen - tag a function as cachable
MakeFile generation is once per module, so mark it as such.
also move the time stamp creation function inside as it's
only called from one place.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-08-20 10:53:05 +08:00
b23414f654 BaseTools: AutoGen refactor ModuleAutoGen caching
1) Add a new file Common/caching.py
    a.	Allows for automated caching of repeated class functions, class
        properties, and non-class functions
    b.	When called the first time the value is cached and if called a
        second time, the cached result is called, not the function.
    c.	When used, this saves lots of memory since the actual function
        pointers are replaced with smaller data elements.
    d.  note that not all features are used yet.
2) Fix AutoGen/GenMake and AutoGen/GetC to not call into private member
    variables of ModuleAutoGen class
    a. use the existing accessor properties for the data
3) Change AutoGen classes to remove a exception for duplicate members in
    __new__ and use ?in? testing to speed up
4)	Work on ModuleAutoGen class
    a.	Change all properties that use caching to use @caching_property
        (see #1 above)
    b.	Change all properties that do not use caching to use standard python
        decorator "@property"
    c.	Change all cases where a dictionary/set/list/object was created
        and then immediately updated to use constructor parameters
    d.	Refactor each property function to remove the internal variable
        that is no longer needed (this helps find items like #2 above)
    e.  Refactor _ApplyBuildRule with optional parameter to work around
        circular dependency with BinaryFileList.

Note that 4e was almost certainly unintended as the functions were acting on
incomplete information since they were directly accessing private instance
variables at the same time another function on the stack was creating the same
private isntance data.

This overall changes behavior slightly, but makes the codebase smaller and
easier to read.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-08-20 10:53:05 +08:00
f843a32877 BaseTools: Add check for VOID* PCD Max Size
Per spec VOID* PCD max size should be a UINT16 value. so this patch
add the value check whether it is in range 0x0 .. 0xFFFF.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-08-17 13:05:14 +08:00
b9130c866d OvmfPkg: link Sha384 and Sha512 support into Tcg2Pei and Tcg2Dxe
Sha384 and Sha512 were added to HashInstanceLib recently. To make them
available in Tcg2Pei and Tcg2Dxe, we have to link both libraries.

Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Marc-André Lureau <marcandre.lureau@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Gary Lin <glin@suse.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Build-tested-by: Laszlo Ersek <lersek@redhat.com>
2018-08-16 20:32:52 +02:00
81502cee20 BaseTools/Source/C: take EXTRA_LDFLAGS from the caller
Allow the caller of the top-level makefile either to set EXTRA_LDFLAGS in
the environment or to pass EXTRA_LDFLAGS as a macro definition on the
command line. EXTRA_LDFLAGS extends (and potentially overrides) default
link-editing flags set in the makefiles.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Ref: https://bugzilla.redhat.com/show_bug.cgi?id=1540244
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-08-16 20:18:57 +02:00
b0ca5dae78 BaseTools/Source/C: take EXTRA_OPTFLAGS from the caller
Allow the caller of the top-level makefile either to set EXTRA_OPTFLAGS in
the environment or to pass EXTRA_OPTFLAGS as a macro definition on the
command line. EXTRA_OPTFLAGS extends (and potentially overrides) default C
compilation flags set in the makefiles.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Ref: https://bugzilla.redhat.com/show_bug.cgi?id=1540244
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-08-16 20:18:49 +02:00
b8a6617026 BaseTools/Source/C: split "-O2" to BUILD_OPTFLAGS
The option "-O2" is not a preprocessor flag, but a code generation
(compilation) flag. Move it from BUILD_CPPFLAGS to BUILD_CFLAGS and
BUILD_CXXFLAGS.

Because "VfrCompile/GNUmakefile" uses "-O2" through BUILD_CPPFLAGS, and
because it doesn't use BUILD_CXXFLAGS, we have to introduce BUILD_OPTFLAGS
separately, so that "VfrCompile/GNUmakefile" can continue using just this
flag.

This patch doesn't change behavior.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Ref: https://bugzilla.redhat.com/show_bug.cgi?id=1540244
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-08-16 20:18:49 +02:00
03252ae287 BaseTools/header.makefile: remove "-c" from BUILD_CFLAGS
Option "-c" is a mode selection flag (choosing between compiling and
linking); it should not be in BUILD_CFLAGS, which applies only to
compiling anyway. The compilation rule for C source files, in
"footer.makefile", already includes "-c" -- currently we have double "-c"
options.

This patch doesn't change behavior.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Ref: https://bugzilla.redhat.com/show_bug.cgi?id=1540244
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-08-16 20:18:49 +02:00
67983484a4 BaseTools/footer.makefile: expand BUILD_CFLAGS last for C files too
BUILD_CPPFLAGS should be expanded before BUILD_CFLAGS. (The rule for C++
source files already does this, with BUILD_CPPFLAGS and BUILD_CXXFLAGS.)

This patch doesn't change behavior.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Ref: https://bugzilla.redhat.com/show_bug.cgi?id=1540244
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-08-16 20:18:49 +02:00
9becf2f075 ShellPkg/UefiShellLib: rebase ShellOpenFileByDevicePath() to UefiLib API
Replace the "old shell method" implementation in
ShellOpenFileByDevicePath() with EfiOpenFileByDevicePath() from UefiLib,
correcting the following issues:

- code duplication between this module and other modules,
- local variable name "EfiSimpleFileSystemProtocol" starting with "Efi"
  prefix,
- bogus "FileHandle = NULL" assignments,
- leaking "Handle1" when the device path type/subtype check or the
  realignment-motivated AllocateCopyPool() fails in the loop.

Cc: Jaben Carsey <jaben.carsey@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1008
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-08-16 20:02:51 +02:00
15d8fe1fee ShellPkg/UefiShellLib: drop DeviceHandle param of ShellOpenFileByDevicePath()
The ShellOpenFileByDevicePath() API promises to set the DeviceHandle
output parameter to the handle of the filesystem identified by the
FilePath input parameter. However, this doesn't actually happen when the
UEFI Shell 2.0 method is used (which is basically "always" nowadays).

Accordingly, the only caller of ShellOpenFileByDevicePath(), namely
ShellOpenFileByName(), defines a (dummy) local DeviceHandle variable just
so it can call ShellOpenFileByDevicePath().

Remove the useless output parameter.

Cc: Jaben Carsey <jaben.carsey@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1008
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-08-16 20:02:50 +02:00
1bf5007428 SecurityPkg/SecureBootConfigDxe: replace OpenFileByDevicePath() with UefiLib API
Replace the OpenFileByDevicePath() function with EfiOpenFileByDevicePath()
from UefiLib, correcting the following issues:

- imprecise comments on OpenFileByDevicePath(),
- code duplication between this module and other modules,
- local variable name "EfiSimpleFileSystemProtocol" starting with "Efi"
  prefix,
- bogus "FileHandle = NULL" assignments,
- leaking "Handle1" when the device path type/subtype check or the
  realignment-motivated AllocateCopyPool() fails in the loop,
- stale SHELL_FILE_HANDLE reference in a comment.

Cc: Chao Zhang <chao.b.zhang@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Roman Bacik <roman.bacik@broadcom.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1008
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Chao Zhang <chao.b.zhang@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2018-08-16 20:02:50 +02:00
9f5d1f7c31 NetworkPkg/TlsAuthConfigDxe: replace OpenFileByDevicePath() with UefiLib API
Replace the OpenFileByDevicePath() function with EfiOpenFileByDevicePath()
from UefiLib, correcting the following issues:

- imprecise comments on OpenFileByDevicePath(),
- code duplication between this module and other modules,
- local variable name "EfiSimpleFileSystemProtocol" starting with "Efi"
  prefix,
- bogus "FileHandle = NULL" assignments,
- passing a potentially unaligned "FILEPATH_DEVICE_PATH.PathName" field to
  a protocol member function (forbidden by the UEFI spec),
- leaking "Handle1" when the device path type/subtype check fails in the
  loop,
- stale SHELL_FILE_HANDLE reference in a comment.

Cc: Jiaxin Wu <jiaxin.wu@intel.com>
Cc: Siyuan Fu <siyuan.fu@intel.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1008
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Jiaxin Wu <jiaxin.wu@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2018-08-16 20:02:50 +02:00
5b080845a1 MdeModulePkg/RamDiskDxe: replace OpenFileByDevicePath() with UefiLib API
Replace the OpenFileByDevicePath() function with EfiOpenFileByDevicePath()
from UefiLib, correcting the following issues:

- imprecise comments on OpenFileByDevicePath(),
- code duplication between this module and other modules,
- local variable name "EfiSimpleFileSystemProtocol" starting with "Efi"
  prefix,
- bogus "FileHandle = NULL" assignments,
- passing a potentially unaligned "FILEPATH_DEVICE_PATH.PathName" field to
  a protocol member function (forbidden by the UEFI spec),
- leaking "Handle1" when the device path type/subtype check fails in the
  loop,
- stale SHELL_FILE_HANDLE reference in a comment.

Cc: Eric Dong <eric.dong@intel.com>
Cc: Jiaxin Wu <jiaxin.wu@intel.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Siyuan Fu <siyuan.fu@intel.com>
Cc: Star Zeng <star.zeng@intel.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1008
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2018-08-16 20:02:50 +02:00
29b0bf26aa IntelFrameworkPkg/FrameworkUefiLib: introduce EfiOpenFileByDevicePath()
Copy the EfiOpenFileByDevicePath() implementation from the previous
(MdePkg/UefiLib) patch to FrameworkUefiLib.

(Note that the FrameworkUefiLib instance too will be updated for
<https://bugzilla.tianocore.org/show_bug.cgi?id=1074>.)

Cc: Liming Gao <liming.gao@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1008
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-08-16 20:02:46 +02:00
768b611136 MdePkg/UefiLib: introduce EfiOpenFileByDevicePath()
The EfiOpenFileByDevicePath() function centralizes functionality from

- MdeModulePkg/Universal/Disk/RamDiskDxe
- NetworkPkg/TlsAuthConfigDxe
- SecurityPkg/VariableAuthenticated/SecureBootConfigDxe
- ShellPkg/Library/UefiShellLib

unifying the implementation and fixing various bugs.

(Ray suggested that we eliminate the special handling of
EFI_FILE_MODE_CREATE in the "OpenMode" input parameter as well. We plan to
implement that separately, under
<https://bugzilla.tianocore.org/show_bug.cgi?id=1074>.)

Cc: Chao Zhang <chao.b.zhang@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Cc: Jaben Carsey <jaben.carsey@intel.com>
Cc: Jiaxin Wu <jiaxin.wu@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Roman Bacik <roman.bacik@broadcom.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Cc: Siyuan Fu <siyuan.fu@intel.com>
Cc: Star Zeng <star.zeng@intel.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1008
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-08-16 20:02:39 +02:00
52047be024 ShellPkg/redirection: Insert \xFEFF for ENV variable redirection
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1080

Per Shell spec 2.2 chapter 3.4.4.2, Unicode file tag should be
inserted in the output from the input redirected variable, to ensure
it looks like a UCS-2 encode file.

The patch fixes this issue.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2018-08-16 16:03:16 +08:00
bc0d3e2912 ShellPkg/redirection: Insert \xFEFF after converting ASCII to Unicode
When "<a" is used to redirect ASCII file to an application, Shell
core reads the ASCII file and converts the ASCII to Unicode as the
input source of the application.
But per Shell spec, the input source should have \xFEFF to indicate
it's a Unicode stream.
The patch adds the missing \xFEFF.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2018-08-16 16:03:15 +08:00
7b85a1afa3 BaseTools: Optimizing code for function doesn't match
Optimizing code for function doesn't match name and comment

Fix https://bugzilla.tianocore.org/show_bug.cgi?id=924

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-08-16 14:46:53 +08:00
4fea08b9c9 BaseTools: Eot - fix variable names
1) currently a couple classes use m instead of self (including some mixed
functions that should have previously failed).
2) deleted some blank lines.

Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-08-16 14:45:29 +08:00
9ddd4f7f94 BaseTools: Update the rule to remove .lib before link it for GCC
We met a case on GCC toolchain for increment build. the case is user
build Helloworld first, then rename the source file Helloworld.c to
Helloworld_new.c and also update the file name to Helloworld_new.c in
.inf file's [sources] section. finally, he rebuild it again.
It cause build failure due to multiple definition of `UefiMain' because
in the .lib file it both have Helloworld.obj and Helloworld_new.obj.
current we use the option 'cr' to create the .lib file while the 'r'
cmd means replace existing or insert new files into the archive. so
in this patch before we create the .lib file, we delete it first.

Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-08-16 14:17:29 +08:00
f64fbdde8c BaseTools: Clean up not used code in BuildClassObject
V2: Add back "from Common.DataType import *"

1. Remove some import statement that are not used.
2. Remove the Type value in the LibraryClassObject because we don't
actually use it.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Jaben Carsey <jaben.carsey@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2018-08-16 14:14:18 +08:00
f52c3ed019 BaseTools: Add Dns and BluetoothLE DevicePath
Add Dns and BluetoothLE for support DevicePath

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-08-16 13:41:55 +08:00
6f06101ddf BaseTools: Remove duplicate function declaration
Remove duplicate function declaration

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-08-16 13:26:04 +08:00
cd25509b1f IntelFrameworkModulePkg/Csm: Set CSM memory executable
Commit b22a62be5c
* IntelFrameworkModule/LegacyBios:Use reserved memory for legacy data
allocates reserved memory for holding legacy code/data.

But with PcdDxeNxMemoryProtectionPolicy set to certain value to
forbid execution when code is in certain type of memory, it's
possible that a platform forbids execution when code is in reserved
memory. The patch calls GCD service to allow such case otherwise
CPU exception may occur.

Code execution in BSCode area should be enabled by platform by
default.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Jian Wang <jian.j.wang@intel.com>
2018-08-16 11:38:29 +08:00
0e967dff06 ShellPkg/set: Fix EfiShellSetEnv to use case sensitive compare
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=777

Per Shell spec, the environment variable has a case-sensitive name.
But today's implementation of EfiShellSetEnv() compares the
environment variable name case insensitively, which causes variable
like "CWD" cannot be set due to "cwd" is pre-defined variable.

The patch fixes this issue.

The EfiShellGetEnv() doesn't have such issue because it will
call into ShellFindEnvVarInList() which uses StrCmp().

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ruiyu Ni <ruiyu.ni@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Jim Dailey <jim_dailey@dell.com>
2018-08-16 11:38:27 +08:00
855698fb69 BaseTools/Ecc: Fix import issues
1. Complete the full path for import statement. Use "EccMain" to
replace "Ecc" for the absolute path support.
2. Fix some issues on configuration file.
3. Fix an issue of RaiseError not working in EdkLogger.

Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Hess Chen <hesheng.chen@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-08-16 09:42:55 +08:00
af24640290 BaseTools: Fix report flexible value issue
Report flexible value in INF file encounter error

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-08-16 09:35:27 +08:00
b2282e5369 BaseTools: Remove the redundant if statement
after analysis the BuildOptionValue function, we found the if statement
IsFieldValueAnArray is redundant because ValueExpressionEx will handle
it.

Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-08-16 09:30:04 +08:00
a9ae5b37bf MdeModulePkg SmmLockBox: Return actual data length in SmmLockBoxRestore
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1082

Current code only returns actual data length in case of return status
buffer too small.
If caller provided a buffer that's large enough to restore data from
lock box, actual data length is not returned. It needs to be updated
in case of return status as success as well.

Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Anbazhagan Baraneedharan <anbazhagan@hp.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Jiewen Yao <jiewen.yao@intel.com>
2018-08-16 09:02:14 +08:00
a6daab1f6c UefiCpuPkg/RegisterCpuFeaturesLib: Combine implementation.
V1 changes:
> Current code logic can't confirm CpuS3DataDxe driver start before
> CpuFeaturesDxe driver. So the assumption in CpuFeaturesDxe not valid.
> Add implementation for AllocateAcpiCpuData function to remove this
> assumption.

V2 changes:
> Because CpuS3Data memory will be copy to smram at SmmReadToLock point,
> so the memory type no need to be ACPI NVS type, also the address not
> limit to below 4G.
> This change remove the limit of ACPI NVS memory type and below 4G.

V3 changes:
> Remove function definition in header file.
> Add STATIC in function implementation.

Pass OS boot and resume from S3 test.

Bugz: https://bugzilla.tianocore.org/show_bug.cgi?id=959

Reported-by: Marvin Häuser <Marvin.Haeuser@outlook.com>
Suggested-by: Fan Jeff <vanjeff_919@hotmail.com>
Cc: Marvin Häuser <Marvin.Haeuser@outlook.com>
Cc: Fan Jeff <vanjeff_919@hotmail.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
2018-08-16 08:42:01 +08:00
6eab8b4368 UefiCpuPkg/CpuS3DataDxe: Remove below 4G limitation.
Because PrepareApStartupVector() stores StackAddress to
"mExchangeInfo->StackStart" (which has type (VOID*)), and because
"UefiCpuPkg/PiSmmCpuDxeSmm/X64/MpFuncs.nasm" reads the latter with:

         add  edi, StackStartAddressLocation
         add  rax, qword [edi]
         mov  rsp, rax
         mov  qword [edi], rax

in long-mode code. So code can remove below 4G limitation.

Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
2018-08-16 08:42:01 +08:00
b581721063 UefiCpuPkg/CpuS3DataDxe: Change Memory Type and address limitation.
Because CpuS3Data memory will be copy to smram at SmmReadyToLock point,
the memory type no need to be ACPI NVS type, also the address not
limit to below 4G.

This change remove the limit of ACPI NVS memory type and below 4G.

Pass OS boot and resume from S3 test.

Cc: Marvin Häuser <Marvin.Haeuser@outlook.com>
Cc: Fan Jeff <vanjeff_919@hotmail.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
2018-08-16 08:42:01 +08:00
fcd92f16af UefiCpuPkg/AcpiCpuData.h: Remove AcpiNVS and Below 4G limitation.
ACPI_CPU_DATA structure first introduced to save data in
normal boot phase. Also this data will be used in S3 phase
by one PEI driver. So in first phase, this data is been
defined to use ACPI NVS memory type and must below 4G.

Later in order to fix potential security issue,
PiSmmCpuDxeSmm driver added logic to copy ACPI_CPU_DATA
(except ResetVector and Stack buffer) to  smram at smm
ready to lock point. ResetVector must below 1M and Stack
buffer is write only in S3 phase, so these two fields not
copy to smram. Also PiSmmCpuDxeSmm driver owned the task
to restore the CPU setting and it's a SMM driver.

After above change, the acpi nvs memory type and below 4G
limitation is no longer needed.

This change remove the limitation in the comments for
ACPI_CPU_DATA definition.

Cc: Marvin Häuser <Marvin.Haeuser@outlook.com>
Cc: Fan Jeff <vanjeff_919@hotmail.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
2018-08-16 08:42:00 +08:00
293f876618 UefiCpuPkg/PiSmmCpuDxeSmm: Use GDT/IDT saved in Smram.
Current implementation will copy GDT/IDT at SmmReadyToLock point
from ACPI NVS memory to Smram. Later at S3 resume phase, it restore
the memory saved in Smram to ACPI NVS. It can directly use GDT/IDT
saved in Smram instead of restore the original ACPI NVS memory.
This patch do this change.

Test Done:
  Do the OS boot and S3 resume test.

Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Ruiyu Ni <ruiyu.ni@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Ruiyu Ni <ruiyu.ni@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
2018-08-16 08:42:00 +08:00
0ca81c0d37 SecurityPkg/TcgStorageCoreLib.h: Use ascii instead of unicode.
_FILE_ parameter is an ASCII string. Current implementation used
as unicode string instead of ascii string. This patch fixed this
issue.

BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1081

Cc: Hao Wu <hao.a.wu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed_off_by: Eric Dong <eric.dong@intel.com>
Reviewed-by: Hao Wu <hao.a.wu@intel.com>
2018-08-16 08:41:59 +08:00
0fc5e71c68 BaseTools: Remove a unused function.
the call statement of _CheckDuplicateInFV() was commented out
in 2014. There is no call statement of _CheckDuplicateInFV(),
so remove it.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-08-16 08:32:58 +08:00
669b55e6d5 BaseTool: Fixed the bug of Boolean Hii Pcd packing.
When packing HiiPcd into PcdNvStoreDefaultValueBuffer,
The boolean type pcd value packing incorrect.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-08-16 08:32:57 +08:00
1531 changed files with 131729 additions and 136370 deletions

View File

@ -1,6 +1,6 @@
/** @file
*
* Copyright (c) 2011-2017, ARM Limited. All rights reserved.
* Copyright (c) 2011-2018, ARM Limited. All rights reserved.
*
* This program and the accompanying materials
* are licensed and made available under the terms and conditions of the BSD License
@ -19,6 +19,16 @@
#include <Library/IoLib.h>
#include <Library/PcdLib.h>
// In GICv3, there are 2 x 64KB frames:
// Redistributor control frame + SGI Control & Generation frame
#define GIC_V3_REDISTRIBUTOR_GRANULARITY (ARM_GICR_CTLR_FRAME_SIZE \
+ ARM_GICR_SGI_PPI_FRAME_SIZE)
// In GICv4, there are 2 additional 64KB frames:
// VLPI frame + Reserved page frame
#define GIC_V4_REDISTRIBUTOR_GRANULARITY (GIC_V3_REDISTRIBUTOR_GRANULARITY \
+ ARM_GICR_SGI_VLPI_FRAME_SIZE \
+ ARM_GICR_SGI_RESERVED_FRAME_SIZE)
#define ISENABLER_ADDRESS(base,offset) ((base) + \
ARM_GICR_CTLR_FRAME_SIZE + ARM_GICR_ISENABLER + (4 * offset))
@ -54,12 +64,11 @@ GicGetCpuRedistributorBase (
IN ARM_GIC_ARCH_REVISION Revision
)
{
UINTN Index;
UINTN MpId;
UINTN CpuAffinity;
UINTN Affinity;
UINTN GicRedistributorGranularity;
UINTN GicCpuRedistributorBase;
UINT64 TypeRegister;
MpId = ArmReadMpidr ();
// Define CPU affinity as:
@ -68,27 +77,30 @@ GicGetCpuRedistributorBase (
CpuAffinity = (MpId & (ARM_CORE_AFF0 | ARM_CORE_AFF1 | ARM_CORE_AFF2)) |
((MpId & ARM_CORE_AFF3) >> 8);
if (Revision == ARM_GIC_ARCH_REVISION_3) {
// 2 x 64KB frame:
// Redistributor control frame + SGI Control & Generation frame
GicRedistributorGranularity = ARM_GICR_CTLR_FRAME_SIZE
+ ARM_GICR_SGI_PPI_FRAME_SIZE;
} else {
if (Revision < ARM_GIC_ARCH_REVISION_3) {
ASSERT_EFI_ERROR (EFI_UNSUPPORTED);
return 0;
}
GicCpuRedistributorBase = GicRedistributorBase;
for (Index = 0; Index < PcdGet32 (PcdCoreCount); Index++) {
Affinity = MmioRead64 (GicCpuRedistributorBase + ARM_GICR_TYPER) >> 32;
do {
TypeRegister = MmioRead64 (GicCpuRedistributorBase + ARM_GICR_TYPER);
Affinity = ARM_GICR_TYPER_GET_AFFINITY (TypeRegister);
if (Affinity == CpuAffinity) {
return GicCpuRedistributorBase;
}
// Move to the next GIC Redistributor frame
GicCpuRedistributorBase += GicRedistributorGranularity;
}
// Move to the next GIC Redistributor frame.
// The GIC specification does not forbid a mixture of redistributors
// with or without support for virtual LPIs, so we test Virtual LPIs
// Support (VLPIS) bit for each frame to decide the granularity.
// Note: The assumption here is that the redistributors are adjacent
// for all CPUs. However this may not be the case for NUMA systems.
GicCpuRedistributorBase += (((ARM_GICR_TYPER_VLPIS & TypeRegister) != 0)
? GIC_V4_REDISTRIBUTOR_GRANULARITY
: GIC_V3_REDISTRIBUTOR_GRANULARITY);
} while ((TypeRegister & ARM_GICR_TYPER_LAST) == 0);
// The Redistributor has not been found for the current CPU
ASSERT_EFI_ERROR (EFI_NOT_FOUND);

View File

@ -1,5 +1,5 @@
#/* @file
# Copyright (c) 2011-2015, ARM Limited. All rights reserved.
# Copyright (c) 2011-2018, ARM Limited. All rights reserved.
#
# This program and the accompanying materials
# are licensed and made available under the terms and conditions of the BSD License
@ -44,8 +44,5 @@
ArmPlatformPkg/ArmPlatformPkg.dec
MdePkg/MdePkg.dec
[Pcd]
gArmPlatformTokenSpaceGuid.PcdCoreCount
[FeaturePcd]
gArmTokenSpaceGuid.PcdArmGicV3WithV2Legacy

View File

@ -469,7 +469,7 @@ GicV3DxeInitialize (
for (Index = 0; Index < (mGicNumInterrupts - 32); Index++) {
MmioWrite32 (
mGicDistributorBase + ARM_GICD_IROUTER + (Index * 8),
CpuTarget | ARM_GICD_IROUTER_IRM
CpuTarget
);
}
}

View File

@ -1,6 +1,6 @@
/** @file
*
* Copyright (c) 2011-2017, ARM Limited. All rights reserved.
* Copyright (c) 2011-2018, ARM Limited. All rights reserved.
*
* This program and the accompanying materials
* are licensed and made available under the terms and conditions of the BSD License
@ -60,12 +60,28 @@
// GIC Redistributor
#define ARM_GICR_CTLR_FRAME_SIZE SIZE_64KB
#define ARM_GICR_SGI_PPI_FRAME_SIZE SIZE_64KB
#define ARM_GICR_CTLR_FRAME_SIZE SIZE_64KB
#define ARM_GICR_SGI_PPI_FRAME_SIZE SIZE_64KB
#define ARM_GICR_SGI_VLPI_FRAME_SIZE SIZE_64KB
#define ARM_GICR_SGI_RESERVED_FRAME_SIZE SIZE_64KB
// GIC Redistributor Control frame
#define ARM_GICR_TYPER 0x0008 // Redistributor Type Register
// GIC Redistributor TYPER bit assignments
#define ARM_GICR_TYPER_PLPIS (1 << 0) // Physical LPIs
#define ARM_GICR_TYPER_VLPIS (1 << 1) // Virtual LPIs
#define ARM_GICR_TYPER_DIRECTLPI (1 << 3) // Direct LPIs
#define ARM_GICR_TYPER_LAST (1 << 4) // Last Redistributor in series
#define ARM_GICR_TYPER_DPGS (1 << 5) // Disable Processor Group
// Selection Support
#define ARM_GICR_TYPER_PROCNO (0xFFFF << 8) // Processor Number
#define ARM_GICR_TYPER_COMMONLPIAFF (0x3 << 24) // Common LPI Affinity
#define ARM_GICR_TYPER_AFFINITY (0xFFFFFFFFULL << 32) // Redistributor Affinity
#define ARM_GICR_TYPER_GET_AFFINITY(TypeReg) (((TypeReg) & \
ARM_GICR_TYPER_AFFINITY) >> 32)
// GIC SGI & PPI Redistributor frame
#define ARM_GICR_ISENABLER 0x0100 // Interrupt Set-Enable Registers
#define ARM_GICR_ICENABLER 0x0180 // Interrupt Clear-Enable Registers

View File

@ -25,10 +25,98 @@
#define OPTEE_OS_UID2 0xaf630002
#define OPTEE_OS_UID3 0xa5d5c51b
#define OPTEE_MESSAGE_ATTRIBUTE_TYPE_NONE 0x0
#define OPTEE_MESSAGE_ATTRIBUTE_TYPE_VALUE_INPUT 0x1
#define OPTEE_MESSAGE_ATTRIBUTE_TYPE_VALUE_OUTPUT 0x2
#define OPTEE_MESSAGE_ATTRIBUTE_TYPE_VALUE_INOUT 0x3
#define OPTEE_MESSAGE_ATTRIBUTE_TYPE_MEMORY_INPUT 0x9
#define OPTEE_MESSAGE_ATTRIBUTE_TYPE_MEMORY_OUTPUT 0xa
#define OPTEE_MESSAGE_ATTRIBUTE_TYPE_MEMORY_INOUT 0xb
#define OPTEE_MESSAGE_ATTRIBUTE_TYPE_MASK 0xff
#define OPTEE_ORIGIN_COMMUNICATION 0x00000002
#define OPTEE_ERROR_COMMUNICATION 0xFFFF000E
typedef struct {
UINT64 BufferAddress;
UINT64 Size;
UINT64 SharedMemoryReference;
} OPTEE_MESSAGE_PARAM_MEMORY;
typedef struct {
UINT64 A;
UINT64 B;
UINT64 C;
} OPTEE_MESSAGE_PARAM_VALUE;
typedef struct {
UINT64 Attribute;
union {
OPTEE_MESSAGE_PARAM_MEMORY Memory;
OPTEE_MESSAGE_PARAM_VALUE Value;
} Union;
} OPTEE_MESSAGE_PARAM;
#define OPTEE_MAX_CALL_PARAMS 4
typedef struct {
UINT32 Command;
UINT32 Function;
UINT32 Session;
UINT32 CancelId;
UINT32 Pad;
UINT32 Return;
UINT32 ReturnOrigin;
UINT32 NumParams;
// NumParams tells the actual number of element in Params
OPTEE_MESSAGE_PARAM Params[OPTEE_MAX_CALL_PARAMS];
} OPTEE_MESSAGE_ARG;
typedef struct {
EFI_GUID Uuid; // [in] GUID/UUID of the Trusted Application
UINT32 Session; // [out] Session id
UINT32 Return; // [out] Return value
UINT32 ReturnOrigin; // [out] Origin of the return value
} OPTEE_OPEN_SESSION_ARG;
typedef struct {
UINT32 Function; // [in] Trusted Application function, specific to the TA
UINT32 Session; // [in] Session id
UINT32 Return; // [out] Return value
UINT32 ReturnOrigin; // [out] Origin of the return value
OPTEE_MESSAGE_PARAM Params[OPTEE_MAX_CALL_PARAMS]; // Params for function to be invoked
} OPTEE_INVOKE_FUNCTION_ARG;
BOOLEAN
EFIAPI
IsOpteePresent (
VOID
);
EFI_STATUS
EFIAPI
OpteeInit (
VOID
);
EFI_STATUS
EFIAPI
OpteeOpenSession (
IN OUT OPTEE_OPEN_SESSION_ARG *OpenSessionArg
);
EFI_STATUS
EFIAPI
OpteeCloseSession (
IN UINT32 Session
);
EFI_STATUS
EFIAPI
OpteeInvokeFunction (
IN OUT OPTEE_INVOKE_FUNCTION_ARG *InvokeFunctionArg
);
#endif

View File

@ -14,11 +14,18 @@
**/
#include <Library/ArmMmuLib.h>
#include <Library/ArmSmcLib.h>
#include <Library/BaseMemoryLib.h>
#include <Library/BaseLib.h>
#include <Library/DebugLib.h>
#include <Library/OpteeLib.h>
#include <IndustryStandard/ArmStdSmc.h>
#include <OpteeSmc.h>
#include <Uefi.h>
STATIC OPTEE_SHARED_MEMORY_INFORMATION OpteeSharedMemoryInformation = { 0 };
/**
Check for OP-TEE presence.
@ -31,6 +38,7 @@ IsOpteePresent (
{
ARM_SMC_ARGS ArmSmcArgs;
ZeroMem (&ArmSmcArgs, sizeof (ARM_SMC_ARGS));
// Send a Trusted OS Calls UID command
ArmSmcArgs.Arg0 = ARM_SMC_ID_TOS_UID;
ArmCallSmc (&ArmSmcArgs);
@ -44,3 +52,387 @@ IsOpteePresent (
return FALSE;
}
}
STATIC
EFI_STATUS
OpteeSharedMemoryRemap (
VOID
)
{
ARM_SMC_ARGS ArmSmcArgs;
EFI_PHYSICAL_ADDRESS PhysicalAddress;
EFI_PHYSICAL_ADDRESS Start;
EFI_PHYSICAL_ADDRESS End;
EFI_STATUS Status;
UINTN Size;
ZeroMem (&ArmSmcArgs, sizeof (ARM_SMC_ARGS));
ArmSmcArgs.Arg0 = OPTEE_SMC_GET_SHARED_MEMORY_CONFIG;
ArmCallSmc (&ArmSmcArgs);
if (ArmSmcArgs.Arg0 != OPTEE_SMC_RETURN_OK) {
DEBUG ((DEBUG_WARN, "OP-TEE shared memory not supported\n"));
return EFI_UNSUPPORTED;
}
if (ArmSmcArgs.Arg3 != OPTEE_SMC_SHARED_MEMORY_CACHED) {
DEBUG ((DEBUG_WARN, "OP-TEE: Only normal cached shared memory supported\n"));
return EFI_UNSUPPORTED;
}
Start = (ArmSmcArgs.Arg1 + SIZE_4KB - 1) & ~(SIZE_4KB - 1);
End = (ArmSmcArgs.Arg1 + ArmSmcArgs.Arg2) & ~(SIZE_4KB - 1);
PhysicalAddress = Start;
Size = End - Start;
if (Size < SIZE_4KB) {
DEBUG ((DEBUG_WARN, "OP-TEE shared memory too small\n"));
return EFI_BUFFER_TOO_SMALL;
}
Status = ArmSetMemoryAttributes (PhysicalAddress, Size, EFI_MEMORY_WB);
if (EFI_ERROR (Status)) {
return Status;
}
OpteeSharedMemoryInformation.Base = (UINTN)PhysicalAddress;
OpteeSharedMemoryInformation.Size = Size;
return EFI_SUCCESS;
}
EFI_STATUS
EFIAPI
OpteeInit (
VOID
)
{
EFI_STATUS Status;
if (!IsOpteePresent ()) {
DEBUG ((DEBUG_WARN, "OP-TEE not present\n"));
return EFI_UNSUPPORTED;
}
Status = OpteeSharedMemoryRemap ();
if (EFI_ERROR (Status)) {
DEBUG ((DEBUG_WARN, "OP-TEE shared memory remap failed\n"));
return Status;
}
return EFI_SUCCESS;
}
/**
Does Standard SMC to OP-TEE in secure world.
@param[in] PhysicalArg Physical address of message to pass to secure world
@return 0 on success, secure world return code otherwise
**/
STATIC
UINT32
OpteeCallWithArg (
IN UINT64 PhysicalArg
)
{
ARM_SMC_ARGS ArmSmcArgs;
ZeroMem (&ArmSmcArgs, sizeof (ARM_SMC_ARGS));
ArmSmcArgs.Arg0 = OPTEE_SMC_CALL_WITH_ARG;
ArmSmcArgs.Arg1 = (UINT32)(PhysicalArg >> 32);
ArmSmcArgs.Arg2 = (UINT32)PhysicalArg;
while (TRUE) {
ArmCallSmc (&ArmSmcArgs);
if (ArmSmcArgs.Arg0 == OPTEE_SMC_RETURN_RPC_FOREIGN_INTERRUPT) {
//
// A foreign interrupt was raised while secure world was
// executing, since they are handled in UEFI a dummy RPC is
// performed to let UEFI take the interrupt through the normal
// vector.
//
ArmSmcArgs.Arg0 = OPTEE_SMC_RETURN_FROM_RPC;
} else {
break;
}
}
return ArmSmcArgs.Arg0;
}
STATIC
VOID
EfiGuidToRfc4122Uuid (
OUT RFC4122_UUID *Rfc4122Uuid,
IN EFI_GUID *Guid
)
{
Rfc4122Uuid->Data1 = SwapBytes32 (Guid->Data1);
Rfc4122Uuid->Data2 = SwapBytes16 (Guid->Data2);
Rfc4122Uuid->Data3 = SwapBytes16 (Guid->Data3);
CopyMem (Rfc4122Uuid->Data4, Guid->Data4, sizeof (Rfc4122Uuid->Data4));
}
EFI_STATUS
EFIAPI
OpteeOpenSession (
IN OUT OPTEE_OPEN_SESSION_ARG *OpenSessionArg
)
{
OPTEE_MESSAGE_ARG *MessageArg;
MessageArg = NULL;
if (OpteeSharedMemoryInformation.Base == 0) {
DEBUG ((DEBUG_WARN, "OP-TEE not initialized\n"));
return EFI_NOT_STARTED;
}
MessageArg = (OPTEE_MESSAGE_ARG *)OpteeSharedMemoryInformation.Base;
ZeroMem (MessageArg, sizeof (OPTEE_MESSAGE_ARG));
MessageArg->Command = OPTEE_MESSAGE_COMMAND_OPEN_SESSION;
//
// Initialize and add the meta parameters needed when opening a
// session.
//
MessageArg->Params[0].Attribute = OPTEE_MESSAGE_ATTRIBUTE_TYPE_VALUE_INPUT |
OPTEE_MESSAGE_ATTRIBUTE_META;
MessageArg->Params[1].Attribute = OPTEE_MESSAGE_ATTRIBUTE_TYPE_VALUE_INPUT |
OPTEE_MESSAGE_ATTRIBUTE_META;
EfiGuidToRfc4122Uuid (
(RFC4122_UUID *)&MessageArg->Params[0].Union.Value,
&OpenSessionArg->Uuid
);
ZeroMem (&MessageArg->Params[1].Union.Value, sizeof (EFI_GUID));
MessageArg->Params[1].Union.Value.C = OPTEE_LOGIN_PUBLIC;
MessageArg->NumParams = 2;
if (OpteeCallWithArg ((UINTN)MessageArg)) {
MessageArg->Return = OPTEE_ERROR_COMMUNICATION;
MessageArg->ReturnOrigin = OPTEE_ORIGIN_COMMUNICATION;
}
OpenSessionArg->Session = MessageArg->Session;
OpenSessionArg->Return = MessageArg->Return;
OpenSessionArg->ReturnOrigin = MessageArg->ReturnOrigin;
return EFI_SUCCESS;
}
EFI_STATUS
EFIAPI
OpteeCloseSession (
IN UINT32 Session
)
{
OPTEE_MESSAGE_ARG *MessageArg;
MessageArg = NULL;
if (OpteeSharedMemoryInformation.Base == 0) {
DEBUG ((DEBUG_WARN, "OP-TEE not initialized\n"));
return EFI_NOT_STARTED;
}
MessageArg = (OPTEE_MESSAGE_ARG *)OpteeSharedMemoryInformation.Base;
ZeroMem (MessageArg, sizeof (OPTEE_MESSAGE_ARG));
MessageArg->Command = OPTEE_MESSAGE_COMMAND_CLOSE_SESSION;
MessageArg->Session = Session;
OpteeCallWithArg ((UINTN)MessageArg);
return EFI_SUCCESS;
}
STATIC
EFI_STATUS
OpteeToMessageParam (
OUT OPTEE_MESSAGE_PARAM *MessageParams,
IN UINT32 NumParams,
IN OPTEE_MESSAGE_PARAM *InParams
)
{
UINT32 Idx;
UINTN ParamSharedMemoryAddress;
UINTN SharedMemorySize;
UINTN Size;
Size = (sizeof (OPTEE_MESSAGE_ARG) + sizeof (UINT64) - 1) &
~(sizeof (UINT64) - 1);
ParamSharedMemoryAddress = OpteeSharedMemoryInformation.Base + Size;
SharedMemorySize = OpteeSharedMemoryInformation.Size - Size;
for (Idx = 0; Idx < NumParams; Idx++) {
CONST OPTEE_MESSAGE_PARAM *InParam;
OPTEE_MESSAGE_PARAM *MessageParam;
UINT32 Attribute;
InParam = InParams + Idx;
MessageParam = MessageParams + Idx;
Attribute = InParam->Attribute & OPTEE_MESSAGE_ATTRIBUTE_TYPE_MASK;
switch (Attribute) {
case OPTEE_MESSAGE_ATTRIBUTE_TYPE_NONE:
MessageParam->Attribute = OPTEE_MESSAGE_ATTRIBUTE_TYPE_NONE;
ZeroMem (&MessageParam->Union, sizeof (MessageParam->Union));
break;
case OPTEE_MESSAGE_ATTRIBUTE_TYPE_VALUE_INPUT:
case OPTEE_MESSAGE_ATTRIBUTE_TYPE_VALUE_OUTPUT:
case OPTEE_MESSAGE_ATTRIBUTE_TYPE_VALUE_INOUT:
MessageParam->Attribute = Attribute;
MessageParam->Union.Value.A = InParam->Union.Value.A;
MessageParam->Union.Value.B = InParam->Union.Value.B;
MessageParam->Union.Value.C = InParam->Union.Value.C;
break;
case OPTEE_MESSAGE_ATTRIBUTE_TYPE_MEMORY_INPUT:
case OPTEE_MESSAGE_ATTRIBUTE_TYPE_MEMORY_OUTPUT:
case OPTEE_MESSAGE_ATTRIBUTE_TYPE_MEMORY_INOUT:
MessageParam->Attribute = Attribute;
if (InParam->Union.Memory.Size > SharedMemorySize) {
return EFI_OUT_OF_RESOURCES;
}
CopyMem (
(VOID *)ParamSharedMemoryAddress,
(VOID *)(UINTN)InParam->Union.Memory.BufferAddress,
InParam->Union.Memory.Size
);
MessageParam->Union.Memory.BufferAddress = (UINT64)ParamSharedMemoryAddress;
MessageParam->Union.Memory.Size = InParam->Union.Memory.Size;
Size = (InParam->Union.Memory.Size + sizeof (UINT64) - 1) &
~(sizeof (UINT64) - 1);
ParamSharedMemoryAddress += Size;
SharedMemorySize -= Size;
break;
default:
return EFI_INVALID_PARAMETER;
}
}
return EFI_SUCCESS;
}
STATIC
EFI_STATUS
OpteeFromMessageParam (
OUT OPTEE_MESSAGE_PARAM *OutParams,
IN UINT32 NumParams,
IN OPTEE_MESSAGE_PARAM *MessageParams
)
{
UINT32 Idx;
for (Idx = 0; Idx < NumParams; Idx++) {
OPTEE_MESSAGE_PARAM *OutParam;
CONST OPTEE_MESSAGE_PARAM *MessageParam;
UINT32 Attribute;
OutParam = OutParams + Idx;
MessageParam = MessageParams + Idx;
Attribute = MessageParam->Attribute & OPTEE_MESSAGE_ATTRIBUTE_TYPE_MASK;
switch (Attribute) {
case OPTEE_MESSAGE_ATTRIBUTE_TYPE_NONE:
OutParam->Attribute = OPTEE_MESSAGE_ATTRIBUTE_TYPE_NONE;
ZeroMem (&OutParam->Union, sizeof (OutParam->Union));
break;
case OPTEE_MESSAGE_ATTRIBUTE_TYPE_VALUE_INPUT:
case OPTEE_MESSAGE_ATTRIBUTE_TYPE_VALUE_OUTPUT:
case OPTEE_MESSAGE_ATTRIBUTE_TYPE_VALUE_INOUT:
OutParam->Attribute = Attribute;
OutParam->Union.Value.A = MessageParam->Union.Value.A;
OutParam->Union.Value.B = MessageParam->Union.Value.B;
OutParam->Union.Value.C = MessageParam->Union.Value.C;
break;
case OPTEE_MESSAGE_ATTRIBUTE_TYPE_MEMORY_INPUT:
case OPTEE_MESSAGE_ATTRIBUTE_TYPE_MEMORY_OUTPUT:
case OPTEE_MESSAGE_ATTRIBUTE_TYPE_MEMORY_INOUT:
OutParam->Attribute = Attribute;
if (MessageParam->Union.Memory.Size > OutParam->Union.Memory.Size) {
return EFI_BAD_BUFFER_SIZE;
}
CopyMem (
(VOID *)(UINTN)OutParam->Union.Memory.BufferAddress,
(VOID *)(UINTN)MessageParam->Union.Memory.BufferAddress,
MessageParam->Union.Memory.Size
);
OutParam->Union.Memory.Size = MessageParam->Union.Memory.Size;
break;
default:
return EFI_INVALID_PARAMETER;
}
}
return EFI_SUCCESS;
}
EFI_STATUS
EFIAPI
OpteeInvokeFunction (
IN OUT OPTEE_INVOKE_FUNCTION_ARG *InvokeFunctionArg
)
{
EFI_STATUS Status;
OPTEE_MESSAGE_ARG *MessageArg;
MessageArg = NULL;
if (OpteeSharedMemoryInformation.Base == 0) {
DEBUG ((DEBUG_WARN, "OP-TEE not initialized\n"));
return EFI_NOT_STARTED;
}
MessageArg = (OPTEE_MESSAGE_ARG *)OpteeSharedMemoryInformation.Base;
ZeroMem (MessageArg, sizeof (OPTEE_MESSAGE_ARG));
MessageArg->Command = OPTEE_MESSAGE_COMMAND_INVOKE_FUNCTION;
MessageArg->Function = InvokeFunctionArg->Function;
MessageArg->Session = InvokeFunctionArg->Session;
Status = OpteeToMessageParam (
MessageArg->Params,
OPTEE_MAX_CALL_PARAMS,
InvokeFunctionArg->Params
);
if (Status) {
return Status;
}
MessageArg->NumParams = OPTEE_MAX_CALL_PARAMS;
if (OpteeCallWithArg ((UINTN)MessageArg)) {
MessageArg->Return = OPTEE_ERROR_COMMUNICATION;
MessageArg->ReturnOrigin = OPTEE_ORIGIN_COMMUNICATION;
}
if (OpteeFromMessageParam (
InvokeFunctionArg->Params,
OPTEE_MAX_CALL_PARAMS,
MessageArg->Params
)) {
MessageArg->Return = OPTEE_ERROR_COMMUNICATION;
MessageArg->ReturnOrigin = OPTEE_ORIGIN_COMMUNICATION;
}
InvokeFunctionArg->Return = MessageArg->Return;
InvokeFunctionArg->ReturnOrigin = MessageArg->ReturnOrigin;
return EFI_SUCCESS;
}

View File

@ -23,11 +23,13 @@
[Sources]
Optee.c
OpteeSmc.h
[Packages]
ArmPkg/ArmPkg.dec
MdePkg/MdePkg.dec
[LibraryClasses]
ArmMmuLib
ArmSmcLib
BaseLib

View File

@ -0,0 +1,53 @@
/** @file
OP-TEE SMC header file.
Copyright (c) 2018, Linaro Ltd. All rights reserved.<BR>
This program and the accompanying materials
are licensed and made available under the terms and conditions of the BSD License
which accompanies this distribution. The full text of the license may be found at
http://opensource.org/licenses/bsd-license.php
THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS,
WITHOUT WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED.
**/
#ifndef _OPTEE_SMC_H_
#define _OPTEE_SMC_H_
/* Returned in Arg0 only from Trusted OS functions */
#define OPTEE_SMC_RETURN_OK 0x0
#define OPTEE_SMC_RETURN_FROM_RPC 0x32000003
#define OPTEE_SMC_CALL_WITH_ARG 0x32000004
#define OPTEE_SMC_GET_SHARED_MEMORY_CONFIG 0xb2000007
#define OPTEE_SMC_SHARED_MEMORY_CACHED 1
#define OPTEE_SMC_RETURN_RPC_FOREIGN_INTERRUPT 0xffff0004
#define OPTEE_MESSAGE_COMMAND_OPEN_SESSION 0
#define OPTEE_MESSAGE_COMMAND_INVOKE_FUNCTION 1
#define OPTEE_MESSAGE_COMMAND_CLOSE_SESSION 2
#define OPTEE_MESSAGE_ATTRIBUTE_META 0x100
#define OPTEE_LOGIN_PUBLIC 0x0
typedef struct {
UINTN Base;
UINTN Size;
} OPTEE_SHARED_MEMORY_INFORMATION;
//
// UUID struct compliant with RFC4122 (network byte order).
//
typedef struct {
UINT32 Data1;
UINT16 Data2;
UINT16 Data3;
UINT8 Data4[8];
} RFC4122_UUID;
#endif

View File

@ -479,7 +479,7 @@ GetPlatformOptions (
NULL,
BootOptionNumber,
0,
BootKeys[Index],
&BootKeys[Index],
NULL
);
if (EFI_ERROR (Status)) {

View File

@ -11,4 +11,4 @@ dir=$(dirname "$full_cmd")
cmd=${full_cmd##*/}
export PYTHONPATH="$dir/../../Source/Python${PYTHONPATH:+:"$PYTHONPATH"}"
exec "${python_exe:-python}" -m $cmd.$cmd "$@"
exec "${python_exe:-python}" -m $cmd.$cmd "$@"

View File

@ -2,7 +2,7 @@
#
# This script will exec Brotli tool with -e/-d options.
#
# Copyright (c) 2017, Intel Corporation. All rights reserved.<BR>
# Copyright (c) 2017 - 2018, Intel Corporation. All rights reserved.<BR>
# This program and the accompanying materials
# are licensed and made available under the terms and conditions of the BSD License
# which accompanies this distribution. The full text of the license may be found at
@ -11,18 +11,15 @@
# THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS,
# WITHOUT WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED.
#
QLT="-q 9"
INPUTFLAG=0
QLT="-q 9 -w 22"
ARGS=
while test $# -gt 0
do
case $1 in
-e)
INPUTFLAG=1
;;
-d)
INPUTFLAG=1
ARGS+="$1 "
;;
-o|-g)
@ -34,16 +31,10 @@ do
shift
;;
*)
if [ $INPUTFLAG -eq 1 ]
then
ARGS+="-i $1 "
INPUTFLAG=0
else
ARGS+="$1 "
fi
ARGS+="$1 "
;;
esac
shift
done
exec Brotli $ARGS $QLT
exec Brotli $QLT $ARGS

View File

@ -11,4 +11,4 @@ dir=$(dirname "$full_cmd")
cmd=${full_cmd##*/}
export PYTHONPATH="$dir/../../Source/Python${PYTHONPATH:+:"$PYTHONPATH"}"
exec "${python_exe:-python}" -m $cmd.$cmd "$@"
exec "${python_exe:-python}" -m $cmd.EccMain "$@"

View File

@ -11,4 +11,4 @@ dir=$(dirname "$full_cmd")
cmd=${full_cmd##*/}
export PYTHONPATH="$dir/../../Source/Python${PYTHONPATH:+:"$PYTHONPATH"}"
exec "${python_exe:-python}" -m $cmd.$cmd "$@"
exec "${python_exe:-python}" -m $cmd.$cmd "$@"

View File

@ -1,7 +1,7 @@
@REM @file
@REM This script will exec Brotli tool with -e/-d options.
@REM
@REM Copyright (c) 2017, Intel Corporation. All rights reserved.<BR>
@REM Copyright (c) 2017 - 2018, Intel Corporation. All rights reserved.<BR>
@REM This program and the accompanying materials
@REM are licensed and made available under the terms and conditions of the BSD License
@REM which accompanies this distribution. The full text of the license may be found at
@ -14,22 +14,19 @@
@echo off
@setlocal
set QLT=-q 9
set INPUTFLAG=0
set QLT=-q 9 -w 22
set ARGS=
:Begin
if "%1"=="" goto End
if "%1"=="-d" (
set INPUTFLAG=1
set ARGS=%ARGS% %1
shift
goto Begin
)
if "%1"=="-e" (
set INPUTFLAG=1
shift
goto Begin
)
@ -55,15 +52,10 @@ if "%1"=="-q" (
goto Begin
)
if %INPUTFLAG% == 1 (
set ARGS=%ARGS% -i %1
set INPUTFLAG=0
) else (
set ARGS=%ARGS% %1
)
set ARGS=%ARGS% %1
shift
goto Begin
:End
Brotli %ARGS% %QLT%
Brotli %QLT% %ARGS%
@echo on

View File

@ -1,4 +1,4 @@
@setlocal
@set ToolName=%~n0%
@set PYTHONPATH=%PYTHONPATH%;%BASE_TOOLS_PATH%\Source\Python
@%PYTHON_HOME%\python.exe -m %ToolName%.%ToolName% %*
@%PYTHON_HOME%\python.exe -m %ToolName%.EccMain %*

View File

@ -13,8 +13,7 @@ Quick Start
-----------
Windows:
a) Set the PYTHON_FREEZER_PATH to the cx_Freeze installation directory
b) Go to the <buildtools_project>/BaseTools and run "toolsetup" script
a) Go to the <buildtools_project>/BaseTools and run "toolsetup" script
Unix-like:
a) make -C <buildtools_project>/BaseTools

View File

@ -1,5 +1,5 @@
#
# Copyright (c) 2007 - 2015, Intel Corporation. All rights reserved.<BR>
# Copyright (c) 2007 - 2018, Intel Corporation. All rights reserved.<BR>
# Portions copyright (c) 2008 - 2010, Apple Inc. All rights reserved.<BR>
# This program and the accompanying materials
# are licensed and made available under the terms and conditions of the BSD License
@ -269,6 +269,7 @@
"$(SLINK)" $(SLINK_FLAGS) /OUT:${dst} @$(OBJECT_FILES_LIST)
<Command.GCC, Command.GCCLD>
$(RM) ${dst}
"$(SLINK)" cr ${dst} $(SLINK_FLAGS) @$(OBJECT_FILES_LIST)
<Command.RVCT>
@ -477,8 +478,8 @@
<Command.XCODE>
"$(ASLCC)" -o $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.obj $(ASLCC_FLAGS) $(INC) ${src}
"$(ASLDLINK)" -o $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.dll $(ASLDLINK_FLAGS) $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.obj
"$(MTOC)" -subsystem $(MODULE_TYPE) $(MTOC_FLAGS) $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.dll $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.efi
"$(GENFW)" -o ${dst} -c $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.efi $(GENFW_FLAGS)
"$(MTOC)" -subsystem $(MODULE_TYPE) $(MTOC_FLAGS) $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.dll $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.pecoff
"$(GENFW)" -o ${dst} -c $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.pecoff $(GENFW_FLAGS)
[Masm16-Code-File]

View File

@ -4253,7 +4253,7 @@ DEFINE GCC48_AARCH64_ASLDLINK_FLAGS = DEF(GCC47_AARCH64_ASLDLINK_FLAGS)
DEFINE GCC49_IA32_CC_FLAGS = DEF(GCC48_IA32_CC_FLAGS)
DEFINE GCC49_X64_CC_FLAGS = DEF(GCC48_X64_CC_FLAGS)
DEFINE GCC49_IA32_X64_DLINK_COMMON = -nostdlib -Wl,-n,-q,--gc-sections -z common-page-size=0x40
DEFINE GCC49_IA32_X64_ASLDLINK_FLAGS = DEF(GCC49_IA32_X64_DLINK_COMMON) -Wl,--entry,ReferenceAcpiTable -u ReferenceAcpiTable
DEFINE GCC49_IA32_X64_ASLDLINK_FLAGS = DEF(GCC49_IA32_X64_DLINK_COMMON) -Wl,--defsym=PECOFF_HEADER_SIZE=0 DEF(GCC_DLINK2_FLAGS_COMMON) -Wl,--entry,ReferenceAcpiTable -u ReferenceAcpiTable
DEFINE GCC49_IA32_X64_DLINK_FLAGS = DEF(GCC49_IA32_X64_DLINK_COMMON) -Wl,--entry,$(IMAGE_ENTRY_POINT) -u $(IMAGE_ENTRY_POINT) -Wl,-Map,$(DEST_DIR_DEBUG)/$(BASE_NAME).map,--whole-archive
DEFINE GCC49_IA32_DLINK2_FLAGS = DEF(GCC48_IA32_DLINK2_FLAGS)
DEFINE GCC49_X64_DLINK_FLAGS = DEF(GCC49_IA32_X64_DLINK_FLAGS) -Wl,-melf_x86_64,--oformat=elf64-x86-64,-pie

View File

@ -1,7 +1,7 @@
## @file
# Windows makefile for Base Tools project build.
#
# Copyright (c) 2007 - 2017, Intel Corporation. All rights reserved.<BR>
# Copyright (c) 2007 - 2018, Intel Corporation. All rights reserved.<BR>
# This program and the accompanying materials
# are licensed and made available under the terms and conditions of the BSD License
# which accompanies this distribution. The full text of the license may be found at
@ -17,22 +17,20 @@
SUBDIRS = $(BASE_TOOLS_PATH)\Source\C $(BASE_TOOLS_PATH)\Source\Python
all: c python
all: c
c :
@$(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.bat all $(BASE_TOOLS_PATH)\Source\C
@$(PYTHON_HOME)\python.exe $(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.py all $(BASE_TOOLS_PATH)\Source\C
python:
@$(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.bat all $(BASE_TOOLS_PATH)\Source\Python
subdirs: $(SUBDIRS)
@$(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.bat all $**
@$(PYTHON_HOME)\python.exe $(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.py all $**
.PHONY: clean
clean:
@$(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.bat clean $(SUBDIRS)
$(PYTHON_HOME)\python.exe $(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.py clean $(SUBDIRS)
.PHONY: cleanall
cleanall:
@$(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.bat cleanall $(SUBDIRS)
$(PYTHON_HOME)\python.exe $(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.py cleanall $(SUBDIRS)

View File

@ -16,7 +16,6 @@ In addition to this, you should set the following environment variables:
* EDK_TOOLS_PATH - Path to the BaseTools sub directory under the edk2 tree
* BASE_TOOLS_PATH - The directory where the BaseTools source is located.
(It is the same directory where this README.txt is located.)
* PYTHON_FREEZER_PATH - Path to where the python freezer tool is installed
After this, you can run the toolsetup.bat file, which is in the same
directory as this file. It should setup the remainder of the environment,

View File

@ -303,7 +303,10 @@ class Config(object):
list1 = [t for t in list1 if t != ''] # remove '' form list
first_num = int(list1[0], 16)
if list1[first_num + 1] == 'STRING': # parser STRING
value = 'L%s' % list1[-1]
if list1[-1] == '""':
value = "{0x0, 0x0}"
else:
value = 'L%s' % list1[-1]
elif list1[first_num + 1] == 'ORDERED_LIST': # parser ORDERED_LIST
value_total = int(list1[first_num + 2])
list2 = list1[-value_total:]
@ -505,12 +508,22 @@ class mainprocess(object):
inf_list = self.del_repeat(inf_list)
header_list = self.plus(self.del_repeat(header_list))
title_all=list(set(title_list))
info_list = self.del_repeat(info_list)
info_list = self.remove_bracket(self.del_repeat(info_list))
for i in range(len(info_list)-1,-1,-1):
if len(info_list[i]) == 0:
info_list.remove(info_list[i])
return keys,title_all,info_list,header_list,inf_list
def remove_bracket(self,List):
for i in List:
for j in i:
tmp = j.split("|")
if (('L"' in j) and ("[" in j)) or (tmp[1].strip() == '{0x0, 0x0}'):
tmp[0] = tmp[0][:tmp[0].index('[')]
List[List.index(i)][i.index(j)] = "|".join(tmp)
else:
List[List.index(i)][i.index(j)] = j
return List
def write_all(self):
title_flag=1

View File

@ -551,7 +551,7 @@ class CheckGitCommits:
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
Result = p.communicate()
return Result[0].decode('utf-8', 'ignore') if Result[0] and Result[0].find("fatal")!=0 else None
return Result[0].decode('utf-8', 'ignore') if Result[0] and Result[0].find(b"fatal")!=0 else None
class CheckOnePatchFile:
"""Performs a patch check for a single file.

View File

@ -52,7 +52,6 @@ if defined SRC_CONF @goto SetEnv
@if not defined EDK_TOOLS_PATH @echo EDK_TOOLS_PATH = Not Set
@if defined BASE_TOOLS_PATH @echo BASE_TOOLS_PATH = %BASE_TOOLS_PATH%
@if defined EDK_TOOLS_BIN @echo EDK_TOOLS_BIN = %EDK_TOOLS_BIN%
@if defined PYTHON_FREEZER_PATH @echo PYTHON_FREEZER_PATH = %PYTHON_FREEZER_PATH%
@if "%NT32PKG%"=="TRUE" (
@echo.
@echo NOTE: Please configure your build to use the following TOOL_CHAIN_TAG

View File

@ -15,20 +15,24 @@ MAKEROOT ?= ..
APPNAME = Brotli
OBJECTS = \
tools/bro.o \
tools/brotli.o \
common/dictionary.o \
common/transform.o \
dec/bit_reader.o \
dec/decode.o \
dec/huffman.o \
dec/state.o \
enc/backward_references.o \
enc/backward_references_hq.o \
enc/bit_cost.o \
enc/block_splitter.o \
enc/brotli_bit_stream.o \
enc/cluster.o \
enc/compress_fragment.o \
enc/compress_fragment_two_pass.o \
enc/dictionary_hash.o \
enc/encode.o \
enc/encoder_dict.o \
enc/entropy_encode.o \
enc/histogram.o \
enc/literal_cost.o \
@ -39,4 +43,5 @@ OBJECTS = \
include $(MAKEROOT)/Makefiles/app.makefile
TOOL_INCLUDE = -I ./include
LIBS += -lm

View File

@ -1,7 +1,7 @@
## @file
# Windows makefile for 'Brotli' module build.
#
# Copyright (c) 2017, Intel Corporation. All rights reserved.<BR>
# Copyright (c) 2017 - 2018, Intel Corporation. All rights reserved.<BR>
# This program and the accompanying materials
# are licensed and made available under the terms and conditions of the BSD License
# which accompanies this distribution. The full text of the license may be found at
@ -12,13 +12,14 @@
#
!INCLUDE ..\Makefiles\ms.common
INC = -I .\include $(INC)
CFLAGS = $(CFLAGS) /W2
APPNAME = Brotli
#LIBS = $(LIB_PATH)\Common.lib
COMMON_OBJ = common\dictionary.obj
COMMON_OBJ = common\dictionary.obj common\transform.obj
DEC_OBJ = \
dec\bit_reader.obj \
dec\decode.obj \
@ -26,13 +27,16 @@ DEC_OBJ = \
dec\state.obj
ENC_OBJ = \
enc\backward_references.obj \
enc\backward_references_hq.obj \
enc\bit_cost.obj \
enc\block_splitter.obj \
enc\brotli_bit_stream.obj \
enc\cluster.obj \
enc\compress_fragment.obj \
enc\compress_fragment_two_pass.obj \
enc\dictionary_hash.obj \
enc\encode.obj \
enc\encoder_dict.obj \
enc\entropy_encode.obj \
enc\histogram.obj \
enc\literal_cost.obj \
@ -42,19 +46,9 @@ ENC_OBJ = \
enc\utf8_util.obj
OBJECTS = \
tools\bro.obj \
tools\brotli.obj \
$(COMMON_OBJ) \
$(DEC_OBJ) \
$(ENC_OBJ)
!INCLUDE ..\Makefiles\ms.app
all: $(BIN_PATH)\BrotliCompress.bat
$(BIN_PATH)\BrotliCompress.bat: BrotliCompress.bat
copy BrotliCompress.bat $(BIN_PATH)\BrotliCompress.bat /Y
cleanall: localCleanall
localCleanall:
del /f /q $(BIN_PATH)\BrotliCompress.bat > nul

View File

@ -1,2 +1,2 @@
It is based on the Brotli v0.5.2.
It is based on the Brotli v1.0.6.
Brotli was released on the website https://github.com/google/brotli.

View File

@ -28,14 +28,25 @@
/* "code length of 8 is repeated" */
#define BROTLI_INITIAL_REPEATED_CODE_LENGTH 8
/* "Large Window Brotli" */
#define BROTLI_LARGE_MAX_DISTANCE_BITS 62U
#define BROTLI_LARGE_MIN_WBITS 10
#define BROTLI_LARGE_MAX_WBITS 30
/* Specification: 4. Encoding of distances */
#define BROTLI_NUM_DISTANCE_SHORT_CODES 16
#define BROTLI_MAX_NPOSTFIX 3
#define BROTLI_MAX_NDIRECT 120
/* BROTLI_NUM_DISTANCE_SYMBOLS == 520 */
#define BROTLI_NUM_DISTANCE_SYMBOLS (BROTLI_NUM_DISTANCE_SHORT_CODES + \
BROTLI_MAX_NDIRECT + \
(24 << (BROTLI_MAX_NPOSTFIX + 1)))
#define BROTLI_MAX_DISTANCE_BITS 24U
#define BROTLI_DISTANCE_ALPHABET_SIZE(NPOSTFIX, NDIRECT, MAXNBITS) ( \
BROTLI_NUM_DISTANCE_SHORT_CODES + (NDIRECT) + \
((MAXNBITS) << ((NPOSTFIX) + 1)))
/* BROTLI_NUM_DISTANCE_SYMBOLS == 1128 */
#define BROTLI_NUM_DISTANCE_SYMBOLS \
BROTLI_DISTANCE_ALPHABET_SIZE( \
BROTLI_MAX_NDIRECT, BROTLI_MAX_NPOSTFIX, BROTLI_LARGE_MAX_DISTANCE_BITS)
#define BROTLI_MAX_DISTANCE 0x3FFFFFC
#define BROTLI_MAX_ALLOWED_DISTANCE 0x7FFFFFFC
/* 7.1. Context modes and context ID lookup for literals */
/* "context IDs for literals are in the range of 0..63" */
@ -44,4 +55,10 @@
/* 7.2. Context ID for distances */
#define BROTLI_DISTANCE_CONTEXT_BITS 2
/* 9.1. Format of the Stream Header */
/* Number of slack bytes for window size. Don't confuse
with BROTLI_NUM_DISTANCE_SHORT_CODES. */
#define BROTLI_WINDOW_GAP 16
#define BROTLI_MAX_BACKWARD_LIMIT(W) (((size_t)1 << (W)) - BROTLI_WINDOW_GAP)
#endif /* BROTLI_COMMON_CONSTANTS_H_ */

View File

@ -6,110 +6,171 @@
/* Lookup table to map the previous two bytes to a context id.
There are four different context modeling modes defined here:
CONTEXT_LSB6: context id is the least significant 6 bits of the last byte,
CONTEXT_MSB6: context id is the most significant 6 bits of the last byte,
CONTEXT_UTF8: second-order context model tuned for UTF8-encoded text,
CONTEXT_SIGNED: second-order context model tuned for signed integers.
There are four different context modeling modes defined here:
CONTEXT_LSB6: context id is the least significant 6 bits of the last byte,
CONTEXT_MSB6: context id is the most significant 6 bits of the last byte,
CONTEXT_UTF8: second-order context model tuned for UTF8-encoded text,
CONTEXT_SIGNED: second-order context model tuned for signed integers.
The context id for the UTF8 context model is calculated as follows. If p1
and p2 are the previous two bytes, we calculate the context as
If |p1| and |p2| are the previous two bytes, and |mode| is current context
mode, we calculate the context as:
context = kContextLookup[p1] | kContextLookup[p2 + 256].
context = ContextLut(mode)[p1] | ContextLut(mode)[p2 + 256].
If the previous two bytes are ASCII characters (i.e. < 128), this will be
equivalent to
For CONTEXT_UTF8 mode, if the previous two bytes are ASCII characters
(i.e. < 128), this will be equivalent to
context = 4 * context1(p1) + context2(p2),
context = 4 * context1(p1) + context2(p2),
where context1 is based on the previous byte in the following way:
where context1 is based on the previous byte in the following way:
0 : non-ASCII control
1 : \t, \n, \r
2 : space
3 : other punctuation
4 : " '
5 : %
6 : ( < [ {
7 : ) > ] }
8 : , ; :
9 : .
10 : =
11 : number
12 : upper-case vowel
13 : upper-case consonant
14 : lower-case vowel
15 : lower-case consonant
0 : non-ASCII control
1 : \t, \n, \r
2 : space
3 : other punctuation
4 : " '
5 : %
6 : ( < [ {
7 : ) > ] }
8 : , ; :
9 : .
10 : =
11 : number
12 : upper-case vowel
13 : upper-case consonant
14 : lower-case vowel
15 : lower-case consonant
and context2 is based on the second last byte:
and context2 is based on the second last byte:
0 : control, space
1 : punctuation
2 : upper-case letter, number
3 : lower-case letter
0 : control, space
1 : punctuation
2 : upper-case letter, number
3 : lower-case letter
If the last byte is ASCII, and the second last byte is not (in a valid UTF8
stream it will be a continuation byte, value between 128 and 191), the
context is the same as if the second last byte was an ASCII control or space.
If the last byte is ASCII, and the second last byte is not (in a valid UTF8
stream it will be a continuation byte, value between 128 and 191), the
context is the same as if the second last byte was an ASCII control or space.
If the last byte is a UTF8 lead byte (value >= 192), then the next byte will
be a continuation byte and the context id is 2 or 3 depending on the LSB of
the last byte and to a lesser extent on the second last byte if it is ASCII.
If the last byte is a UTF8 lead byte (value >= 192), then the next byte will
be a continuation byte and the context id is 2 or 3 depending on the LSB of
the last byte and to a lesser extent on the second last byte if it is ASCII.
If the last byte is a UTF8 continuation byte, the second last byte can be:
- continuation byte: the next byte is probably ASCII or lead byte (assuming
4-byte UTF8 characters are rare) and the context id is 0 or 1.
- lead byte (192 - 207): next byte is ASCII or lead byte, context is 0 or 1
- lead byte (208 - 255): next byte is continuation byte, context is 2 or 3
If the last byte is a UTF8 continuation byte, the second last byte can be:
- continuation byte: the next byte is probably ASCII or lead byte (assuming
4-byte UTF8 characters are rare) and the context id is 0 or 1.
- lead byte (192 - 207): next byte is ASCII or lead byte, context is 0 or 1
- lead byte (208 - 255): next byte is continuation byte, context is 2 or 3
The possible value combinations of the previous two bytes, the range of
context ids and the type of the next byte is summarized in the table below:
The possible value combinations of the previous two bytes, the range of
context ids and the type of the next byte is summarized in the table below:
|--------\-----------------------------------------------------------------|
| \ Last byte |
| Second \---------------------------------------------------------------|
| last byte \ ASCII | cont. byte | lead byte |
| \ (0-127) | (128-191) | (192-) |
|=============|===================|=====================|==================|
| ASCII | next: ASCII/lead | not valid | next: cont. |
| (0-127) | context: 4 - 63 | | context: 2 - 3 |
|-------------|-------------------|---------------------|------------------|
| cont. byte | next: ASCII/lead | next: ASCII/lead | next: cont. |
| (128-191) | context: 4 - 63 | context: 0 - 1 | context: 2 - 3 |
|-------------|-------------------|---------------------|------------------|
| lead byte | not valid | next: ASCII/lead | not valid |
| (192-207) | | context: 0 - 1 | |
|-------------|-------------------|---------------------|------------------|
| lead byte | not valid | next: cont. | not valid |
| (208-) | | context: 2 - 3 | |
|-------------|-------------------|---------------------|------------------|
The context id for the signed context mode is calculated as:
context = (kContextLookup[512 + p1] << 3) | kContextLookup[512 + p2].
For any context modeling modes, the context ids can be calculated by |-ing
together two lookups from one table using context model dependent offsets:
context = kContextLookup[offset1 + p1] | kContextLookup[offset2 + p2].
where offset1 and offset2 are dependent on the context mode.
|--------\-----------------------------------------------------------------|
| \ Last byte |
| Second \---------------------------------------------------------------|
| last byte \ ASCII | cont. byte | lead byte |
| \ (0-127) | (128-191) | (192-) |
|=============|===================|=====================|==================|
| ASCII | next: ASCII/lead | not valid | next: cont. |
| (0-127) | context: 4 - 63 | | context: 2 - 3 |
|-------------|-------------------|---------------------|------------------|
| cont. byte | next: ASCII/lead | next: ASCII/lead | next: cont. |
| (128-191) | context: 4 - 63 | context: 0 - 1 | context: 2 - 3 |
|-------------|-------------------|---------------------|------------------|
| lead byte | not valid | next: ASCII/lead | not valid |
| (192-207) | | context: 0 - 1 | |
|-------------|-------------------|---------------------|------------------|
| lead byte | not valid | next: cont. | not valid |
| (208-) | | context: 2 - 3 | |
|-------------|-------------------|---------------------|------------------|
*/
#ifndef BROTLI_DEC_CONTEXT_H_
#define BROTLI_DEC_CONTEXT_H_
#ifndef BROTLI_COMMON_CONTEXT_H_
#define BROTLI_COMMON_CONTEXT_H_
#include "../common/types.h"
#include <brotli/types.h>
enum ContextType {
typedef enum ContextType {
CONTEXT_LSB6 = 0,
CONTEXT_MSB6 = 1,
CONTEXT_UTF8 = 2,
CONTEXT_SIGNED = 3
};
} ContextType;
/* Common context lookup table for all context modes. */
static const uint8_t kContextLookup[1792] = {
static const uint8_t kContextLookup[2048] = {
/* CONTEXT_LSB6, last byte. */
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,
16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31,
32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47,
48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63,
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,
16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31,
32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47,
48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63,
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,
16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31,
32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47,
48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63,
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,
16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31,
32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47,
48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63,
/* CONTEXT_LSB6, second last byte, */
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
/* CONTEXT_MSB6, last byte. */
0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3,
4, 4, 4, 4, 5, 5, 5, 5, 6, 6, 6, 6, 7, 7, 7, 7,
8, 8, 8, 8, 9, 9, 9, 9, 10, 10, 10, 10, 11, 11, 11, 11,
12, 12, 12, 12, 13, 13, 13, 13, 14, 14, 14, 14, 15, 15, 15, 15,
16, 16, 16, 16, 17, 17, 17, 17, 18, 18, 18, 18, 19, 19, 19, 19,
20, 20, 20, 20, 21, 21, 21, 21, 22, 22, 22, 22, 23, 23, 23, 23,
24, 24, 24, 24, 25, 25, 25, 25, 26, 26, 26, 26, 27, 27, 27, 27,
28, 28, 28, 28, 29, 29, 29, 29, 30, 30, 30, 30, 31, 31, 31, 31,
32, 32, 32, 32, 33, 33, 33, 33, 34, 34, 34, 34, 35, 35, 35, 35,
36, 36, 36, 36, 37, 37, 37, 37, 38, 38, 38, 38, 39, 39, 39, 39,
40, 40, 40, 40, 41, 41, 41, 41, 42, 42, 42, 42, 43, 43, 43, 43,
44, 44, 44, 44, 45, 45, 45, 45, 46, 46, 46, 46, 47, 47, 47, 47,
48, 48, 48, 48, 49, 49, 49, 49, 50, 50, 50, 50, 51, 51, 51, 51,
52, 52, 52, 52, 53, 53, 53, 53, 54, 54, 54, 54, 55, 55, 55, 55,
56, 56, 56, 56, 57, 57, 57, 57, 58, 58, 58, 58, 59, 59, 59, 59,
60, 60, 60, 60, 61, 61, 61, 61, 62, 62, 62, 62, 63, 63, 63, 63,
/* CONTEXT_MSB6, second last byte, */
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
/* CONTEXT_UTF8, last byte. */
/* ASCII range. */
0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 4, 0, 0, 4, 0, 0,
@ -130,6 +191,7 @@ static const uint8_t kContextLookup[1792] = {
2, 3, 2, 3, 2, 3, 2, 3, 2, 3, 2, 3, 2, 3, 2, 3,
2, 3, 2, 3, 2, 3, 2, 3, 2, 3, 2, 3, 2, 3, 2, 3,
2, 3, 2, 3, 2, 3, 2, 3, 2, 3, 2, 3, 2, 3, 2, 3,
/* CONTEXT_UTF8 second last byte. */
/* ASCII range. */
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
@ -150,23 +212,7 @@ static const uint8_t kContextLookup[1792] = {
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
/* CONTEXT_SIGNED, second last byte. */
0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5,
5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5,
5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5,
6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 7,
/* CONTEXT_SIGNED, last byte, same as the above values shifted by 3 bits. */
0, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8,
16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16,
@ -184,68 +230,32 @@ static const uint8_t kContextLookup[1792] = {
40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40,
40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40,
48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 56,
/* CONTEXT_LSB6, last byte. */
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,
16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31,
32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47,
48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63,
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,
16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31,
32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47,
48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63,
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,
16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31,
32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47,
48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63,
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,
16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31,
32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47,
48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63,
/* CONTEXT_MSB6, last byte. */
0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3,
4, 4, 4, 4, 5, 5, 5, 5, 6, 6, 6, 6, 7, 7, 7, 7,
8, 8, 8, 8, 9, 9, 9, 9, 10, 10, 10, 10, 11, 11, 11, 11,
12, 12, 12, 12, 13, 13, 13, 13, 14, 14, 14, 14, 15, 15, 15, 15,
16, 16, 16, 16, 17, 17, 17, 17, 18, 18, 18, 18, 19, 19, 19, 19,
20, 20, 20, 20, 21, 21, 21, 21, 22, 22, 22, 22, 23, 23, 23, 23,
24, 24, 24, 24, 25, 25, 25, 25, 26, 26, 26, 26, 27, 27, 27, 27,
28, 28, 28, 28, 29, 29, 29, 29, 30, 30, 30, 30, 31, 31, 31, 31,
32, 32, 32, 32, 33, 33, 33, 33, 34, 34, 34, 34, 35, 35, 35, 35,
36, 36, 36, 36, 37, 37, 37, 37, 38, 38, 38, 38, 39, 39, 39, 39,
40, 40, 40, 40, 41, 41, 41, 41, 42, 42, 42, 42, 43, 43, 43, 43,
44, 44, 44, 44, 45, 45, 45, 45, 46, 46, 46, 46, 47, 47, 47, 47,
48, 48, 48, 48, 49, 49, 49, 49, 50, 50, 50, 50, 51, 51, 51, 51,
52, 52, 52, 52, 53, 53, 53, 53, 54, 54, 54, 54, 55, 55, 55, 55,
56, 56, 56, 56, 57, 57, 57, 57, 58, 58, 58, 58, 59, 59, 59, 59,
60, 60, 60, 60, 61, 61, 61, 61, 62, 62, 62, 62, 63, 63, 63, 63,
/* CONTEXT_{M,L}SB6, second last byte, */
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
/* CONTEXT_SIGNED, second last byte. */
0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5,
5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5,
5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5,
6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 7,
};
static const int kContextLookupOffsets[8] = {
/* CONTEXT_LSB6 */
1024, 1536,
/* CONTEXT_MSB6 */
1280, 1536,
/* CONTEXT_UTF8 */
0, 256,
/* CONTEXT_SIGNED */
768, 512,
};
typedef const uint8_t* ContextLut;
#endif /* BROTLI_DEC_CONTEXT_H_ */
/* typeof(MODE) == ContextType; returns ContextLut */
#define BROTLI_CONTEXT_LUT(MODE) (&kContextLookup[(MODE) << 9])
/* typeof(LUT) == ContextLut */
#define BROTLI_CONTEXT(P1, P2, LUT) ((LUT)[P1] | ((LUT) + 256)[P2])
#endif /* BROTLI_COMMON_CONTEXT_H_ */

File diff suppressed because it is too large Load Diff

View File

@ -9,18 +9,53 @@
#ifndef BROTLI_COMMON_DICTIONARY_H_
#define BROTLI_COMMON_DICTIONARY_H_
#include "./types.h"
#include <brotli/port.h>
#include <brotli/types.h>
#if defined(__cplusplus) || defined(c_plusplus)
extern "C" {
#endif
extern const uint8_t kBrotliDictionary[122784];
extern const uint32_t kBrotliDictionaryOffsetsByLength[25];
extern const uint8_t kBrotliDictionarySizeBitsByLength[25];
typedef struct BrotliDictionary {
/**
* Number of bits to encode index of dictionary word in a bucket.
*
* Specification: Appendix A. Static Dictionary Data
*
* Words in a dictionary are bucketed by length.
* @c 0 means that there are no words of a given length.
* Dictionary consists of words with length of [4..24] bytes.
* Values at [0..3] and [25..31] indices should not be addressed.
*/
uint8_t size_bits_by_length[32];
#define kBrotliMinDictionaryWordLength 4
#define kBrotliMaxDictionaryWordLength 24
/* assert(offset[i + 1] == offset[i] + (bits[i] ? (i << bits[i]) : 0)) */
uint32_t offsets_by_length[32];
/* assert(data_size == offsets_by_length[31]) */
size_t data_size;
/* Data array is not bound, and should obey to size_bits_by_length values.
Specified size matches default (RFC 7932) dictionary. Its size is
defined by data_size */
const uint8_t* data;
} BrotliDictionary;
BROTLI_COMMON_API const BrotliDictionary* BrotliGetDictionary(void);
/**
* Sets dictionary data.
*
* When dictionary data is already set / present, this method is no-op.
*
* Dictionary data MUST be provided before BrotliGetDictionary is invoked.
* This method is used ONLY in multi-client environment (e.g. C + Java),
* to reduce storage by sharing single dictionary between implementations.
*/
BROTLI_COMMON_API void BrotliSetDictionaryData(const uint8_t* data);
#define BROTLI_MIN_DICTIONARY_WORD_LENGTH 4
#define BROTLI_MAX_DICTIONARY_WORD_LENGTH 24
#if defined(__cplusplus) || defined(c_plusplus)
} /* extern "C" */

View File

@ -0,0 +1,558 @@
/* Copyright 2016 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* Macros for compiler / platform specific features and build options.
Build options are:
* BROTLI_BUILD_32_BIT disables 64-bit optimizations
* BROTLI_BUILD_64_BIT forces to use 64-bit optimizations
* BROTLI_BUILD_BIG_ENDIAN forces to use big-endian optimizations
* BROTLI_BUILD_ENDIAN_NEUTRAL disables endian-aware optimizations
* BROTLI_BUILD_LITTLE_ENDIAN forces to use little-endian optimizations
* BROTLI_BUILD_PORTABLE disables dangerous optimizations, like unaligned
read and overlapping memcpy; this reduces decompression speed by 5%
* BROTLI_BUILD_NO_RBIT disables "rbit" optimization for ARM CPUs
* BROTLI_DEBUG dumps file name and line number when decoder detects stream
or memory error
* BROTLI_ENABLE_LOG enables asserts and dumps various state information
*/
#ifndef BROTLI_COMMON_PLATFORM_H_
#define BROTLI_COMMON_PLATFORM_H_
#include <string.h> /* memcpy */
#include <stdlib.h> /* malloc, free */
#include <brotli/port.h>
#include <brotli/types.h>
#if defined(OS_LINUX) || defined(OS_CYGWIN)
#include <endian.h>
#elif defined(OS_FREEBSD)
#include <machine/endian.h>
#elif defined(OS_MACOSX)
#include <machine/endian.h>
/* Let's try and follow the Linux convention */
#define BROTLI_X_BYTE_ORDER BYTE_ORDER
#define BROTLI_X_LITTLE_ENDIAN LITTLE_ENDIAN
#define BROTLI_X_BIG_ENDIAN BIG_ENDIAN
#endif
#if defined(BROTLI_ENABLE_LOG) || defined(BROTLI_DEBUG)
#include <assert.h>
#include <stdio.h>
#endif
/* The following macros were borrowed from https://github.com/nemequ/hedley
* with permission of original author - Evan Nemerson <evan@nemerson.com> */
/* >>> >>> >>> hedley macros */
/* Define "BROTLI_PREDICT_TRUE" and "BROTLI_PREDICT_FALSE" macros for capable
compilers.
To apply compiler hint, enclose the branching condition into macros, like this:
if (BROTLI_PREDICT_TRUE(zero == 0)) {
// main execution path
} else {
// compiler should place this code outside of main execution path
}
OR:
if (BROTLI_PREDICT_FALSE(something_rare_or_unexpected_happens)) {
// compiler should place this code outside of main execution path
}
*/
#if BROTLI_GNUC_HAS_BUILTIN(__builtin_expect, 3, 0, 0) || \
BROTLI_INTEL_VERSION_CHECK(16, 0, 0) || \
BROTLI_SUNPRO_VERSION_CHECK(5, 12, 0) || \
BROTLI_ARM_VERSION_CHECK(4, 1, 0) || \
BROTLI_IBM_VERSION_CHECK(10, 1, 0) || \
BROTLI_TI_VERSION_CHECK(7, 3, 0) || \
BROTLI_TINYC_VERSION_CHECK(0, 9, 27)
#define BROTLI_PREDICT_TRUE(x) (__builtin_expect(!!(x), 1))
#define BROTLI_PREDICT_FALSE(x) (__builtin_expect(x, 0))
#else
#define BROTLI_PREDICT_FALSE(x) (x)
#define BROTLI_PREDICT_TRUE(x) (x)
#endif
#if defined(__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) && \
!defined(__cplusplus)
#define BROTLI_RESTRICT restrict
#elif BROTLI_GNUC_VERSION_CHECK(3, 1, 0) || \
BROTLI_MSVC_VERSION_CHECK(14, 0, 0) || \
BROTLI_INTEL_VERSION_CHECK(16, 0, 0) || \
BROTLI_ARM_VERSION_CHECK(4, 1, 0) || \
BROTLI_IBM_VERSION_CHECK(10, 1, 0) || \
BROTLI_PGI_VERSION_CHECK(17, 10, 0) || \
BROTLI_TI_VERSION_CHECK(8, 0, 0) || \
BROTLI_IAR_VERSION_CHECK(8, 0, 0) || \
(BROTLI_SUNPRO_VERSION_CHECK(5, 14, 0) && defined(__cplusplus))
#define BROTLI_RESTRICT __restrict
#elif BROTLI_SUNPRO_VERSION_CHECK(5, 3, 0) && !defined(__cplusplus)
#define BROTLI_RESTRICT _Restrict
#else
#define BROTLI_RESTRICT
#endif
#if (defined(__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L)) || \
(defined(__cplusplus) && (__cplusplus >= 199711L))
#define BROTLI_MAYBE_INLINE inline
#elif defined(__GNUC_STDC_INLINE__) || defined(__GNUC_GNU_INLINE__) || \
BROTLI_ARM_VERSION_CHECK(6, 2, 0)
#define BROTLI_MAYBE_INLINE __inline__
#elif BROTLI_MSVC_VERSION_CHECK(12, 0, 0) || \
BROTLI_ARM_VERSION_CHECK(4, 1, 0) || BROTLI_TI_VERSION_CHECK(8, 0, 0)
#define BROTLI_MAYBE_INLINE __inline
#else
#define BROTLI_MAYBE_INLINE
#endif
#if BROTLI_GNUC_HAS_ATTRIBUTE(always_inline, 4, 0, 0) || \
BROTLI_INTEL_VERSION_CHECK(16, 0, 0) || \
BROTLI_SUNPRO_VERSION_CHECK(5, 11, 0) || \
BROTLI_ARM_VERSION_CHECK(4, 1, 0) || \
BROTLI_IBM_VERSION_CHECK(10, 1, 0) || \
BROTLI_TI_VERSION_CHECK(8, 0, 0) || \
(BROTLI_TI_VERSION_CHECK(7, 3, 0) && defined(__TI_GNU_ATTRIBUTE_SUPPORT__))
#define BROTLI_INLINE BROTLI_MAYBE_INLINE __attribute__((__always_inline__))
#elif BROTLI_MSVC_VERSION_CHECK(12, 0, 0)
#define BROTLI_INLINE BROTLI_MAYBE_INLINE __forceinline
#elif BROTLI_TI_VERSION_CHECK(7, 0, 0) && defined(__cplusplus)
#define BROTLI_INLINE BROTLI_MAYBE_INLINE _Pragma("FUNC_ALWAYS_INLINE;")
#elif BROTLI_IAR_VERSION_CHECK(8, 0, 0)
#define BROTLI_INLINE BROTLI_MAYBE_INLINE _Pragma("inline=forced")
#else
#define BROTLI_INLINE BROTLI_MAYBE_INLINE
#endif
#if BROTLI_GNUC_HAS_ATTRIBUTE(noinline, 4, 0, 0) || \
BROTLI_INTEL_VERSION_CHECK(16, 0, 0) || \
BROTLI_SUNPRO_VERSION_CHECK(5, 11, 0) || \
BROTLI_ARM_VERSION_CHECK(4, 1, 0) || \
BROTLI_IBM_VERSION_CHECK(10, 1, 0) || \
BROTLI_TI_VERSION_CHECK(8, 0, 0) || \
(BROTLI_TI_VERSION_CHECK(7, 3, 0) && defined(__TI_GNU_ATTRIBUTE_SUPPORT__))
#define BROTLI_NOINLINE __attribute__((__noinline__))
#elif BROTLI_MSVC_VERSION_CHECK(13, 10, 0)
#define BROTLI_NOINLINE __declspec(noinline)
#elif BROTLI_PGI_VERSION_CHECK(10, 2, 0)
#define BROTLI_NOINLINE _Pragma("noinline")
#elif BROTLI_TI_VERSION_CHECK(6, 0, 0) && defined(__cplusplus)
#define BROTLI_NOINLINE _Pragma("FUNC_CANNOT_INLINE;")
#elif BROTLI_IAR_VERSION_CHECK(8, 0, 0)
#define BROTLI_NOINLINE _Pragma("inline=never")
#else
#define BROTLI_NOINLINE
#endif
/* BROTLI_INTERNAL could be defined to override visibility, e.g. for tests. */
#if !defined(BROTLI_INTERNAL)
#if defined(_WIN32) || defined(__CYGWIN__)
#define BROTLI_INTERNAL
#elif BROTLI_GNUC_VERSION_CHECK(3, 3, 0) || \
BROTLI_TI_VERSION_CHECK(8, 0, 0) || \
BROTLI_INTEL_VERSION_CHECK(16, 0, 0) || \
BROTLI_ARM_VERSION_CHECK(4, 1, 0) || \
BROTLI_IBM_VERSION_CHECK(13, 1, 0) || \
BROTLI_SUNPRO_VERSION_CHECK(5, 11, 0) || \
(BROTLI_TI_VERSION_CHECK(7, 3, 0) && \
defined(__TI_GNU_ATTRIBUTE_SUPPORT__) && defined(__TI_EABI__))
#define BROTLI_INTERNAL __attribute__ ((visibility ("hidden")))
#else
#define BROTLI_INTERNAL
#endif
#endif
/* <<< <<< <<< end of hedley macros. */
#if BROTLI_GNUC_HAS_ATTRIBUTE(unused, 2, 7, 0) || \
BROTLI_INTEL_VERSION_CHECK(16, 0, 0)
#define BROTLI_UNUSED_FUNCTION static BROTLI_INLINE __attribute__ ((unused))
#else
#define BROTLI_UNUSED_FUNCTION static BROTLI_INLINE
#endif
#if (defined(__ARM_ARCH) && (__ARM_ARCH == 7)) || \
(defined(M_ARM) && (M_ARM == 7))
#define BROTLI_TARGET_ARMV7
#endif /* ARMv7 */
#if (defined(__ARM_ARCH) && (__ARM_ARCH == 8)) || \
defined(__aarch64__) || defined(__ARM64_ARCH_8__)
#define BROTLI_TARGET_ARMV8_ANY
#if defined(__ARM_32BIT_STATE)
#define BROTLI_TARGET_ARMV8_32
#elif defined(__ARM_64BIT_STATE)
#define BROTLI_TARGET_ARMV8_64
#endif
#endif /* ARMv8 */
#if defined(__i386) || defined(_M_IX86)
#define BROTLI_TARGET_X86
#endif
#if defined(__x86_64__) || defined(_M_X64)
#define BROTLI_TARGET_X64
#endif
#if defined(__PPC64__)
#define BROTLI_TARGET_POWERPC64
#endif
#if defined(__riscv) && defined(__riscv_xlen) && __riscv_xlen == 64
#define BROTLI_TARGET_RISCV64
#endif
#if defined(BROTLI_BUILD_64_BIT)
#define BROTLI_64_BITS 1
#elif defined(BROTLI_BUILD_32_BIT)
#define BROTLI_64_BITS 0
#elif defined(BROTLI_TARGET_X64) || defined(BROTLI_TARGET_ARMV8_64) || \
defined(BROTLI_TARGET_POWERPC64) || defined(BROTLI_TARGET_RISCV64)
#define BROTLI_64_BITS 1
#else
#define BROTLI_64_BITS 0
#endif
#if (BROTLI_64_BITS)
#define brotli_reg_t uint64_t
#else
#define brotli_reg_t uint32_t
#endif
#if defined(BROTLI_BUILD_BIG_ENDIAN)
#define BROTLI_BIG_ENDIAN 1
#elif defined(BROTLI_BUILD_LITTLE_ENDIAN)
#define BROTLI_LITTLE_ENDIAN 1
#elif defined(BROTLI_BUILD_ENDIAN_NEUTRAL)
/* Just break elif chain. */
#elif defined(__BYTE_ORDER__) && (__BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__)
#define BROTLI_LITTLE_ENDIAN 1
#elif defined(_WIN32) || defined(BROTLI_TARGET_X64)
/* Win32 & x64 can currently always be assumed to be little endian */
#define BROTLI_LITTLE_ENDIAN 1
#elif defined(__BYTE_ORDER__) && (__BYTE_ORDER__ == __ORDER_BIG_ENDIAN__)
#define BROTLI_BIG_ENDIAN 1
#elif defined(BROTLI_X_BYTE_ORDER)
#if BROTLI_X_BYTE_ORDER == BROTLI_X_LITTLE_ENDIAN
#define BROTLI_LITTLE_ENDIAN 1
#elif BROTLI_X_BYTE_ORDER == BROTLI_X_BIG_ENDIAN
#define BROTLI_BIG_ENDIAN 1
#endif
#endif /* BROTLI_X_BYTE_ORDER */
#if !defined(BROTLI_LITTLE_ENDIAN)
#define BROTLI_LITTLE_ENDIAN 0
#endif
#if !defined(BROTLI_BIG_ENDIAN)
#define BROTLI_BIG_ENDIAN 0
#endif
#if defined(BROTLI_X_BYTE_ORDER)
#undef BROTLI_X_BYTE_ORDER
#undef BROTLI_X_LITTLE_ENDIAN
#undef BROTLI_X_BIG_ENDIAN
#endif
#if defined(BROTLI_BUILD_PORTABLE)
#define BROTLI_ALIGNED_READ (!!1)
#elif defined(BROTLI_TARGET_X86) || defined(BROTLI_TARGET_X64) || \
defined(BROTLI_TARGET_ARMV7) || defined(BROTLI_TARGET_ARMV8_ANY) || \
defined(BROTLI_TARGET_RISCV64)
/* Allow unaligned read only for white-listed CPUs. */
#define BROTLI_ALIGNED_READ (!!0)
#else
#define BROTLI_ALIGNED_READ (!!1)
#endif
#if BROTLI_ALIGNED_READ
/* Portable unaligned memory access: read / write values via memcpy. */
static BROTLI_INLINE uint16_t BrotliUnalignedRead16(const void* p) {
uint16_t t;
memcpy(&t, p, sizeof t);
return t;
}
static BROTLI_INLINE uint32_t BrotliUnalignedRead32(const void* p) {
uint32_t t;
memcpy(&t, p, sizeof t);
return t;
}
static BROTLI_INLINE uint64_t BrotliUnalignedRead64(const void* p) {
uint64_t t;
memcpy(&t, p, sizeof t);
return t;
}
static BROTLI_INLINE void BrotliUnalignedWrite64(void* p, uint64_t v) {
memcpy(p, &v, sizeof v);
}
#else /* BROTLI_ALIGNED_READ */
/* Unaligned memory access is allowed: just cast pointer to requested type. */
#if defined(ADDRESS_SANITIZER) || defined(THREAD_SANITIZER) || \
defined(MEMORY_SANITIZER)
/* Consider we have an unaligned load/store of 4 bytes from address 0x...05.
AddressSanitizer will treat it as a 3-byte access to the range 05:07 and
will miss a bug if 08 is the first unaddressable byte.
ThreadSanitizer will also treat this as a 3-byte access to 05:07 and will
miss a race between this access and some other accesses to 08.
MemorySanitizer will correctly propagate the shadow on unaligned stores
and correctly report bugs on unaligned loads, but it may not properly
update and report the origin of the uninitialized memory.
For all three tools, replacing an unaligned access with a tool-specific
callback solves the problem. */
#if defined(__cplusplus)
extern "C" {
#endif /* __cplusplus */
uint16_t __sanitizer_unaligned_load16(const void* p);
uint32_t __sanitizer_unaligned_load32(const void* p);
uint64_t __sanitizer_unaligned_load64(const void* p);
void __sanitizer_unaligned_store64(void* p, uint64_t v);
#if defined(__cplusplus)
} /* extern "C" */
#endif /* __cplusplus */
#define BrotliUnalignedRead16 __sanitizer_unaligned_load16
#define BrotliUnalignedRead32 __sanitizer_unaligned_load32
#define BrotliUnalignedRead64 __sanitizer_unaligned_load64
#define BrotliUnalignedWrite64 __sanitizer_unaligned_store64
#else
static BROTLI_INLINE uint16_t BrotliUnalignedRead16(const void* p) {
return *(const uint16_t*)p;
}
static BROTLI_INLINE uint32_t BrotliUnalignedRead32(const void* p) {
return *(const uint32_t*)p;
}
#if (BROTLI_64_BITS)
static BROTLI_INLINE uint64_t BrotliUnalignedRead64(const void* p) {
return *(const uint64_t*)p;
}
static BROTLI_INLINE void BrotliUnalignedWrite64(void* p, uint64_t v) {
*(uint64_t*)p = v;
}
#else /* BROTLI_64_BITS */
/* Avoid emitting LDRD / STRD, which require properly aligned address. */
/* If __attribute__(aligned) is available, use that. Otherwise, memcpy. */
#if BROTLI_GNUC_HAS_ATTRIBUTE(aligned, 2, 7, 0)
typedef __attribute__((aligned(1))) uint64_t brotli_unaligned_uint64_t;
static BROTLI_INLINE uint64_t BrotliUnalignedRead64(const void* p) {
return (uint64_t) ((brotli_unaligned_uint64_t*) p)[0];
}
static BROTLI_INLINE void BrotliUnalignedWrite64(void* p, uint64_t v) {
brotli_unaligned_uint64_t* dwords = (brotli_unaligned_uint64_t*) p;
dwords[0] = (brotli_unaligned_uint64_t) v;
}
#else /* BROTLI_GNUC_HAS_ATTRIBUTE(aligned, 2, 7, 0) */
static BROTLI_INLINE uint64_t BrotliUnalignedRead64(const void* p) {
uint64_t v;
memcpy(&v, p, sizeof(uint64_t));
return v;
}
static BROTLI_INLINE void BrotliUnalignedWrite64(void* p, uint64_t v) {
memcpy(p, &v, sizeof(uint64_t));
}
#endif /* BROTLI_GNUC_HAS_ATTRIBUTE(aligned, 2, 7, 0) */
#endif /* BROTLI_64_BITS */
#endif /* ASAN / TSAN / MSAN */
#endif /* BROTLI_ALIGNED_READ */
#if BROTLI_LITTLE_ENDIAN
/* Straight endianness. Just read / write values. */
#define BROTLI_UNALIGNED_LOAD16LE BrotliUnalignedRead16
#define BROTLI_UNALIGNED_LOAD32LE BrotliUnalignedRead32
#define BROTLI_UNALIGNED_LOAD64LE BrotliUnalignedRead64
#define BROTLI_UNALIGNED_STORE64LE BrotliUnalignedWrite64
#elif BROTLI_BIG_ENDIAN /* BROTLI_LITTLE_ENDIAN */
/* Explain compiler to byte-swap values. */
#define BROTLI_BSWAP16_(V) ((uint16_t)( \
(((V) & 0xFFU) << 8) | \
(((V) >> 8) & 0xFFU)))
static BROTLI_INLINE uint16_t BROTLI_UNALIGNED_LOAD16LE(const void* p) {
uint16_t value = BrotliUnalignedRead16(p);
return BROTLI_BSWAP16_(value);
}
#define BROTLI_BSWAP32_(V) ( \
(((V) & 0xFFU) << 24) | (((V) & 0xFF00U) << 8) | \
(((V) >> 8) & 0xFF00U) | (((V) >> 24) & 0xFFU))
static BROTLI_INLINE uint32_t BROTLI_UNALIGNED_LOAD32LE(const void* p) {
uint32_t value = BrotliUnalignedRead32(p);
return BROTLI_BSWAP32_(value);
}
#define BROTLI_BSWAP64_(V) ( \
(((V) & 0xFFU) << 56) | (((V) & 0xFF00U) << 40) | \
(((V) & 0xFF0000U) << 24) | (((V) & 0xFF000000U) << 8) | \
(((V) >> 8) & 0xFF000000U) | (((V) >> 24) & 0xFF0000U) | \
(((V) >> 40) & 0xFF00U) | (((V) >> 56) & 0xFFU))
static BROTLI_INLINE uint64_t BROTLI_UNALIGNED_LOAD64LE(const void* p) {
uint64_t value = BrotliUnalignedRead64(p);
return BROTLI_BSWAP64_(value);
}
static BROTLI_INLINE void BROTLI_UNALIGNED_STORE64LE(void* p, uint64_t v) {
uint64_t value = BROTLI_BSWAP64_(v);
BrotliUnalignedWrite64(p, value);
}
#else /* BROTLI_LITTLE_ENDIAN */
/* Read / store values byte-wise; hopefully compiler will understand. */
static BROTLI_INLINE uint16_t BROTLI_UNALIGNED_LOAD16LE(const void* p) {
const uint8_t* in = (const uint8_t*)p;
return (uint16_t)(in[0] | (in[1] << 8));
}
static BROTLI_INLINE uint32_t BROTLI_UNALIGNED_LOAD32LE(const void* p) {
const uint8_t* in = (const uint8_t*)p;
uint32_t value = (uint32_t)(in[0]);
value |= (uint32_t)(in[1]) << 8;
value |= (uint32_t)(in[2]) << 16;
value |= (uint32_t)(in[3]) << 24;
return value;
}
static BROTLI_INLINE uint64_t BROTLI_UNALIGNED_LOAD64LE(const void* p) {
const uint8_t* in = (const uint8_t*)p;
uint64_t value = (uint64_t)(in[0]);
value |= (uint64_t)(in[1]) << 8;
value |= (uint64_t)(in[2]) << 16;
value |= (uint64_t)(in[3]) << 24;
value |= (uint64_t)(in[4]) << 32;
value |= (uint64_t)(in[5]) << 40;
value |= (uint64_t)(in[6]) << 48;
value |= (uint64_t)(in[7]) << 56;
return value;
}
static BROTLI_INLINE void BROTLI_UNALIGNED_STORE64LE(void* p, uint64_t v) {
uint8_t* out = (uint8_t*)p;
out[0] = (uint8_t)v;
out[1] = (uint8_t)(v >> 8);
out[2] = (uint8_t)(v >> 16);
out[3] = (uint8_t)(v >> 24);
out[4] = (uint8_t)(v >> 32);
out[5] = (uint8_t)(v >> 40);
out[6] = (uint8_t)(v >> 48);
out[7] = (uint8_t)(v >> 56);
}
#endif /* BROTLI_LITTLE_ENDIAN */
/* BROTLI_IS_CONSTANT macros returns true for compile-time constants. */
#if BROTLI_GNUC_HAS_BUILTIN(__builtin_constant_p, 3, 0, 1) || \
BROTLI_INTEL_VERSION_CHECK(16, 0, 0)
#define BROTLI_IS_CONSTANT(x) (!!__builtin_constant_p(x))
#else
#define BROTLI_IS_CONSTANT(x) (!!0)
#endif
#if defined(BROTLI_TARGET_ARMV7) || defined(BROTLI_TARGET_ARMV8_ANY)
#define BROTLI_HAS_UBFX (!!1)
#else
#define BROTLI_HAS_UBFX (!!0)
#endif
#if defined(BROTLI_ENABLE_LOG)
#define BROTLI_DCHECK(x) assert(x)
#define BROTLI_LOG(x) printf x
#else
#define BROTLI_DCHECK(x)
#define BROTLI_LOG(x)
#endif
#if defined(BROTLI_DEBUG) || defined(BROTLI_ENABLE_LOG)
static BROTLI_INLINE void BrotliDump(const char* f, int l, const char* fn) {
fprintf(stderr, "%s:%d (%s)\n", f, l, fn);
fflush(stderr);
}
#define BROTLI_DUMP() BrotliDump(__FILE__, __LINE__, __FUNCTION__)
#else
#define BROTLI_DUMP() (void)(0)
#endif
/* TODO: add appropriate icc/sunpro/arm/ibm/ti checks. */
#if (BROTLI_GNUC_VERSION_CHECK(3, 0, 0) || defined(__llvm__)) && \
!defined(BROTLI_BUILD_NO_RBIT)
#if defined(BROTLI_TARGET_ARMV7) || defined(BROTLI_TARGET_ARMV8_ANY)
/* TODO: detect ARMv6T2 and enable this code for it. */
static BROTLI_INLINE brotli_reg_t BrotliRBit(brotli_reg_t input) {
brotli_reg_t output;
__asm__("rbit %0, %1\n" : "=r"(output) : "r"(input));
return output;
}
#define BROTLI_RBIT(x) BrotliRBit(x)
#endif /* armv7 / armv8 */
#endif /* gcc || clang */
#if !defined(BROTLI_RBIT)
static BROTLI_INLINE void BrotliRBit(void) { /* Should break build if used. */ }
#endif /* BROTLI_RBIT */
#define BROTLI_REPEAT(N, X) { \
if ((N & 1) != 0) {X;} \
if ((N & 2) != 0) {X; X;} \
if ((N & 4) != 0) {X; X; X; X;} \
}
#define BROTLI_UNUSED(X) (void)(X)
#define BROTLI_MIN_MAX(T) \
static BROTLI_INLINE T brotli_min_ ## T (T a, T b) { return a < b ? a : b; } \
static BROTLI_INLINE T brotli_max_ ## T (T a, T b) { return a > b ? a : b; }
BROTLI_MIN_MAX(double) BROTLI_MIN_MAX(float) BROTLI_MIN_MAX(int)
BROTLI_MIN_MAX(size_t) BROTLI_MIN_MAX(uint32_t) BROTLI_MIN_MAX(uint8_t)
#undef BROTLI_MIN_MAX
#define BROTLI_MIN(T, A, B) (brotli_min_ ## T((A), (B)))
#define BROTLI_MAX(T, A, B) (brotli_max_ ## T((A), (B)))
#define BROTLI_SWAP(T, A, I, J) { \
T __brotli_swap_tmp = (A)[(I)]; \
(A)[(I)] = (A)[(J)]; \
(A)[(J)] = __brotli_swap_tmp; \
}
/* Default brotli_alloc_func */
static void* BrotliDefaultAllocFunc(void* opaque, size_t size) {
BROTLI_UNUSED(opaque);
return malloc(size);
}
/* Default brotli_free_func */
static void BrotliDefaultFreeFunc(void* opaque, void* address) {
BROTLI_UNUSED(opaque);
free(address);
}
BROTLI_UNUSED_FUNCTION void BrotliSuppressUnusedFunctions(void) {
BROTLI_UNUSED(&BrotliSuppressUnusedFunctions);
BROTLI_UNUSED(&BrotliUnalignedRead16);
BROTLI_UNUSED(&BrotliUnalignedRead32);
BROTLI_UNUSED(&BrotliUnalignedRead64);
BROTLI_UNUSED(&BrotliUnalignedWrite64);
BROTLI_UNUSED(&BROTLI_UNALIGNED_LOAD16LE);
BROTLI_UNUSED(&BROTLI_UNALIGNED_LOAD32LE);
BROTLI_UNUSED(&BROTLI_UNALIGNED_LOAD64LE);
BROTLI_UNUSED(&BROTLI_UNALIGNED_STORE64LE);
BROTLI_UNUSED(&BrotliRBit);
BROTLI_UNUSED(&brotli_min_double);
BROTLI_UNUSED(&brotli_max_double);
BROTLI_UNUSED(&brotli_min_float);
BROTLI_UNUSED(&brotli_max_float);
BROTLI_UNUSED(&brotli_min_int);
BROTLI_UNUSED(&brotli_max_int);
BROTLI_UNUSED(&brotli_min_size_t);
BROTLI_UNUSED(&brotli_max_size_t);
BROTLI_UNUSED(&brotli_min_uint32_t);
BROTLI_UNUSED(&brotli_max_uint32_t);
BROTLI_UNUSED(&brotli_min_uint8_t);
BROTLI_UNUSED(&brotli_max_uint8_t);
BROTLI_UNUSED(&BrotliDefaultAllocFunc);
BROTLI_UNUSED(&BrotliDefaultFreeFunc);
#if defined(BROTLI_DEBUG) || defined(BROTLI_ENABLE_LOG)
BROTLI_UNUSED(&BrotliDump);
#endif
}
#endif /* BROTLI_COMMON_PLATFORM_H_ */

View File

@ -1,107 +0,0 @@
/* Copyright 2016 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* Macros for compiler / platform specific features and build options. */
#ifndef BROTLI_COMMON_PORT_H_
#define BROTLI_COMMON_PORT_H_
/* Compatibility with non-clang compilers. */
#ifndef __has_builtin
#define __has_builtin(x) 0
#endif
#ifndef __has_attribute
#define __has_attribute(x) 0
#endif
#ifndef __has_feature
#define __has_feature(x) 0
#endif
#if defined(__GNUC__) && defined(__GNUC_MINOR__)
#define BROTLI_GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__)
#else
#define BROTLI_GCC_VERSION 0
#endif
#if defined(__ICC)
#define BROTLI_ICC_VERSION __ICC
#else
#define BROTLI_ICC_VERSION 0
#endif
#if defined(BROTLI_BUILD_MODERN_COMPILER)
#define BROTLI_MODERN_COMPILER 1
#elif BROTLI_GCC_VERSION > 300 || BROTLI_ICC_VERSION >= 1600
#define BROTLI_MODERN_COMPILER 1
#else
#define BROTLI_MODERN_COMPILER 0
#endif
/* Define "PREDICT_TRUE" and "PREDICT_FALSE" macros for capable compilers.
To apply compiler hint, enclose the branching condition into macros, like this:
if (PREDICT_TRUE(zero == 0)) {
// main execution path
} else {
// compiler should place this code outside of main execution path
}
OR:
if (PREDICT_FALSE(something_rare_or_unexpected_happens)) {
// compiler should place this code outside of main execution path
}
*/
#if BROTLI_MODERN_COMPILER || __has_builtin(__builtin_expect)
#define PREDICT_TRUE(x) (__builtin_expect(!!(x), 1))
#define PREDICT_FALSE(x) (__builtin_expect(x, 0))
#else
#define PREDICT_FALSE(x) (x)
#define PREDICT_TRUE(x) (x)
#endif
#if BROTLI_MODERN_COMPILER || __has_attribute(always_inline)
#define ATTRIBUTE_ALWAYS_INLINE __attribute__ ((always_inline))
#else
#define ATTRIBUTE_ALWAYS_INLINE
#endif
#if defined(_WIN32) || defined(__CYGWIN__)
#define ATTRIBUTE_VISIBILITY_HIDDEN
#elif BROTLI_MODERN_COMPILER || __has_attribute(visibility)
#define ATTRIBUTE_VISIBILITY_HIDDEN __attribute__ ((visibility ("hidden")))
#else
#define ATTRIBUTE_VISIBILITY_HIDDEN
#endif
#ifndef BROTLI_INTERNAL
#define BROTLI_INTERNAL ATTRIBUTE_VISIBILITY_HIDDEN
#endif
#ifndef _MSC_VER
#if defined(__cplusplus) || !defined(__STRICT_ANSI__) || \
__STDC_VERSION__ >= 199901L
#define BROTLI_INLINE inline ATTRIBUTE_ALWAYS_INLINE
#else
#define BROTLI_INLINE
#endif
#else /* _MSC_VER */
#define BROTLI_INLINE __forceinline
#endif /* _MSC_VER */
#if BROTLI_MODERN_COMPILER || __has_attribute(noinline)
#define BROTLI_NOINLINE __attribute__((noinline))
#else
#define BROTLI_NOINLINE
#endif
#define BROTLI_UNUSED(X) (void)(X)
#endif /* BROTLI_COMMON_PORT_H_ */

View File

@ -0,0 +1,235 @@
/* Copyright 2013 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
#include "./transform.h"
#if defined(__cplusplus) || defined(c_plusplus)
extern "C" {
#endif
/* RFC 7932 transforms string data */
static const char kPrefixSuffix[217] =
"\1 \2, \10 of the \4 of \2s \1.\5 and \4 "
/* 0x _0 _2 __5 _E _3 _6 _8 _E */
"in \1\"\4 to \2\">\1\n\2. \1]\5 for \3 a \6 "
/* 2x _3_ _5 _A_ _D_ _F _2 _4 _A _E */
"that \1\'\6 with \6 from \4 by \1(\6. T"
/* 4x _5_ _7 _E _5 _A _C */
"he \4 on \4 as \4 is \4ing \2\n\t\1:\3ed "
/* 6x _3 _8 _D _2 _7_ _ _A _C */
"\2=\"\4 at \3ly \1,\2=\'\5.com/\7. This \5"
/* 8x _0 _ _3 _8 _C _E _ _1 _7 _F */
" not \3er \3al \4ful \4ive \5less \4es"
/* Ax _5 _9 _D _2 _7 _D */
"t \4ize \2\xc2\xa0\4ous \5 the \2e \0";
/* Cx _2 _7___ ___ _A _F _5 _8 */
static const uint16_t kPrefixSuffixMap[50] = {
0x00, 0x02, 0x05, 0x0E, 0x13, 0x16, 0x18, 0x1E, 0x23, 0x25,
0x2A, 0x2D, 0x2F, 0x32, 0x34, 0x3A, 0x3E, 0x45, 0x47, 0x4E,
0x55, 0x5A, 0x5C, 0x63, 0x68, 0x6D, 0x72, 0x77, 0x7A, 0x7C,
0x80, 0x83, 0x88, 0x8C, 0x8E, 0x91, 0x97, 0x9F, 0xA5, 0xA9,
0xAD, 0xB2, 0xB7, 0xBD, 0xC2, 0xC7, 0xCA, 0xCF, 0xD5, 0xD8
};
/* RFC 7932 transforms */
static const uint8_t kTransformsData[] = {
49, BROTLI_TRANSFORM_IDENTITY, 49,
49, BROTLI_TRANSFORM_IDENTITY, 0,
0, BROTLI_TRANSFORM_IDENTITY, 0,
49, BROTLI_TRANSFORM_OMIT_FIRST_1, 49,
49, BROTLI_TRANSFORM_UPPERCASE_FIRST, 0,
49, BROTLI_TRANSFORM_IDENTITY, 47,
0, BROTLI_TRANSFORM_IDENTITY, 49,
4, BROTLI_TRANSFORM_IDENTITY, 0,
49, BROTLI_TRANSFORM_IDENTITY, 3,
49, BROTLI_TRANSFORM_UPPERCASE_FIRST, 49,
49, BROTLI_TRANSFORM_IDENTITY, 6,
49, BROTLI_TRANSFORM_OMIT_FIRST_2, 49,
49, BROTLI_TRANSFORM_OMIT_LAST_1, 49,
1, BROTLI_TRANSFORM_IDENTITY, 0,
49, BROTLI_TRANSFORM_IDENTITY, 1,
0, BROTLI_TRANSFORM_UPPERCASE_FIRST, 0,
49, BROTLI_TRANSFORM_IDENTITY, 7,
49, BROTLI_TRANSFORM_IDENTITY, 9,
48, BROTLI_TRANSFORM_IDENTITY, 0,
49, BROTLI_TRANSFORM_IDENTITY, 8,
49, BROTLI_TRANSFORM_IDENTITY, 5,
49, BROTLI_TRANSFORM_IDENTITY, 10,
49, BROTLI_TRANSFORM_IDENTITY, 11,
49, BROTLI_TRANSFORM_OMIT_LAST_3, 49,
49, BROTLI_TRANSFORM_IDENTITY, 13,
49, BROTLI_TRANSFORM_IDENTITY, 14,
49, BROTLI_TRANSFORM_OMIT_FIRST_3, 49,
49, BROTLI_TRANSFORM_OMIT_LAST_2, 49,
49, BROTLI_TRANSFORM_IDENTITY, 15,
49, BROTLI_TRANSFORM_IDENTITY, 16,
0, BROTLI_TRANSFORM_UPPERCASE_FIRST, 49,
49, BROTLI_TRANSFORM_IDENTITY, 12,
5, BROTLI_TRANSFORM_IDENTITY, 49,
0, BROTLI_TRANSFORM_IDENTITY, 1,
49, BROTLI_TRANSFORM_OMIT_FIRST_4, 49,
49, BROTLI_TRANSFORM_IDENTITY, 18,
49, BROTLI_TRANSFORM_IDENTITY, 17,
49, BROTLI_TRANSFORM_IDENTITY, 19,
49, BROTLI_TRANSFORM_IDENTITY, 20,
49, BROTLI_TRANSFORM_OMIT_FIRST_5, 49,
49, BROTLI_TRANSFORM_OMIT_FIRST_6, 49,
47, BROTLI_TRANSFORM_IDENTITY, 49,
49, BROTLI_TRANSFORM_OMIT_LAST_4, 49,
49, BROTLI_TRANSFORM_IDENTITY, 22,
49, BROTLI_TRANSFORM_UPPERCASE_ALL, 49,
49, BROTLI_TRANSFORM_IDENTITY, 23,
49, BROTLI_TRANSFORM_IDENTITY, 24,
49, BROTLI_TRANSFORM_IDENTITY, 25,
49, BROTLI_TRANSFORM_OMIT_LAST_7, 49,
49, BROTLI_TRANSFORM_OMIT_LAST_1, 26,
49, BROTLI_TRANSFORM_IDENTITY, 27,
49, BROTLI_TRANSFORM_IDENTITY, 28,
0, BROTLI_TRANSFORM_IDENTITY, 12,
49, BROTLI_TRANSFORM_IDENTITY, 29,
49, BROTLI_TRANSFORM_OMIT_FIRST_9, 49,
49, BROTLI_TRANSFORM_OMIT_FIRST_7, 49,
49, BROTLI_TRANSFORM_OMIT_LAST_6, 49,
49, BROTLI_TRANSFORM_IDENTITY, 21,
49, BROTLI_TRANSFORM_UPPERCASE_FIRST, 1,
49, BROTLI_TRANSFORM_OMIT_LAST_8, 49,
49, BROTLI_TRANSFORM_IDENTITY, 31,
49, BROTLI_TRANSFORM_IDENTITY, 32,
47, BROTLI_TRANSFORM_IDENTITY, 3,
49, BROTLI_TRANSFORM_OMIT_LAST_5, 49,
49, BROTLI_TRANSFORM_OMIT_LAST_9, 49,
0, BROTLI_TRANSFORM_UPPERCASE_FIRST, 1,
49, BROTLI_TRANSFORM_UPPERCASE_FIRST, 8,
5, BROTLI_TRANSFORM_IDENTITY, 21,
49, BROTLI_TRANSFORM_UPPERCASE_ALL, 0,
49, BROTLI_TRANSFORM_UPPERCASE_FIRST, 10,
49, BROTLI_TRANSFORM_IDENTITY, 30,
0, BROTLI_TRANSFORM_IDENTITY, 5,
35, BROTLI_TRANSFORM_IDENTITY, 49,
47, BROTLI_TRANSFORM_IDENTITY, 2,
49, BROTLI_TRANSFORM_UPPERCASE_FIRST, 17,
49, BROTLI_TRANSFORM_IDENTITY, 36,
49, BROTLI_TRANSFORM_IDENTITY, 33,
5, BROTLI_TRANSFORM_IDENTITY, 0,
49, BROTLI_TRANSFORM_UPPERCASE_FIRST, 21,
49, BROTLI_TRANSFORM_UPPERCASE_FIRST, 5,
49, BROTLI_TRANSFORM_IDENTITY, 37,
0, BROTLI_TRANSFORM_IDENTITY, 30,
49, BROTLI_TRANSFORM_IDENTITY, 38,
0, BROTLI_TRANSFORM_UPPERCASE_ALL, 0,
49, BROTLI_TRANSFORM_IDENTITY, 39,
0, BROTLI_TRANSFORM_UPPERCASE_ALL, 49,
49, BROTLI_TRANSFORM_IDENTITY, 34,
49, BROTLI_TRANSFORM_UPPERCASE_ALL, 8,
49, BROTLI_TRANSFORM_UPPERCASE_FIRST, 12,
0, BROTLI_TRANSFORM_IDENTITY, 21,
49, BROTLI_TRANSFORM_IDENTITY, 40,
0, BROTLI_TRANSFORM_UPPERCASE_FIRST, 12,
49, BROTLI_TRANSFORM_IDENTITY, 41,
49, BROTLI_TRANSFORM_IDENTITY, 42,
49, BROTLI_TRANSFORM_UPPERCASE_ALL, 17,
49, BROTLI_TRANSFORM_IDENTITY, 43,
0, BROTLI_TRANSFORM_UPPERCASE_FIRST, 5,
49, BROTLI_TRANSFORM_UPPERCASE_ALL, 10,
0, BROTLI_TRANSFORM_IDENTITY, 34,
49, BROTLI_TRANSFORM_UPPERCASE_FIRST, 33,
49, BROTLI_TRANSFORM_IDENTITY, 44,
49, BROTLI_TRANSFORM_UPPERCASE_ALL, 5,
45, BROTLI_TRANSFORM_IDENTITY, 49,
0, BROTLI_TRANSFORM_IDENTITY, 33,
49, BROTLI_TRANSFORM_UPPERCASE_FIRST, 30,
49, BROTLI_TRANSFORM_UPPERCASE_ALL, 30,
49, BROTLI_TRANSFORM_IDENTITY, 46,
49, BROTLI_TRANSFORM_UPPERCASE_ALL, 1,
49, BROTLI_TRANSFORM_UPPERCASE_FIRST, 34,
0, BROTLI_TRANSFORM_UPPERCASE_FIRST, 33,
0, BROTLI_TRANSFORM_UPPERCASE_ALL, 30,
0, BROTLI_TRANSFORM_UPPERCASE_ALL, 1,
49, BROTLI_TRANSFORM_UPPERCASE_ALL, 33,
49, BROTLI_TRANSFORM_UPPERCASE_ALL, 21,
49, BROTLI_TRANSFORM_UPPERCASE_ALL, 12,
0, BROTLI_TRANSFORM_UPPERCASE_ALL, 5,
49, BROTLI_TRANSFORM_UPPERCASE_ALL, 34,
0, BROTLI_TRANSFORM_UPPERCASE_ALL, 12,
0, BROTLI_TRANSFORM_UPPERCASE_FIRST, 30,
0, BROTLI_TRANSFORM_UPPERCASE_ALL, 34,
0, BROTLI_TRANSFORM_UPPERCASE_FIRST, 34,
};
static BrotliTransforms kBrotliTransforms = {
sizeof(kPrefixSuffix),
(const uint8_t*)kPrefixSuffix,
kPrefixSuffixMap,
sizeof(kTransformsData) / (3 * sizeof(kTransformsData[0])),
kTransformsData,
{0, 12, 27, 23, 42, 63, 56, 48, 59, 64}
};
const BrotliTransforms* BrotliGetTransforms(void) {
return &kBrotliTransforms;
}
static int ToUpperCase(uint8_t* p) {
if (p[0] < 0xC0) {
if (p[0] >= 'a' && p[0] <= 'z') {
p[0] ^= 32;
}
return 1;
}
/* An overly simplified uppercasing model for UTF-8. */
if (p[0] < 0xE0) {
p[1] ^= 32;
return 2;
}
/* An arbitrary transform for three byte characters. */
p[2] ^= 5;
return 3;
}
int BrotliTransformDictionaryWord(uint8_t* dst, const uint8_t* word, int len,
const BrotliTransforms* transforms, int transfom_idx) {
int idx = 0;
const uint8_t* prefix = BROTLI_TRANSFORM_PREFIX(transforms, transfom_idx);
uint8_t type = BROTLI_TRANSFORM_TYPE(transforms, transfom_idx);
const uint8_t* suffix = BROTLI_TRANSFORM_SUFFIX(transforms, transfom_idx);
{
int prefix_len = *prefix++;
while (prefix_len--) { dst[idx++] = *prefix++; }
}
{
const int t = type;
int i = 0;
if (t <= BROTLI_TRANSFORM_OMIT_LAST_9) {
len -= t;
} else if (t >= BROTLI_TRANSFORM_OMIT_FIRST_1
&& t <= BROTLI_TRANSFORM_OMIT_FIRST_9) {
int skip = t - (BROTLI_TRANSFORM_OMIT_FIRST_1 - 1);
word += skip;
len -= skip;
}
while (i < len) { dst[idx++] = word[i++]; }
if (t == BROTLI_TRANSFORM_UPPERCASE_FIRST) {
ToUpperCase(&dst[idx - len]);
} else if (t == BROTLI_TRANSFORM_UPPERCASE_ALL) {
uint8_t* uppercase = &dst[idx - len];
while (len > 0) {
int step = ToUpperCase(uppercase);
uppercase += step;
len -= step;
}
}
}
{
int suffix_len = *suffix++;
while (suffix_len--) { dst[idx++] = *suffix++; }
return idx;
}
}
#if defined(__cplusplus) || defined(c_plusplus)
} /* extern "C" */
#endif

View File

@ -0,0 +1,80 @@
/* transforms is a part of ABI, but not API.
It means that there are some functions that are supposed to be in "common"
library, but header itself is not placed into include/brotli. This way,
aforementioned functions will be available only to brotli internals.
*/
#ifndef BROTLI_COMMON_TRANSFORM_H_
#define BROTLI_COMMON_TRANSFORM_H_
#include <brotli/port.h>
#include <brotli/types.h>
#if defined(__cplusplus) || defined(c_plusplus)
extern "C" {
#endif
enum BrotliWordTransformType {
BROTLI_TRANSFORM_IDENTITY = 0,
BROTLI_TRANSFORM_OMIT_LAST_1 = 1,
BROTLI_TRANSFORM_OMIT_LAST_2 = 2,
BROTLI_TRANSFORM_OMIT_LAST_3 = 3,
BROTLI_TRANSFORM_OMIT_LAST_4 = 4,
BROTLI_TRANSFORM_OMIT_LAST_5 = 5,
BROTLI_TRANSFORM_OMIT_LAST_6 = 6,
BROTLI_TRANSFORM_OMIT_LAST_7 = 7,
BROTLI_TRANSFORM_OMIT_LAST_8 = 8,
BROTLI_TRANSFORM_OMIT_LAST_9 = 9,
BROTLI_TRANSFORM_UPPERCASE_FIRST = 10,
BROTLI_TRANSFORM_UPPERCASE_ALL = 11,
BROTLI_TRANSFORM_OMIT_FIRST_1 = 12,
BROTLI_TRANSFORM_OMIT_FIRST_2 = 13,
BROTLI_TRANSFORM_OMIT_FIRST_3 = 14,
BROTLI_TRANSFORM_OMIT_FIRST_4 = 15,
BROTLI_TRANSFORM_OMIT_FIRST_5 = 16,
BROTLI_TRANSFORM_OMIT_FIRST_6 = 17,
BROTLI_TRANSFORM_OMIT_FIRST_7 = 18,
BROTLI_TRANSFORM_OMIT_FIRST_8 = 19,
BROTLI_TRANSFORM_OMIT_FIRST_9 = 20,
BROTLI_NUM_TRANSFORM_TYPES /* Counts transforms, not a transform itself. */
};
#define BROTLI_TRANSFORMS_MAX_CUT_OFF BROTLI_TRANSFORM_OMIT_LAST_9
typedef struct BrotliTransforms {
uint16_t prefix_suffix_size;
/* Last character must be null, so prefix_suffix_size must be at least 1. */
const uint8_t* prefix_suffix;
const uint16_t* prefix_suffix_map;
uint32_t num_transforms;
/* Each entry is a [prefix_id, transform, suffix_id] triplet. */
const uint8_t* transforms;
/* Indices of transforms like ["", BROTLI_TRANSFORM_OMIT_LAST_#, ""].
0-th element corresponds to ["", BROTLI_TRANSFORM_IDENTITY, ""].
-1, if cut-off transform does not exist. */
int16_t cutOffTransforms[BROTLI_TRANSFORMS_MAX_CUT_OFF + 1];
} BrotliTransforms;
/* T is BrotliTransforms*; result is uint8_t. */
#define BROTLI_TRANSFORM_PREFIX_ID(T, I) ((T)->transforms[((I) * 3) + 0])
#define BROTLI_TRANSFORM_TYPE(T, I) ((T)->transforms[((I) * 3) + 1])
#define BROTLI_TRANSFORM_SUFFIX_ID(T, I) ((T)->transforms[((I) * 3) + 2])
/* T is BrotliTransforms*; result is const uint8_t*. */
#define BROTLI_TRANSFORM_PREFIX(T, I) (&(T)->prefix_suffix[ \
(T)->prefix_suffix_map[BROTLI_TRANSFORM_PREFIX_ID(T, I)]])
#define BROTLI_TRANSFORM_SUFFIX(T, I) (&(T)->prefix_suffix[ \
(T)->prefix_suffix_map[BROTLI_TRANSFORM_SUFFIX_ID(T, I)]])
BROTLI_COMMON_API const BrotliTransforms* BrotliGetTransforms(void);
BROTLI_COMMON_API int BrotliTransformDictionaryWord(
uint8_t* dst, const uint8_t* word, int len,
const BrotliTransforms* transforms, int transform_idx);
#if defined(__cplusplus) || defined(c_plusplus)
} /* extern "C" */
#endif
#endif /* BROTLI_COMMON_TRANSFORM_H_ */

View File

@ -1,58 +0,0 @@
/* Copyright 2013 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* Common types */
#ifndef BROTLI_COMMON_TYPES_H_
#define BROTLI_COMMON_TYPES_H_
#include <stddef.h> /* for size_t */
#if defined(_MSC_VER) && (_MSC_VER < 1600)
typedef __int8 int8_t;
typedef unsigned __int8 uint8_t;
typedef __int16 int16_t;
typedef unsigned __int16 uint16_t;
typedef __int32 int32_t;
typedef unsigned __int32 uint32_t;
typedef unsigned __int64 uint64_t;
typedef __int64 int64_t;
#else
#include <stdint.h>
#endif /* defined(_MSC_VER) && (_MSC_VER < 1600) */
#if (!defined(_MSC_VER) || (_MSC_VER >= 1800)) && \
(defined(__cplusplus) || (defined(__STDC_VERSION__) && __STDC_VERSION__ >= 199901L))
#include <stdbool.h>
#define BROTLI_BOOL bool
#define BROTLI_TRUE true
#define BROTLI_FALSE false
#define TO_BROTLI_BOOL(X) (!!(X))
#else
typedef enum {
BROTLI_FALSE = 0,
BROTLI_TRUE = !BROTLI_FALSE
} BROTLI_BOOL;
#define TO_BROTLI_BOOL(X) (!!(X) ? BROTLI_TRUE : BROTLI_FALSE)
#endif
#define MAKE_UINT64_T(high, low) ((((uint64_t)(high)) << 32) | low)
#define BROTLI_UINT32_MAX (~((uint32_t)0))
#define BROTLI_SIZE_MAX (~((size_t)0))
/* Allocating function pointer. Function MUST return 0 in the case of failure.
Otherwise it MUST return a valid pointer to a memory region of at least
size length. Neither items nor size are allowed to be 0.
opaque argument is a pointer provided by client and could be used to bind
function to specific object (memory pool). */
typedef void* (*brotli_alloc_func)(void* opaque, size_t size);
/* Deallocating function pointer. Function SHOULD be no-op in the case the
address is 0. */
typedef void (*brotli_free_func)(void* opaque, void* address);
#endif /* BROTLI_COMMON_TYPES_H_ */

View File

@ -0,0 +1,26 @@
/* Copyright 2016 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* Version definition. */
#ifndef BROTLI_COMMON_VERSION_H_
#define BROTLI_COMMON_VERSION_H_
/* This macro should only be used when library is compiled together with client.
If library is dynamically linked, use BrotliDecoderVersion and
BrotliEncoderVersion methods. */
/* Semantic version, calculated as (MAJOR << 24) | (MINOR << 12) | PATCH */
#define BROTLI_VERSION 0x1000006
/* This macro is used by build system to produce Libtool-friendly soname. See
https://www.gnu.org/software/libtool/manual/html_node/Libtool-versioning.html
*/
/* ABI version, calculated as (CURRENT << 24) | (REVISION << 12) | AGE */
#define BROTLI_ABI_VERSION 0x1006000
#endif /* BROTLI_COMMON_VERSION_H_ */

View File

@ -8,8 +8,8 @@
#include "./bit_reader.h"
#include "../common/types.h"
#include "./port.h"
#include "../common/platform.h"
#include <brotli/types.h>
#if defined(__cplusplus) || defined(c_plusplus)
extern "C" {
@ -24,7 +24,7 @@ BROTLI_BOOL BrotliWarmupBitReader(BrotliBitReader* const br) {
size_t aligned_read_mask = (sizeof(br->val_) >> 1) - 1;
/* Fixing alignment after unaligned BrotliFillWindow would result accumulator
overflow. If unalignment is caused by BrotliSafeReadBits, then there is
enough space in accumulator to fix aligment. */
enough space in accumulator to fix alignment. */
if (!BROTLI_ALIGNED_READ) {
aligned_read_mask = 0;
}

View File

@ -11,22 +11,16 @@
#include <string.h> /* memcpy */
#include "../common/types.h"
#include "./port.h"
#include "../common/platform.h"
#include <brotli/types.h>
#if defined(__cplusplus) || defined(c_plusplus)
extern "C" {
#endif
#if (BROTLI_64_BITS)
#define BROTLI_SHORT_FILL_BIT_WINDOW_READ 4
typedef uint64_t reg_t;
#else
#define BROTLI_SHORT_FILL_BIT_WINDOW_READ 2
typedef uint32_t reg_t;
#endif
#define BROTLI_SHORT_FILL_BIT_WINDOW_READ (sizeof(brotli_reg_t) >> 1)
static const uint32_t kBitMask[33] = { 0x0000,
static const uint32_t kBitMask[33] = { 0x00000000,
0x00000001, 0x00000003, 0x00000007, 0x0000000F,
0x0000001F, 0x0000003F, 0x0000007F, 0x000000FF,
0x000001FF, 0x000003FF, 0x000007FF, 0x00000FFF,
@ -38,34 +32,35 @@ static const uint32_t kBitMask[33] = { 0x0000,
};
static BROTLI_INLINE uint32_t BitMask(uint32_t n) {
if (IS_CONSTANT(n) || BROTLI_HAS_UBFX) {
if (BROTLI_IS_CONSTANT(n) || BROTLI_HAS_UBFX) {
/* Masking with this expression turns to a single
"Unsigned Bit Field Extract" UBFX instruction on ARM. */
return ~((0xffffffffU) << n);
return ~((0xFFFFFFFFu) << n);
} else {
return kBitMask[n];
}
}
typedef struct {
reg_t val_; /* pre-fetched bits */
brotli_reg_t val_; /* pre-fetched bits */
uint32_t bit_pos_; /* current bit-reading position in val_ */
const uint8_t* next_in; /* the byte we're reading from */
size_t avail_in;
} BrotliBitReader;
typedef struct {
reg_t val_;
brotli_reg_t val_;
uint32_t bit_pos_;
const uint8_t* next_in;
size_t avail_in;
} BrotliBitReaderState;
/* Initializes the bitreader fields. */
/* Initializes the BrotliBitReader fields. */
BROTLI_INTERNAL void BrotliInitBitReader(BrotliBitReader* const br);
/* Ensures that accumulator is not empty. May consume one byte of input.
Returns 0 if data is required but there is no input available.
/* Ensures that accumulator is not empty.
May consume up to sizeof(brotli_reg_t) - 1 bytes of input.
Returns BROTLI_FALSE if data is required but there is no input available.
For BROTLI_ALIGNED_READ this function also prepares bit reader for aligned
reading. */
BROTLI_INTERNAL BROTLI_BOOL BrotliWarmupBitReader(BrotliBitReader* const br);
@ -97,89 +92,34 @@ static BROTLI_INLINE size_t BrotliGetRemainingBytes(BrotliBitReader* br) {
return br->avail_in + (BrotliGetAvailableBits(br) >> 3);
}
/* Checks if there is at least num bytes left in the input ringbuffer (excluding
the bits remaining in br->val_). */
/* Checks if there is at least |num| bytes left in the input ring-buffer
(excluding the bits remaining in br->val_). */
static BROTLI_INLINE BROTLI_BOOL BrotliCheckInputAmount(
BrotliBitReader* const br, size_t num) {
return TO_BROTLI_BOOL(br->avail_in >= num);
}
static BROTLI_INLINE uint16_t BrotliLoad16LE(const uint8_t* in) {
if (BROTLI_LITTLE_ENDIAN) {
return *((const uint16_t*)in);
} else if (BROTLI_BIG_ENDIAN) {
uint16_t value = *((const uint16_t*)in);
return (uint16_t)(((value & 0xFFU) << 8) | ((value & 0xFF00U) >> 8));
} else {
return (uint16_t)(in[0] | (in[1] << 8));
}
}
static BROTLI_INLINE uint32_t BrotliLoad32LE(const uint8_t* in) {
if (BROTLI_LITTLE_ENDIAN) {
return *((const uint32_t*)in);
} else if (BROTLI_BIG_ENDIAN) {
uint32_t value = *((const uint32_t*)in);
return ((value & 0xFFU) << 24) | ((value & 0xFF00U) << 8) |
((value & 0xFF0000U) >> 8) | ((value & 0xFF000000U) >> 24);
} else {
uint32_t value = (uint32_t)(*(in++));
value |= (uint32_t)(*(in++)) << 8;
value |= (uint32_t)(*(in++)) << 16;
value |= (uint32_t)(*(in++)) << 24;
return value;
}
}
#if (BROTLI_64_BITS)
static BROTLI_INLINE uint64_t BrotliLoad64LE(const uint8_t* in) {
if (BROTLI_LITTLE_ENDIAN) {
return *((const uint64_t*)in);
} else if (BROTLI_BIG_ENDIAN) {
uint64_t value = *((const uint64_t*)in);
return
((value & 0xFFU) << 56) |
((value & 0xFF00U) << 40) |
((value & 0xFF0000U) << 24) |
((value & 0xFF000000U) << 8) |
((value & 0xFF00000000U) >> 8) |
((value & 0xFF0000000000U) >> 24) |
((value & 0xFF000000000000U) >> 40) |
((value & 0xFF00000000000000U) >> 56);
} else {
uint64_t value = (uint64_t)(*(in++));
value |= (uint64_t)(*(in++)) << 8;
value |= (uint64_t)(*(in++)) << 16;
value |= (uint64_t)(*(in++)) << 24;
value |= (uint64_t)(*(in++)) << 32;
value |= (uint64_t)(*(in++)) << 40;
value |= (uint64_t)(*(in++)) << 48;
value |= (uint64_t)(*(in++)) << 56;
return value;
}
}
#endif
/* Guarantees that there are at least n_bits + 1 bits in accumulator.
/* Guarantees that there are at least |n_bits| + 1 bits in accumulator.
Precondition: accumulator contains at least 1 bit.
n_bits should be in the range [1..24] for regular build. For portable
non-64-bit little endian build only 16 bits are safe to request. */
|n_bits| should be in the range [1..24] for regular build. For portable
non-64-bit little-endian build only 16 bits are safe to request. */
static BROTLI_INLINE void BrotliFillBitWindow(
BrotliBitReader* const br, uint32_t n_bits) {
#if (BROTLI_64_BITS)
if (!BROTLI_ALIGNED_READ && IS_CONSTANT(n_bits) && (n_bits <= 8)) {
if (!BROTLI_ALIGNED_READ && BROTLI_IS_CONSTANT(n_bits) && (n_bits <= 8)) {
if (br->bit_pos_ >= 56) {
br->val_ >>= 56;
br->bit_pos_ ^= 56; /* here same as -= 56 because of the if condition */
br->val_ |= BrotliLoad64LE(br->next_in) << 8;
br->val_ |= BROTLI_UNALIGNED_LOAD64LE(br->next_in) << 8;
br->avail_in -= 7;
br->next_in += 7;
}
} else if (!BROTLI_ALIGNED_READ && IS_CONSTANT(n_bits) && (n_bits <= 16)) {
} else if (
!BROTLI_ALIGNED_READ && BROTLI_IS_CONSTANT(n_bits) && (n_bits <= 16)) {
if (br->bit_pos_ >= 48) {
br->val_ >>= 48;
br->bit_pos_ ^= 48; /* here same as -= 48 because of the if condition */
br->val_ |= BrotliLoad64LE(br->next_in) << 16;
br->val_ |= BROTLI_UNALIGNED_LOAD64LE(br->next_in) << 16;
br->avail_in -= 6;
br->next_in += 6;
}
@ -187,17 +127,17 @@ static BROTLI_INLINE void BrotliFillBitWindow(
if (br->bit_pos_ >= 32) {
br->val_ >>= 32;
br->bit_pos_ ^= 32; /* here same as -= 32 because of the if condition */
br->val_ |= ((uint64_t)BrotliLoad32LE(br->next_in)) << 32;
br->val_ |= ((uint64_t)BROTLI_UNALIGNED_LOAD32LE(br->next_in)) << 32;
br->avail_in -= BROTLI_SHORT_FILL_BIT_WINDOW_READ;
br->next_in += BROTLI_SHORT_FILL_BIT_WINDOW_READ;
}
}
#else
if (!BROTLI_ALIGNED_READ && IS_CONSTANT(n_bits) && (n_bits <= 8)) {
if (!BROTLI_ALIGNED_READ && BROTLI_IS_CONSTANT(n_bits) && (n_bits <= 8)) {
if (br->bit_pos_ >= 24) {
br->val_ >>= 24;
br->bit_pos_ ^= 24; /* here same as -= 24 because of the if condition */
br->val_ |= BrotliLoad32LE(br->next_in) << 8;
br->val_ |= BROTLI_UNALIGNED_LOAD32LE(br->next_in) << 8;
br->avail_in -= 3;
br->next_in += 3;
}
@ -205,7 +145,7 @@ static BROTLI_INLINE void BrotliFillBitWindow(
if (br->bit_pos_ >= 16) {
br->val_ >>= 16;
br->bit_pos_ ^= 16; /* here same as -= 16 because of the if condition */
br->val_ |= ((uint32_t)BrotliLoad16LE(br->next_in)) << 16;
br->val_ |= ((uint32_t)BROTLI_UNALIGNED_LOAD16LE(br->next_in)) << 16;
br->avail_in -= BROTLI_SHORT_FILL_BIT_WINDOW_READ;
br->next_in += BROTLI_SHORT_FILL_BIT_WINDOW_READ;
}
@ -213,13 +153,14 @@ static BROTLI_INLINE void BrotliFillBitWindow(
#endif
}
/* Mosltly like BrotliFillBitWindow, but guarantees only 16 bits and reads no
/* Mostly like BrotliFillBitWindow, but guarantees only 16 bits and reads no
more than BROTLI_SHORT_FILL_BIT_WINDOW_READ bytes of input. */
static BROTLI_INLINE void BrotliFillBitWindow16(BrotliBitReader* const br) {
BrotliFillBitWindow(br, 17);
}
/* Pulls one byte of input to accumulator. */
/* Tries to pull one byte of input to accumulator.
Returns BROTLI_FALSE if there is no input available. */
static BROTLI_INLINE BROTLI_BOOL BrotliPullByte(BrotliBitReader* const br) {
if (br->avail_in == 0) {
return BROTLI_FALSE;
@ -237,8 +178,9 @@ static BROTLI_INLINE BROTLI_BOOL BrotliPullByte(BrotliBitReader* const br) {
}
/* Returns currently available bits.
The number of valid bits could be calclulated by BrotliGetAvailableBits. */
static BROTLI_INLINE reg_t BrotliGetBitsUnmasked(BrotliBitReader* const br) {
The number of valid bits could be calculated by BrotliGetAvailableBits. */
static BROTLI_INLINE brotli_reg_t BrotliGetBitsUnmasked(
BrotliBitReader* const br) {
return br->val_ >> br->bit_pos_;
}
@ -250,15 +192,16 @@ static BROTLI_INLINE uint32_t BrotliGet16BitsUnmasked(
return (uint32_t)BrotliGetBitsUnmasked(br);
}
/* Returns the specified number of bits from br without advancing bit pos. */
/* Returns the specified number of bits from |br| without advancing bit
position. */
static BROTLI_INLINE uint32_t BrotliGetBits(
BrotliBitReader* const br, uint32_t n_bits) {
BrotliFillBitWindow(br, n_bits);
return (uint32_t)BrotliGetBitsUnmasked(br) & BitMask(n_bits);
}
/* Tries to peek the specified amount of bits. Returns 0, if there is not
enough input. */
/* Tries to peek the specified amount of bits. Returns BROTLI_FALSE, if there
is not enough input. */
static BROTLI_INLINE BROTLI_BOOL BrotliSafeGetBits(
BrotliBitReader* const br, uint32_t n_bits, uint32_t* val) {
while (BrotliGetAvailableBits(br) < n_bits) {
@ -270,7 +213,7 @@ static BROTLI_INLINE BROTLI_BOOL BrotliSafeGetBits(
return BROTLI_TRUE;
}
/* Advances the bit pos by n_bits. */
/* Advances the bit pos by |n_bits|. */
static BROTLI_INLINE void BrotliDropBits(
BrotliBitReader* const br, uint32_t n_bits) {
br->bit_pos_ += n_bits;
@ -289,17 +232,17 @@ static BROTLI_INLINE void BrotliBitReaderUnload(BrotliBitReader* br) {
br->bit_pos_ += unused_bits;
}
/* Reads the specified number of bits from br and advances the bit pos.
Precondition: accumulator MUST contain at least n_bits. */
/* Reads the specified number of bits from |br| and advances the bit pos.
Precondition: accumulator MUST contain at least |n_bits|. */
static BROTLI_INLINE void BrotliTakeBits(
BrotliBitReader* const br, uint32_t n_bits, uint32_t* val) {
*val = (uint32_t)BrotliGetBitsUnmasked(br) & BitMask(n_bits);
BROTLI_LOG(("[BrotliReadBits] %d %d %d val: %6x\n",
(int)br->avail_in, (int)br->bit_pos_, n_bits, (int)*val));
(int)br->avail_in, (int)br->bit_pos_, (int)n_bits, (int)*val));
BrotliDropBits(br, n_bits);
}
/* Reads the specified number of bits from br and advances the bit pos.
/* Reads the specified number of bits from |br| and advances the bit pos.
Assumes that there is enough input to perform BrotliFillBitWindow. */
static BROTLI_INLINE uint32_t BrotliReadBits(
BrotliBitReader* const br, uint32_t n_bits) {
@ -319,8 +262,8 @@ static BROTLI_INLINE uint32_t BrotliReadBits(
}
}
/* Tries to read the specified amount of bits. Returns 0, if there is not
enough input. n_bits MUST be positive. */
/* Tries to read the specified amount of bits. Returns BROTLI_FALSE, if there
is not enough input. |n_bits| MUST be positive. */
static BROTLI_INLINE BROTLI_BOOL BrotliSafeReadBits(
BrotliBitReader* const br, uint32_t n_bits, uint32_t* val) {
while (BrotliGetAvailableBits(br) < n_bits) {
@ -343,25 +286,8 @@ static BROTLI_INLINE BROTLI_BOOL BrotliJumpToByteBoundary(BrotliBitReader* br) {
return TO_BROTLI_BOOL(pad_bits == 0);
}
/* Peeks a byte at specified offset.
Precondition: bit reader is parked to a byte boundary.
Returns -1 if operation is not feasible. */
static BROTLI_INLINE int BrotliPeekByte(BrotliBitReader* br, size_t offset) {
uint32_t available_bits = BrotliGetAvailableBits(br);
size_t bytes_left = available_bits >> 3;
BROTLI_DCHECK((available_bits & 7) == 0);
if (offset < bytes_left) {
return (BrotliGetBitsUnmasked(br) >> (unsigned)(offset << 3)) & 0xFF;
}
offset -= bytes_left;
if (offset < br->avail_in) {
return br->next_in[offset];
}
return -1;
}
/* Copies remaining input bytes stored in the bit reader to the output. Value
num may not be larger than BrotliGetRemainingBytes. The bit reader must be
|num| may not be larger than BrotliGetRemainingBytes. The bit reader must be
warmed up again after this. */
static BROTLI_INLINE void BrotliCopyBytes(uint8_t* dest,
BrotliBitReader* br, size_t num) {

File diff suppressed because it is too large Load Diff

View File

@ -1,188 +0,0 @@
/* Copyright 2013 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* API for Brotli decompression */
#ifndef BROTLI_DEC_DECODE_H_
#define BROTLI_DEC_DECODE_H_
#include "../common/types.h"
#if defined(__cplusplus) || defined(c_plusplus)
extern "C" {
#endif
typedef struct BrotliDecoderStateStruct BrotliDecoderState;
typedef enum {
/* Decoding error, e.g. corrupt input or memory allocation problem */
BROTLI_DECODER_RESULT_ERROR = 0,
/* Decoding successfully completed */
BROTLI_DECODER_RESULT_SUCCESS = 1,
/* Partially done; should be called again with more input */
BROTLI_DECODER_RESULT_NEEDS_MORE_INPUT = 2,
/* Partially done; should be called again with more output */
BROTLI_DECODER_RESULT_NEEDS_MORE_OUTPUT = 3
} BrotliDecoderResult;
#define BROTLI_DECODER_ERROR_CODES_LIST(BROTLI_ERROR_CODE, SEPARATOR) \
BROTLI_ERROR_CODE(_, NO_ERROR, 0) SEPARATOR \
/* Same as BrotliDecoderResult values */ \
BROTLI_ERROR_CODE(_, SUCCESS, 1) SEPARATOR \
BROTLI_ERROR_CODE(_, NEEDS_MORE_INPUT, 2) SEPARATOR \
BROTLI_ERROR_CODE(_, NEEDS_MORE_OUTPUT, 3) SEPARATOR \
\
/* Errors caused by invalid input */ \
BROTLI_ERROR_CODE(_ERROR_FORMAT_, EXUBERANT_NIBBLE, -1) SEPARATOR \
BROTLI_ERROR_CODE(_ERROR_FORMAT_, RESERVED, -2) SEPARATOR \
BROTLI_ERROR_CODE(_ERROR_FORMAT_, EXUBERANT_META_NIBBLE, -3) SEPARATOR \
BROTLI_ERROR_CODE(_ERROR_FORMAT_, SIMPLE_HUFFMAN_ALPHABET, -4) SEPARATOR \
BROTLI_ERROR_CODE(_ERROR_FORMAT_, SIMPLE_HUFFMAN_SAME, -5) SEPARATOR \
BROTLI_ERROR_CODE(_ERROR_FORMAT_, CL_SPACE, -6) SEPARATOR \
BROTLI_ERROR_CODE(_ERROR_FORMAT_, HUFFMAN_SPACE, -7) SEPARATOR \
BROTLI_ERROR_CODE(_ERROR_FORMAT_, CONTEXT_MAP_REPEAT, -8) SEPARATOR \
BROTLI_ERROR_CODE(_ERROR_FORMAT_, BLOCK_LENGTH_1, -9) SEPARATOR \
BROTLI_ERROR_CODE(_ERROR_FORMAT_, BLOCK_LENGTH_2, -10) SEPARATOR \
BROTLI_ERROR_CODE(_ERROR_FORMAT_, TRANSFORM, -11) SEPARATOR \
BROTLI_ERROR_CODE(_ERROR_FORMAT_, DICTIONARY, -12) SEPARATOR \
BROTLI_ERROR_CODE(_ERROR_FORMAT_, WINDOW_BITS, -13) SEPARATOR \
BROTLI_ERROR_CODE(_ERROR_FORMAT_, PADDING_1, -14) SEPARATOR \
BROTLI_ERROR_CODE(_ERROR_FORMAT_, PADDING_2, -15) SEPARATOR \
\
/* -16..-20 codes are reserved */ \
\
/* Memory allocation problems */ \
BROTLI_ERROR_CODE(_ERROR_ALLOC_, CONTEXT_MODES, -21) SEPARATOR \
/* Literal, insert and distance trees together */ \
BROTLI_ERROR_CODE(_ERROR_ALLOC_, TREE_GROUPS, -22) SEPARATOR \
/* -23..-24 codes are reserved for distinct tree groups */ \
BROTLI_ERROR_CODE(_ERROR_ALLOC_, CONTEXT_MAP, -25) SEPARATOR \
BROTLI_ERROR_CODE(_ERROR_ALLOC_, RING_BUFFER_1, -26) SEPARATOR \
BROTLI_ERROR_CODE(_ERROR_ALLOC_, RING_BUFFER_2, -27) SEPARATOR \
/* -28..-29 codes are reserved for dynamic ringbuffer allocation */ \
BROTLI_ERROR_CODE(_ERROR_ALLOC_, BLOCK_TYPE_TREES, -30) SEPARATOR \
\
/* "Impossible" states */ \
BROTLI_ERROR_CODE(_ERROR_, UNREACHABLE, -31)
typedef enum {
#define _BROTLI_COMMA ,
#define _BROTLI_ERROR_CODE_ENUM_ITEM(PREFIX, NAME, CODE) \
BROTLI_DECODER ## PREFIX ## NAME = CODE
BROTLI_DECODER_ERROR_CODES_LIST(_BROTLI_ERROR_CODE_ENUM_ITEM, _BROTLI_COMMA)
#undef _BROTLI_ERROR_CODE_ENUM_ITEM
#undef _BROTLI_COMMA
} BrotliDecoderErrorCode;
#define BROTLI_LAST_ERROR_CODE BROTLI_DECODER_ERROR_UNREACHABLE
/* Creates the instance of BrotliDecoderState and initializes it. |alloc_func|
and |free_func| MUST be both zero or both non-zero. In the case they are both
zero, default memory allocators are used. |opaque| is passed to |alloc_func|
and |free_func| when they are called. */
BrotliDecoderState* BrotliDecoderCreateInstance(
brotli_alloc_func alloc_func, brotli_free_func free_func, void* opaque);
/* Deinitializes and frees BrotliDecoderState instance. */
void BrotliDecoderDestroyInstance(BrotliDecoderState* state);
/* Decompresses the data in |encoded_buffer| into |decoded_buffer|, and sets
|*decoded_size| to the decompressed length. */
BrotliDecoderResult BrotliDecoderDecompress(
size_t encoded_size, const uint8_t* encoded_buffer, size_t* decoded_size,
uint8_t* decoded_buffer);
/* Decompresses the data. Supports partial input and output.
Must be called with an allocated input buffer in |*next_in| and an allocated
output buffer in |*next_out|. The values |*available_in| and |*available_out|
must specify the allocated size in |*next_in| and |*next_out| respectively.
After each call, |*available_in| will be decremented by the amount of input
bytes consumed, and the |*next_in| pointer will be incremented by that
amount. Similarly, |*available_out| will be decremented by the amount of
output bytes written, and the |*next_out| pointer will be incremented by that
amount. |total_out|, if it is not a null-pointer, will be set to the number
of bytes decompressed since the last state initialization.
Input is never overconsumed, so |next_in| and |available_in| could be passed
to the next consumer after decoding is complete. */
BrotliDecoderResult BrotliDecoderDecompressStream(
BrotliDecoderState* s, size_t* available_in, const uint8_t** next_in,
size_t* available_out, uint8_t** next_out, size_t* total_out);
/* Fills the new state with a dictionary for LZ77, warming up the ringbuffer,
e.g. for custom static dictionaries for data formats.
Not to be confused with the built-in transformable dictionary of Brotli.
|size| should be less or equal to 2^24 (16MiB), otherwise the dictionary will
be ignored. The dictionary must exist in memory until decoding is done and
is owned by the caller. To use:
1) Allocate and initialize state with BrotliCreateInstance
2) Use BrotliSetCustomDictionary
3) Use BrotliDecompressStream
4) Clean up and free state with BrotliDestroyState
*/
void BrotliDecoderSetCustomDictionary(
BrotliDecoderState* s, size_t size, const uint8_t* dict);
/* Returns true, if decoder has some unconsumed output.
Otherwise returns false. */
BROTLI_BOOL BrotliDecoderHasMoreOutput(const BrotliDecoderState* s);
/* Returns true, if decoder has already received some input bytes.
Otherwise returns false. */
BROTLI_BOOL BrotliDecoderIsUsed(const BrotliDecoderState* s);
/* Returns true, if decoder is in a state where we reached the end of the input
and produced all of the output; returns false otherwise. */
BROTLI_BOOL BrotliDecoderIsFinished(const BrotliDecoderState* s);
/* Returns detailed error code after BrotliDecompressStream returns
BROTLI_DECODER_RESULT_ERROR. */
BrotliDecoderErrorCode BrotliDecoderGetErrorCode(const BrotliDecoderState* s);
const char* BrotliDecoderErrorString(BrotliDecoderErrorCode c);
/* DEPRECATED >>> */
typedef enum {
BROTLI_RESULT_ERROR = 0,
BROTLI_RESULT_SUCCESS = 1,
BROTLI_RESULT_NEEDS_MORE_INPUT = 2,
BROTLI_RESULT_NEEDS_MORE_OUTPUT = 3
} BrotliResult;
typedef enum {
#define _BROTLI_COMMA ,
#define _BROTLI_ERROR_CODE_ENUM_ITEM(PREFIX, NAME, CODE) \
BROTLI ## PREFIX ## NAME = CODE
BROTLI_DECODER_ERROR_CODES_LIST(_BROTLI_ERROR_CODE_ENUM_ITEM, _BROTLI_COMMA)
#undef _BROTLI_ERROR_CODE_ENUM_ITEM
#undef _BROTLI_COMMA
} BrotliErrorCode;
typedef struct BrotliStateStruct BrotliState;
BrotliState* BrotliCreateState(
brotli_alloc_func alloc, brotli_free_func free, void* opaque);
void BrotliDestroyState(BrotliState* state);
BROTLI_BOOL BrotliDecompressedSize(
size_t encoded_size, const uint8_t* encoded_buffer, size_t* decoded_size);
BrotliResult BrotliDecompressBuffer(
size_t encoded_size, const uint8_t* encoded_buffer, size_t* decoded_size,
uint8_t* decoded_buffer);
BrotliResult BrotliDecompressStream(
size_t* available_in, const uint8_t** next_in, size_t* available_out,
uint8_t** next_out, size_t* total_out, BrotliState* s);
void BrotliSetCustomDictionary(
size_t size, const uint8_t* dict, BrotliState* s);
BROTLI_BOOL BrotliStateIsStreamStart(const BrotliState* s);
BROTLI_BOOL BrotliStateIsStreamEnd(const BrotliState* s);
BrotliErrorCode BrotliGetErrorCode(const BrotliState* s);
const char* BrotliErrorString(BrotliErrorCode c);
/* <<< DEPRECATED */
#if defined(__cplusplus) || defined(c_plusplus)
} /* extern "C" */
#endif
#endif /* BROTLI_DEC_DECODE_H_ */

View File

@ -11,8 +11,8 @@
#include <string.h> /* memcpy, memset */
#include "../common/constants.h"
#include "../common/types.h"
#include "./port.h"
#include "../common/platform.h"
#include <brotli/types.h>
#if defined(__cplusplus) || defined(c_plusplus)
extern "C" {
@ -20,8 +20,9 @@ extern "C" {
#define BROTLI_REVERSE_BITS_MAX 8
#ifdef BROTLI_RBIT
#define BROTLI_REVERSE_BITS_BASE (32 - BROTLI_REVERSE_BITS_MAX)
#if defined(BROTLI_RBIT)
#define BROTLI_REVERSE_BITS_BASE \
((sizeof(brotli_reg_t) << 3) - BROTLI_REVERSE_BITS_MAX)
#else
#define BROTLI_REVERSE_BITS_BASE 0
static uint8_t kReverseBits[1 << BROTLI_REVERSE_BITS_MAX] = {
@ -61,13 +62,13 @@ static uint8_t kReverseBits[1 << BROTLI_REVERSE_BITS_MAX] = {
#endif /* BROTLI_RBIT */
#define BROTLI_REVERSE_BITS_LOWEST \
(1U << (BROTLI_REVERSE_BITS_MAX - 1 + BROTLI_REVERSE_BITS_BASE))
((brotli_reg_t)1 << (BROTLI_REVERSE_BITS_MAX - 1 + BROTLI_REVERSE_BITS_BASE))
/* Returns reverse(num >> BROTLI_REVERSE_BITS_BASE, BROTLI_REVERSE_BITS_MAX),
where reverse(value, len) is the bit-wise reversal of the len least
significant bits of value. */
static BROTLI_INLINE uint32_t BrotliReverseBits(uint32_t num) {
#ifdef BROTLI_RBIT
static BROTLI_INLINE brotli_reg_t BrotliReverseBits(brotli_reg_t num) {
#if defined(BROTLI_RBIT)
return BROTLI_RBIT(num);
#else
return kReverseBits[num];
@ -85,9 +86,9 @@ static BROTLI_INLINE void ReplicateValue(HuffmanCode* table,
} while (end > 0);
}
/* Returns the table width of the next 2nd level table. count is the histogram
of bit lengths for the remaining symbols, len is the code length of the next
processed symbol */
/* Returns the table width of the next 2nd level table. |count| is the histogram
of bit lengths for the remaining symbols, |len| is the code length of the
next processed symbol. */
static BROTLI_INLINE int NextTableBitSize(const uint16_t* const count,
int len, int root_bits) {
int left = 1 << (len - root_bits);
@ -103,12 +104,12 @@ static BROTLI_INLINE int NextTableBitSize(const uint16_t* const count,
void BrotliBuildCodeLengthsHuffmanTable(HuffmanCode* table,
const uint8_t* const code_lengths,
uint16_t* count) {
HuffmanCode code; /* current table entry */
int symbol; /* symbol index in original or sorted table */
uint32_t key; /* prefix code */
uint32_t key_step; /* prefix code addend */
int step; /* step size to replicate values in current table */
int table_size; /* size of current table */
HuffmanCode code; /* current table entry */
int symbol; /* symbol index in original or sorted table */
brotli_reg_t key; /* prefix code */
brotli_reg_t key_step; /* prefix code addend */
int step; /* step size to replicate values in current table */
int table_size; /* size of current table */
int sorted[BROTLI_CODE_LENGTH_CODES]; /* symbols sorted by code length */
/* offsets in sorted table for each length */
int offset[BROTLI_HUFFMAN_MAX_CODE_LENGTH_CODE_LENGTH + 1];
@ -117,7 +118,7 @@ void BrotliBuildCodeLengthsHuffmanTable(HuffmanCode* table,
BROTLI_DCHECK(BROTLI_HUFFMAN_MAX_CODE_LENGTH_CODE_LENGTH <=
BROTLI_REVERSE_BITS_MAX);
/* generate offsets into sorted symbol table by code length */
/* Generate offsets into sorted symbol table by code length. */
symbol = -1;
bits = 1;
BROTLI_REPEAT(BROTLI_HUFFMAN_MAX_CODE_LENGTH_CODE_LENGTH, {
@ -128,7 +129,7 @@ void BrotliBuildCodeLengthsHuffmanTable(HuffmanCode* table,
/* Symbols with code length 0 are placed after all other symbols. */
offset[0] = BROTLI_CODE_LENGTH_CODES - 1;
/* sort symbols by length, by symbol order within each length */
/* Sort symbols by length, by symbol order within each length. */
symbol = BROTLI_CODE_LENGTH_CODES;
do {
BROTLI_REPEAT(6, {
@ -143,13 +144,13 @@ void BrotliBuildCodeLengthsHuffmanTable(HuffmanCode* table,
if (offset[0] == 0) {
code.bits = 0;
code.value = (uint16_t)sorted[0];
for (key = 0; key < (uint32_t)table_size; ++key) {
for (key = 0; key < (brotli_reg_t)table_size; ++key) {
table[key] = code;
}
return;
}
/* fill in table */
/* Fill in table. */
key = 0;
key_step = BROTLI_REVERSE_BITS_LOWEST;
symbol = 0;
@ -175,10 +176,10 @@ uint32_t BrotliBuildHuffmanTable(HuffmanCode* root_table,
HuffmanCode* table; /* next available space in table */
int len; /* current code length */
int symbol; /* symbol index in original or sorted table */
uint32_t key; /* prefix code */
uint32_t key_step; /* prefix code addend */
uint32_t sub_key; /* 2nd level table prefix code */
uint32_t sub_key_step; /* 2nd level table prefix code addend */
brotli_reg_t key; /* prefix code */
brotli_reg_t key_step; /* prefix code addend */
brotli_reg_t sub_key; /* 2nd level table prefix code */
brotli_reg_t sub_key_step; /* 2nd level table prefix code addend */
int step; /* step size to replicate values in current table */
int table_bits; /* key length of current table */
int table_size; /* size of current table */
@ -199,9 +200,8 @@ uint32_t BrotliBuildHuffmanTable(HuffmanCode* root_table,
table_size = 1 << table_bits;
total_size = table_size;
/* fill in root table */
/* let's reduce the table size to a smaller size if possible, and */
/* create the repetitions by memcpy if possible in the coming loop */
/* Fill in the root table. Reduce the table size to if possible,
and create the repetitions by memcpy. */
if (table_bits > max_length) {
table_bits = max_length;
table_size = 1 << table_bits;
@ -223,15 +223,14 @@ uint32_t BrotliBuildHuffmanTable(HuffmanCode* root_table,
key_step >>= 1;
} while (++bits <= table_bits);
/* if root_bits != table_bits we only created one fraction of the */
/* table, and we need to replicate it now. */
/* If root_bits != table_bits then replicate to fill the remaining slots. */
while (total_size != table_size) {
memcpy(&table[table_size], &table[0],
(size_t)table_size * sizeof(table[0]));
table_size <<= 1;
}
/* fill in 2nd level tables and add pointers to root table */
/* Fill in 2nd level tables and add pointers to root table. */
key_step = BROTLI_REVERSE_BITS_LOWEST >> (root_bits - 1);
sub_key = (BROTLI_REVERSE_BITS_LOWEST << 1);
sub_key_step = BROTLI_REVERSE_BITS_LOWEST;

View File

@ -9,8 +9,8 @@
#ifndef BROTLI_DEC_HUFFMAN_H_
#define BROTLI_DEC_HUFFMAN_H_
#include "../common/types.h"
#include "./port.h"
#include "../common/platform.h"
#include <brotli/types.h>
#if defined(__cplusplus) || defined(c_plusplus)
extern "C" {
@ -19,10 +19,11 @@ extern "C" {
#define BROTLI_HUFFMAN_MAX_CODE_LENGTH 15
/* Maximum possible Huffman table size for an alphabet size of (index * 32),
* max code length 15 and root table bits 8. */
max code length 15 and root table bits 8. */
static const uint16_t kMaxHuffmanTableSize[] = {
256, 402, 436, 468, 500, 534, 566, 598, 630, 662, 694, 726, 758, 790, 822,
854, 886, 920, 952, 984, 1016, 1048, 1080};
854, 886, 920, 952, 984, 1016, 1048, 1080, 1112, 1144, 1176, 1208, 1240, 1272,
1304, 1336, 1368, 1400, 1432, 1464, 1496, 1528};
/* BROTLI_NUM_BLOCK_LEN_SYMBOLS == 26 */
#define BROTLI_HUFFMAN_MAX_SIZE_26 396
/* BROTLI_MAX_BLOCK_TYPE_SYMBOLS == 258 */
@ -41,23 +42,26 @@ typedef struct {
BROTLI_INTERNAL void BrotliBuildCodeLengthsHuffmanTable(HuffmanCode* root_table,
const uint8_t* const code_lengths, uint16_t* count);
/* Builds Huffman lookup table assuming code lengths are in symbol order. */
/* Returns size of resulting table. */
/* Builds Huffman lookup table assuming code lengths are in symbol order.
Returns size of resulting table. */
BROTLI_INTERNAL uint32_t BrotliBuildHuffmanTable(HuffmanCode* root_table,
int root_bits, const uint16_t* const symbol_lists, uint16_t* count_arg);
/* Builds a simple Huffman table. The num_symbols parameter is to be */
/* interpreted as follows: 0 means 1 symbol, 1 means 2 symbols, 2 means 3 */
/* symbols, 3 means 4 symbols with lengths 2,2,2,2, 4 means 4 symbols with */
/* lengths 1,2,3,3. */
/* Builds a simple Huffman table. The |num_symbols| parameter is to be
interpreted as follows: 0 means 1 symbol, 1 means 2 symbols,
2 means 3 symbols, 3 means 4 symbols with lengths [2, 2, 2, 2],
4 means 4 symbols with lengths [1, 2, 3, 3]. */
BROTLI_INTERNAL uint32_t BrotliBuildSimpleHuffmanTable(HuffmanCode* table,
int root_bits, uint16_t* symbols, uint32_t num_symbols);
/* Contains a collection of Huffman trees with the same alphabet size. */
/* max_symbol is needed due to simple codes since log2(alphabet_size) could be
greater than log2(max_symbol). */
typedef struct {
HuffmanCode** htrees;
HuffmanCode* codes;
uint16_t alphabet_size;
uint16_t max_symbol;
uint16_t num_htrees;
} HuffmanTreeGroup;

View File

@ -1,159 +0,0 @@
/* Copyright 2015 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* Macros for compiler / platform specific features and build options.
Build options are:
* BROTLI_BUILD_32_BIT disables 64-bit optimizations
* BROTLI_BUILD_64_BIT forces to use 64-bit optimizations
* BROTLI_BUILD_BIG_ENDIAN forces to use big-endian optimizations
* BROTLI_BUILD_ENDIAN_NEUTRAL disables endian-aware optimizations
* BROTLI_BUILD_LITTLE_ENDIAN forces to use little-endian optimizations
* BROTLI_BUILD_MODERN_COMPILER forces to use modern compilers built-ins,
features and attributes
* BROTLI_BUILD_PORTABLE disables dangerous optimizations, like unaligned
read and overlapping memcpy; this reduces decompression speed by 5%
* BROTLI_DEBUG dumps file name and line number when decoder detects stream
or memory error
* BROTLI_ENABLE_LOG enables asserts and dumps various state information
*/
#ifndef BROTLI_DEC_PORT_H_
#define BROTLI_DEC_PORT_H_
#if defined(BROTLI_ENABLE_LOG) || defined(BROTLI_DEBUG)
#include <assert.h>
#include <stdio.h>
#endif
#include "../common/port.h"
#if defined(__arm__) || defined(__thumb__) || \
defined(_M_ARM) || defined(_M_ARMT)
#define BROTLI_TARGET_ARM
#if (defined(__ARM_ARCH) && (__ARM_ARCH >= 7)) || \
(defined(M_ARM) && (M_ARM >= 7))
#define BROTLI_TARGET_ARMV7
#endif /* ARMv7 */
#if defined(__aarch64__)
#define BROTLI_TARGET_ARMV8
#endif /* ARMv8 */
#endif /* ARM */
#if defined(__i386) || defined(_M_IX86)
#define BROTLI_TARGET_X86
#endif
#if defined(__x86_64__) || defined(_M_X64)
#define BROTLI_TARGET_X64
#endif
#if defined(__PPC64__)
#define BROTLI_TARGET_POWERPC64
#endif
#ifdef BROTLI_BUILD_PORTABLE
#define BROTLI_ALIGNED_READ (!!1)
#elif defined(BROTLI_TARGET_X86) || defined(BROTLI_TARGET_X64) || \
defined(BROTLI_TARGET_ARMV7) || defined(BROTLI_TARGET_ARMV8)
/* Allow unaligned read only for whitelisted CPUs. */
#define BROTLI_ALIGNED_READ (!!0)
#else
#define BROTLI_ALIGNED_READ (!!1)
#endif
/* IS_CONSTANT macros returns true for compile-time constant expressions. */
#if BROTLI_MODERN_COMPILER || __has_builtin(__builtin_constant_p)
#define IS_CONSTANT(x) (!!__builtin_constant_p(x))
#else
#define IS_CONSTANT(x) (!!0)
#endif
#ifdef BROTLI_ENABLE_LOG
#define BROTLI_DCHECK(x) assert(x)
#define BROTLI_LOG(x) printf x
#else
#define BROTLI_DCHECK(x)
#define BROTLI_LOG(x)
#endif
#if defined(BROTLI_DEBUG) || defined(BROTLI_ENABLE_LOG)
static BROTLI_INLINE void BrotliDump(const char* f, int l, const char* fn) {
fprintf(stderr, "%s:%d (%s)\n", f, l, fn);
fflush(stderr);
}
#define BROTLI_DUMP() BrotliDump(__FILE__, __LINE__, __FUNCTION__)
#else
#define BROTLI_DUMP() (void)(0)
#endif
#if defined(BROTLI_BUILD_64_BIT)
#define BROTLI_64_BITS 1
#elif defined(BROTLI_BUILD_32_BIT)
#define BROTLI_64_BITS 0
#elif defined(BROTLI_TARGET_X64) || defined(BROTLI_TARGET_ARMV8) || \
defined(BROTLI_TARGET_POWERPC64)
#define BROTLI_64_BITS 1
#else
#define BROTLI_64_BITS 0
#endif
#if defined(BROTLI_BUILD_BIG_ENDIAN)
#define BROTLI_LITTLE_ENDIAN 0
#define BROTLI_BIG_ENDIAN 1
#elif defined(BROTLI_BUILD_LITTLE_ENDIAN)
#define BROTLI_LITTLE_ENDIAN 1
#define BROTLI_BIG_ENDIAN 0
#elif defined(BROTLI_BUILD_ENDIAN_NEUTRAL)
#define BROTLI_LITTLE_ENDIAN 0
#define BROTLI_BIG_ENDIAN 0
#elif defined(__BYTE_ORDER__) && (__BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__)
#define BROTLI_LITTLE_ENDIAN 1
#define BROTLI_BIG_ENDIAN 0
#elif defined(_WIN32)
/* Win32 can currently always be assumed to be little endian */
#define BROTLI_LITTLE_ENDIAN 1
#define BROTLI_BIG_ENDIAN 0
#else
#if (defined(__BYTE_ORDER__) && (__BYTE_ORDER__ == __ORDER_BIG_ENDIAN__))
#define BROTLI_BIG_ENDIAN 1
#else
#define BROTLI_BIG_ENDIAN 0
#endif
#define BROTLI_LITTLE_ENDIAN 0
#endif
#define BROTLI_REPEAT(N, X) { \
if ((N & 1) != 0) {X;} \
if ((N & 2) != 0) {X; X;} \
if ((N & 4) != 0) {X; X; X; X;} \
}
#if BROTLI_MODERN_COMPILER || defined(__llvm__)
#if defined(BROTLI_TARGET_ARMV7)
static BROTLI_INLINE unsigned BrotliRBit(unsigned input) {
unsigned output;
__asm__("rbit %0, %1\n" : "=r"(output) : "r"(input));
return output;
}
#define BROTLI_RBIT(x) BrotliRBit(x)
#endif /* armv7 */
#endif /* gcc || clang */
#if defined(BROTLI_TARGET_ARM)
#define BROTLI_HAS_UBFX (!!1)
#else
#define BROTLI_HAS_UBFX (!!0)
#endif
#define BROTLI_ALLOC(S, L) S->alloc_func(S->memory_manager_opaque, L)
#define BROTLI_FREE(S, X) { \
S->free_func(S->memory_manager_opaque, X); \
X = NULL; \
}
#endif /* BROTLI_DEC_PORT_H_ */

View File

@ -5,17 +5,16 @@
*/
/* Lookup tables to map prefix codes to value ranges. This is used during
decoding of the block lengths, literal insertion lengths and copy lengths.
*/
decoding of the block lengths, literal insertion lengths and copy lengths. */
#ifndef BROTLI_DEC_PREFIX_H_
#define BROTLI_DEC_PREFIX_H_
#include "../common/constants.h"
#include "../common/types.h"
#include <brotli/types.h>
/* Represents the range of values belonging to a prefix code: */
/* [offset, offset + 2^nbits) */
/* Represents the range of values belonging to a prefix code:
[offset, offset + 2^nbits) */
struct PrefixCodeRange {
uint16_t offset;
uint8_t nbits;

View File

@ -8,32 +8,18 @@
#include <stdlib.h> /* free, malloc */
#include "../common/types.h"
#include <brotli/types.h>
#include "./huffman.h"
#if defined(__cplusplus) || defined(c_plusplus)
extern "C" {
#endif
static void* DefaultAllocFunc(void* opaque, size_t size) {
BROTLI_UNUSED(opaque);
return malloc(size);
}
static void DefaultFreeFunc(void* opaque, void* address) {
BROTLI_UNUSED(opaque);
free(address);
}
void BrotliDecoderStateInit(BrotliDecoderState* s) {
BrotliDecoderStateInitWithCustomAllocators(s, 0, 0, 0);
}
void BrotliDecoderStateInitWithCustomAllocators(BrotliDecoderState* s,
BROTLI_BOOL BrotliDecoderStateInit(BrotliDecoderState* s,
brotli_alloc_func alloc_func, brotli_free_func free_func, void* opaque) {
if (!alloc_func) {
s->alloc_func = DefaultAllocFunc;
s->free_func = DefaultFreeFunc;
s->alloc_func = BrotliDefaultAllocFunc;
s->free_func = BrotliDefaultFreeFunc;
s->memory_manager_opaque = 0;
} else {
s->alloc_func = alloc_func;
@ -41,8 +27,11 @@ void BrotliDecoderStateInitWithCustomAllocators(BrotliDecoderState* s,
s->memory_manager_opaque = opaque;
}
s->error_code = 0; /* BROTLI_DECODER_NO_ERROR */
BrotliInitBitReader(&s->br);
s->state = BROTLI_STATE_UNINITED;
s->large_window = 0;
s->substate_metablock_header = BROTLI_STATE_METABLOCK_HEADER_NONE;
s->substate_tree_group = BROTLI_STATE_TREE_GROUP_NONE;
s->substate_context_map = BROTLI_STATE_CONTEXT_MAP_NONE;
@ -60,6 +49,9 @@ void BrotliDecoderStateInitWithCustomAllocators(BrotliDecoderState* s,
s->block_type_trees = NULL;
s->block_len_trees = NULL;
s->ringbuffer = NULL;
s->ringbuffer_size = 0;
s->new_ringbuffer_size = 0;
s->ringbuffer_mask = 0;
s->context_map = NULL;
s->context_modes = NULL;
@ -76,10 +68,12 @@ void BrotliDecoderStateInitWithCustomAllocators(BrotliDecoderState* s,
s->distance_hgroup.codes = NULL;
s->distance_hgroup.htrees = NULL;
s->custom_dict = NULL;
s->custom_dict_size = 0;
s->is_last_metablock = 0;
s->is_uncompressed = 0;
s->is_metadata = 0;
s->should_wrap_ringbuffer = 0;
s->canny_ringbuffer_allocation = 1;
s->window_bits = 0;
s->max_distance = 0;
s->dist_rb[0] = 16;
@ -93,14 +87,19 @@ void BrotliDecoderStateInitWithCustomAllocators(BrotliDecoderState* s,
/* Make small negative indexes addressable. */
s->symbol_lists = &s->symbols_lists_array[BROTLI_HUFFMAN_MAX_CODE_LENGTH + 1];
s->mtf_upper_bound = 255;
s->mtf_upper_bound = 63;
s->dictionary = BrotliGetDictionary();
s->transforms = BrotliGetTransforms();
return BROTLI_TRUE;
}
void BrotliDecoderStateMetablockBegin(BrotliDecoderState* s) {
s->meta_block_remaining_len = 0;
s->block_length[0] = 1U << 28;
s->block_length[1] = 1U << 28;
s->block_length[2] = 1U << 28;
s->block_length[0] = 1U << 24;
s->block_length[1] = 1U << 24;
s->block_length[2] = 1U << 24;
s->num_block_types[0] = 1;
s->num_block_types[1] = 1;
s->num_block_types[2] = 1;
@ -117,8 +116,7 @@ void BrotliDecoderStateMetablockBegin(BrotliDecoderState* s) {
s->literal_htree = NULL;
s->dist_context_map_slice = NULL;
s->dist_htree_index = 0;
s->context_lookup1 = NULL;
s->context_lookup2 = NULL;
s->context_lookup = NULL;
s->literal_hgroup.codes = NULL;
s->literal_hgroup.htrees = NULL;
s->insert_copy_hgroup.codes = NULL;
@ -128,39 +126,37 @@ void BrotliDecoderStateMetablockBegin(BrotliDecoderState* s) {
}
void BrotliDecoderStateCleanupAfterMetablock(BrotliDecoderState* s) {
BROTLI_FREE(s, s->context_modes);
BROTLI_FREE(s, s->context_map);
BROTLI_FREE(s, s->dist_context_map);
BrotliDecoderHuffmanTreeGroupRelease(s, &s->literal_hgroup);
BrotliDecoderHuffmanTreeGroupRelease(s, &s->insert_copy_hgroup);
BrotliDecoderHuffmanTreeGroupRelease(s, &s->distance_hgroup);
BROTLI_DECODER_FREE(s, s->context_modes);
BROTLI_DECODER_FREE(s, s->context_map);
BROTLI_DECODER_FREE(s, s->dist_context_map);
BROTLI_DECODER_FREE(s, s->literal_hgroup.htrees);
BROTLI_DECODER_FREE(s, s->insert_copy_hgroup.htrees);
BROTLI_DECODER_FREE(s, s->distance_hgroup.htrees);
}
void BrotliDecoderStateCleanup(BrotliDecoderState* s) {
BrotliDecoderStateCleanupAfterMetablock(s);
BROTLI_FREE(s, s->ringbuffer);
BROTLI_FREE(s, s->block_type_trees);
BROTLI_DECODER_FREE(s, s->ringbuffer);
BROTLI_DECODER_FREE(s, s->block_type_trees);
}
void BrotliDecoderHuffmanTreeGroupInit(BrotliDecoderState* s,
HuffmanTreeGroup* group, uint32_t alphabet_size, uint32_t ntrees) {
BROTLI_BOOL BrotliDecoderHuffmanTreeGroupInit(BrotliDecoderState* s,
HuffmanTreeGroup* group, uint32_t alphabet_size, uint32_t max_symbol,
uint32_t ntrees) {
/* Pack two allocations into one */
const size_t max_table_size = kMaxHuffmanTableSize[(alphabet_size + 31) >> 5];
const size_t code_size = sizeof(HuffmanCode) * ntrees * max_table_size;
const size_t htree_size = sizeof(HuffmanCode*) * ntrees;
char* p = (char*)BROTLI_ALLOC(s, code_size + htree_size);
/* Pointer alignment is, hopefully, wider than sizeof(HuffmanCode). */
HuffmanCode** p = (HuffmanCode**)BROTLI_DECODER_ALLOC(s,
code_size + htree_size);
group->alphabet_size = (uint16_t)alphabet_size;
group->max_symbol = (uint16_t)max_symbol;
group->num_htrees = (uint16_t)ntrees;
group->codes = (HuffmanCode*)p;
group->htrees = (HuffmanCode**)(p + code_size);
}
void BrotliDecoderHuffmanTreeGroupRelease(
BrotliDecoderState* s, HuffmanTreeGroup* group) {
BROTLI_FREE(s, group->codes);
group->htrees = NULL;
group->htrees = p;
group->codes = (HuffmanCode*)(&p[ntrees]);
return !!p;
}
#if defined(__cplusplus) || defined(c_plusplus)

View File

@ -10,10 +10,12 @@
#define BROTLI_DEC_STATE_H_
#include "../common/constants.h"
#include "../common/types.h"
#include "../common/dictionary.h"
#include "../common/platform.h"
#include "../common/transform.h"
#include <brotli/types.h>
#include "./bit_reader.h"
#include "./huffman.h"
#include "./port.h"
#if defined(__cplusplus) || defined(c_plusplus)
extern "C" {
@ -21,6 +23,8 @@ extern "C" {
typedef enum {
BROTLI_STATE_UNINITED,
BROTLI_STATE_LARGE_WINDOW_BITS,
BROTLI_STATE_INITIALIZE,
BROTLI_STATE_METABLOCK_BEGIN,
BROTLI_STATE_METABLOCK_HEADER,
BROTLI_STATE_METABLOCK_HEADER_2,
@ -115,7 +119,6 @@ struct BrotliDecoderStateStruct {
int pos;
int max_backward_distance;
int max_backward_distance_minus_custom_dict_size;
int max_distance;
int ringbuffer_size;
int ringbuffer_mask;
@ -126,21 +129,22 @@ struct BrotliDecoderStateStruct {
uint8_t* ringbuffer;
uint8_t* ringbuffer_end;
HuffmanCode* htree_command;
const uint8_t* context_lookup1;
const uint8_t* context_lookup2;
const uint8_t* context_lookup;
uint8_t* context_map_slice;
uint8_t* dist_context_map_slice;
/* This ring buffer holds a few past copy distances that will be used by */
/* some special distance codes. */
/* This ring buffer holds a few past copy distances that will be used by
some special distance codes. */
HuffmanTreeGroup literal_hgroup;
HuffmanTreeGroup insert_copy_hgroup;
HuffmanTreeGroup distance_hgroup;
HuffmanCode* block_type_trees;
HuffmanCode* block_len_trees;
/* This is true if the literal context map histogram type always matches the
block type. It is then not needed to keep the context (faster decoding). */
block type. It is then not needed to keep the context (faster decoding). */
int trivial_literal_context;
/* Distance context is actual after command is decoded and before distance is
computed. After distance computation it is used as a temporary variable. */
int distance_context;
int meta_block_remaining_len;
uint32_t block_length_index;
@ -160,17 +164,17 @@ struct BrotliDecoderStateStruct {
int copy_length;
int distance_code;
/* For partial write operations */
size_t rb_roundtrips; /* How many times we went around the ringbuffer */
size_t partial_pos_out; /* How much output to the user in total (<= rb) */
/* For partial write operations. */
size_t rb_roundtrips; /* how many times we went around the ring-buffer */
size_t partial_pos_out; /* how much output to the user in total */
/* For ReadHuffmanCode */
/* For ReadHuffmanCode. */
uint32_t symbol;
uint32_t repeat;
uint32_t space;
HuffmanCode table[32];
/* List of of symbol chains. */
/* List of heads of symbol chains. */
uint16_t* symbol_lists;
/* Storage from symbol_lists. */
uint16_t symbols_lists_array[BROTLI_HUFFMAN_MAX_CODE_LENGTH + 1 +
@ -178,29 +182,26 @@ struct BrotliDecoderStateStruct {
/* Tails of symbol chains. */
int next_symbol[32];
uint8_t code_length_code_lengths[BROTLI_CODE_LENGTH_CODES];
/* Population counts for the code lengths */
/* Population counts for the code lengths. */
uint16_t code_length_histo[16];
/* For HuffmanTreeGroupDecode */
/* For HuffmanTreeGroupDecode. */
int htree_index;
HuffmanCode* next;
/* For DecodeContextMap */
/* For DecodeContextMap. */
uint32_t context_index;
uint32_t max_run_length_prefix;
uint32_t code;
HuffmanCode context_map_table[BROTLI_HUFFMAN_MAX_SIZE_272];
/* For InverseMoveToFrontTransform */
/* For InverseMoveToFrontTransform. */
uint32_t mtf_upper_bound;
uint8_t mtf[256 + 4];
uint32_t mtf[64 + 1];
/* For custom dictionaries */
const uint8_t* custom_dict;
int custom_dict_size;
/* Less used attributes are at the end of this struct. */
/* less used attributes are in the end of this struct */
/* States inside function calls */
/* States inside function calls. */
BrotliRunningMetablockHeaderState substate_metablock_header;
BrotliRunningTreeGroupState substate_tree_group;
BrotliRunningContextMapState substate_context_map;
@ -209,35 +210,46 @@ struct BrotliDecoderStateStruct {
BrotliRunningDecodeUint8State substate_decode_uint8;
BrotliRunningReadBlockLengthState substate_read_block_length;
uint8_t is_last_metablock;
uint8_t is_uncompressed;
uint8_t is_metadata;
uint8_t size_nibbles;
unsigned int is_last_metablock : 1;
unsigned int is_uncompressed : 1;
unsigned int is_metadata : 1;
unsigned int should_wrap_ringbuffer : 1;
unsigned int canny_ringbuffer_allocation : 1;
unsigned int large_window : 1;
unsigned int size_nibbles : 8;
uint32_t window_bits;
int new_ringbuffer_size;
uint32_t num_literal_htrees;
uint8_t* context_map;
uint8_t* context_modes;
const BrotliDictionary* dictionary;
const BrotliTransforms* transforms;
uint32_t trivial_literal_contexts[8]; /* 256 bits */
};
typedef struct BrotliDecoderStateStruct BrotliDecoderStateInternal;
#define BrotliDecoderState BrotliDecoderStateInternal
BROTLI_INTERNAL void BrotliDecoderStateInit(BrotliDecoderState* s);
BROTLI_INTERNAL void BrotliDecoderStateInitWithCustomAllocators(
BrotliDecoderState* s, brotli_alloc_func alloc_func,
brotli_free_func free_func, void* opaque);
BROTLI_INTERNAL BROTLI_BOOL BrotliDecoderStateInit(BrotliDecoderState* s,
brotli_alloc_func alloc_func, brotli_free_func free_func, void* opaque);
BROTLI_INTERNAL void BrotliDecoderStateCleanup(BrotliDecoderState* s);
BROTLI_INTERNAL void BrotliDecoderStateMetablockBegin(BrotliDecoderState* s);
BROTLI_INTERNAL void BrotliDecoderStateCleanupAfterMetablock(
BrotliDecoderState* s);
BROTLI_INTERNAL void BrotliDecoderHuffmanTreeGroupInit(
BROTLI_INTERNAL BROTLI_BOOL BrotliDecoderHuffmanTreeGroupInit(
BrotliDecoderState* s, HuffmanTreeGroup* group, uint32_t alphabet_size,
uint32_t ntrees);
BROTLI_INTERNAL void BrotliDecoderHuffmanTreeGroupRelease(
BrotliDecoderState* s, HuffmanTreeGroup* group);
uint32_t max_symbol, uint32_t ntrees);
#define BROTLI_DECODER_ALLOC(S, L) S->alloc_func(S->memory_manager_opaque, L)
#define BROTLI_DECODER_FREE(S, X) { \
S->free_func(S->memory_manager_opaque, X); \
X = NULL; \
}
#if defined(__cplusplus) || defined(c_plusplus)
} /* extern "C" */

View File

@ -1,300 +0,0 @@
/* Copyright 2013 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* Transformations on dictionary words. */
#ifndef BROTLI_DEC_TRANSFORM_H_
#define BROTLI_DEC_TRANSFORM_H_
#include "../common/types.h"
#include "./port.h"
#if defined(__cplusplus) || defined(c_plusplus)
extern "C" {
#endif
enum WordTransformType {
kIdentity = 0,
kOmitLast1 = 1,
kOmitLast2 = 2,
kOmitLast3 = 3,
kOmitLast4 = 4,
kOmitLast5 = 5,
kOmitLast6 = 6,
kOmitLast7 = 7,
kOmitLast8 = 8,
kOmitLast9 = 9,
kUppercaseFirst = 10,
kUppercaseAll = 11,
kOmitFirst1 = 12,
kOmitFirst2 = 13,
kOmitFirst3 = 14,
kOmitFirst4 = 15,
kOmitFirst5 = 16,
kOmitFirst6 = 17,
kOmitFirst7 = 18,
kOmitFirst8 = 19,
kOmitFirst9 = 20
};
typedef struct {
const uint8_t prefix_id;
const uint8_t transform;
const uint8_t suffix_id;
} Transform;
static const char kPrefixSuffix[208] =
"\0 \0, \0 of the \0 of \0s \0.\0 and \0 in \0\"\0 to \0\">\0\n\0. \0]\0"
" for \0 a \0 that \0\'\0 with \0 from \0 by \0(\0. The \0 on \0 as \0"
" is \0ing \0\n\t\0:\0ed \0=\"\0 at \0ly \0,\0=\'\0.com/\0. This \0"
" not \0er \0al \0ful \0ive \0less \0est \0ize \0\xc2\xa0\0ous ";
enum {
/* EMPTY = ""
SP = " "
DQUOT = "\""
SQUOT = "'"
CLOSEBR = "]"
OPEN = "("
SLASH = "/"
NBSP = non-breaking space "\0xc2\xa0"
*/
kPFix_EMPTY = 0,
kPFix_SP = 1,
kPFix_COMMASP = 3,
kPFix_SPofSPtheSP = 6,
kPFix_SPtheSP = 9,
kPFix_eSP = 12,
kPFix_SPofSP = 15,
kPFix_sSP = 20,
kPFix_DOT = 23,
kPFix_SPandSP = 25,
kPFix_SPinSP = 31,
kPFix_DQUOT = 36,
kPFix_SPtoSP = 38,
kPFix_DQUOTGT = 43,
kPFix_NEWLINE = 46,
kPFix_DOTSP = 48,
kPFix_CLOSEBR = 51,
kPFix_SPforSP = 53,
kPFix_SPaSP = 59,
kPFix_SPthatSP = 63,
kPFix_SQUOT = 70,
kPFix_SPwithSP = 72,
kPFix_SPfromSP = 79,
kPFix_SPbySP = 86,
kPFix_OPEN = 91,
kPFix_DOTSPTheSP = 93,
kPFix_SPonSP = 100,
kPFix_SPasSP = 105,
kPFix_SPisSP = 110,
kPFix_ingSP = 115,
kPFix_NEWLINETAB = 120,
kPFix_COLON = 123,
kPFix_edSP = 125,
kPFix_EQDQUOT = 129,
kPFix_SPatSP = 132,
kPFix_lySP = 137,
kPFix_COMMA = 141,
kPFix_EQSQUOT = 143,
kPFix_DOTcomSLASH = 146,
kPFix_DOTSPThisSP = 152,
kPFix_SPnotSP = 160,
kPFix_erSP = 166,
kPFix_alSP = 170,
kPFix_fulSP = 174,
kPFix_iveSP = 179,
kPFix_lessSP = 184,
kPFix_estSP = 190,
kPFix_izeSP = 195,
kPFix_NBSP = 200,
kPFix_ousSP = 203
};
static const Transform kTransforms[] = {
{ kPFix_EMPTY, kIdentity, kPFix_EMPTY },
{ kPFix_EMPTY, kIdentity, kPFix_SP },
{ kPFix_SP, kIdentity, kPFix_SP },
{ kPFix_EMPTY, kOmitFirst1, kPFix_EMPTY },
{ kPFix_EMPTY, kUppercaseFirst, kPFix_SP },
{ kPFix_EMPTY, kIdentity, kPFix_SPtheSP },
{ kPFix_SP, kIdentity, kPFix_EMPTY },
{ kPFix_sSP, kIdentity, kPFix_SP },
{ kPFix_EMPTY, kIdentity, kPFix_SPofSP },
{ kPFix_EMPTY, kUppercaseFirst, kPFix_EMPTY },
{ kPFix_EMPTY, kIdentity, kPFix_SPandSP },
{ kPFix_EMPTY, kOmitFirst2, kPFix_EMPTY },
{ kPFix_EMPTY, kOmitLast1, kPFix_EMPTY },
{ kPFix_COMMASP, kIdentity, kPFix_SP },
{ kPFix_EMPTY, kIdentity, kPFix_COMMASP },
{ kPFix_SP, kUppercaseFirst, kPFix_SP },
{ kPFix_EMPTY, kIdentity, kPFix_SPinSP },
{ kPFix_EMPTY, kIdentity, kPFix_SPtoSP },
{ kPFix_eSP, kIdentity, kPFix_SP },
{ kPFix_EMPTY, kIdentity, kPFix_DQUOT },
{ kPFix_EMPTY, kIdentity, kPFix_DOT },
{ kPFix_EMPTY, kIdentity, kPFix_DQUOTGT },
{ kPFix_EMPTY, kIdentity, kPFix_NEWLINE },
{ kPFix_EMPTY, kOmitLast3, kPFix_EMPTY },
{ kPFix_EMPTY, kIdentity, kPFix_CLOSEBR },
{ kPFix_EMPTY, kIdentity, kPFix_SPforSP },
{ kPFix_EMPTY, kOmitFirst3, kPFix_EMPTY },
{ kPFix_EMPTY, kOmitLast2, kPFix_EMPTY },
{ kPFix_EMPTY, kIdentity, kPFix_SPaSP },
{ kPFix_EMPTY, kIdentity, kPFix_SPthatSP },
{ kPFix_SP, kUppercaseFirst, kPFix_EMPTY },
{ kPFix_EMPTY, kIdentity, kPFix_DOTSP },
{ kPFix_DOT, kIdentity, kPFix_EMPTY },
{ kPFix_SP, kIdentity, kPFix_COMMASP },
{ kPFix_EMPTY, kOmitFirst4, kPFix_EMPTY },
{ kPFix_EMPTY, kIdentity, kPFix_SPwithSP },
{ kPFix_EMPTY, kIdentity, kPFix_SQUOT },
{ kPFix_EMPTY, kIdentity, kPFix_SPfromSP },
{ kPFix_EMPTY, kIdentity, kPFix_SPbySP },
{ kPFix_EMPTY, kOmitFirst5, kPFix_EMPTY },
{ kPFix_EMPTY, kOmitFirst6, kPFix_EMPTY },
{ kPFix_SPtheSP, kIdentity, kPFix_EMPTY },
{ kPFix_EMPTY, kOmitLast4, kPFix_EMPTY },
{ kPFix_EMPTY, kIdentity, kPFix_DOTSPTheSP },
{ kPFix_EMPTY, kUppercaseAll, kPFix_EMPTY },
{ kPFix_EMPTY, kIdentity, kPFix_SPonSP },
{ kPFix_EMPTY, kIdentity, kPFix_SPasSP },
{ kPFix_EMPTY, kIdentity, kPFix_SPisSP },
{ kPFix_EMPTY, kOmitLast7, kPFix_EMPTY },
{ kPFix_EMPTY, kOmitLast1, kPFix_ingSP },
{ kPFix_EMPTY, kIdentity, kPFix_NEWLINETAB },
{ kPFix_EMPTY, kIdentity, kPFix_COLON },
{ kPFix_SP, kIdentity, kPFix_DOTSP },
{ kPFix_EMPTY, kIdentity, kPFix_edSP },
{ kPFix_EMPTY, kOmitFirst9, kPFix_EMPTY },
{ kPFix_EMPTY, kOmitFirst7, kPFix_EMPTY },
{ kPFix_EMPTY, kOmitLast6, kPFix_EMPTY },
{ kPFix_EMPTY, kIdentity, kPFix_OPEN },
{ kPFix_EMPTY, kUppercaseFirst, kPFix_COMMASP },
{ kPFix_EMPTY, kOmitLast8, kPFix_EMPTY },
{ kPFix_EMPTY, kIdentity, kPFix_SPatSP },
{ kPFix_EMPTY, kIdentity, kPFix_lySP },
{ kPFix_SPtheSP, kIdentity, kPFix_SPofSP },
{ kPFix_EMPTY, kOmitLast5, kPFix_EMPTY },
{ kPFix_EMPTY, kOmitLast9, kPFix_EMPTY },
{ kPFix_SP, kUppercaseFirst, kPFix_COMMASP },
{ kPFix_EMPTY, kUppercaseFirst, kPFix_DQUOT },
{ kPFix_DOT, kIdentity, kPFix_OPEN },
{ kPFix_EMPTY, kUppercaseAll, kPFix_SP },
{ kPFix_EMPTY, kUppercaseFirst, kPFix_DQUOTGT },
{ kPFix_EMPTY, kIdentity, kPFix_EQDQUOT },
{ kPFix_SP, kIdentity, kPFix_DOT },
{ kPFix_DOTcomSLASH, kIdentity, kPFix_EMPTY },
{ kPFix_SPtheSP, kIdentity, kPFix_SPofSPtheSP },
{ kPFix_EMPTY, kUppercaseFirst, kPFix_SQUOT },
{ kPFix_EMPTY, kIdentity, kPFix_DOTSPThisSP },
{ kPFix_EMPTY, kIdentity, kPFix_COMMA },
{ kPFix_DOT, kIdentity, kPFix_SP },
{ kPFix_EMPTY, kUppercaseFirst, kPFix_OPEN },
{ kPFix_EMPTY, kUppercaseFirst, kPFix_DOT },
{ kPFix_EMPTY, kIdentity, kPFix_SPnotSP },
{ kPFix_SP, kIdentity, kPFix_EQDQUOT },
{ kPFix_EMPTY, kIdentity, kPFix_erSP },
{ kPFix_SP, kUppercaseAll, kPFix_SP },
{ kPFix_EMPTY, kIdentity, kPFix_alSP },
{ kPFix_SP, kUppercaseAll, kPFix_EMPTY },
{ kPFix_EMPTY, kIdentity, kPFix_EQSQUOT },
{ kPFix_EMPTY, kUppercaseAll, kPFix_DQUOT },
{ kPFix_EMPTY, kUppercaseFirst, kPFix_DOTSP },
{ kPFix_SP, kIdentity, kPFix_OPEN },
{ kPFix_EMPTY, kIdentity, kPFix_fulSP },
{ kPFix_SP, kUppercaseFirst, kPFix_DOTSP },
{ kPFix_EMPTY, kIdentity, kPFix_iveSP },
{ kPFix_EMPTY, kIdentity, kPFix_lessSP },
{ kPFix_EMPTY, kUppercaseAll, kPFix_SQUOT },
{ kPFix_EMPTY, kIdentity, kPFix_estSP },
{ kPFix_SP, kUppercaseFirst, kPFix_DOT },
{ kPFix_EMPTY, kUppercaseAll, kPFix_DQUOTGT },
{ kPFix_SP, kIdentity, kPFix_EQSQUOT },
{ kPFix_EMPTY, kUppercaseFirst, kPFix_COMMA },
{ kPFix_EMPTY, kIdentity, kPFix_izeSP },
{ kPFix_EMPTY, kUppercaseAll, kPFix_DOT },
{ kPFix_NBSP, kIdentity, kPFix_EMPTY },
{ kPFix_SP, kIdentity, kPFix_COMMA },
{ kPFix_EMPTY, kUppercaseFirst, kPFix_EQDQUOT },
{ kPFix_EMPTY, kUppercaseAll, kPFix_EQDQUOT },
{ kPFix_EMPTY, kIdentity, kPFix_ousSP },
{ kPFix_EMPTY, kUppercaseAll, kPFix_COMMASP },
{ kPFix_EMPTY, kUppercaseFirst, kPFix_EQSQUOT },
{ kPFix_SP, kUppercaseFirst, kPFix_COMMA },
{ kPFix_SP, kUppercaseAll, kPFix_EQDQUOT },
{ kPFix_SP, kUppercaseAll, kPFix_COMMASP },
{ kPFix_EMPTY, kUppercaseAll, kPFix_COMMA },
{ kPFix_EMPTY, kUppercaseAll, kPFix_OPEN },
{ kPFix_EMPTY, kUppercaseAll, kPFix_DOTSP },
{ kPFix_SP, kUppercaseAll, kPFix_DOT },
{ kPFix_EMPTY, kUppercaseAll, kPFix_EQSQUOT },
{ kPFix_SP, kUppercaseAll, kPFix_DOTSP },
{ kPFix_SP, kUppercaseFirst, kPFix_EQDQUOT },
{ kPFix_SP, kUppercaseAll, kPFix_EQSQUOT },
{ kPFix_SP, kUppercaseFirst, kPFix_EQSQUOT },
};
static const int kNumTransforms = sizeof(kTransforms) / sizeof(kTransforms[0]);
static int ToUpperCase(uint8_t* p) {
if (p[0] < 0xc0) {
if (p[0] >= 'a' && p[0] <= 'z') {
p[0] ^= 32;
}
return 1;
}
/* An overly simplified uppercasing model for utf-8. */
if (p[0] < 0xe0) {
p[1] ^= 32;
return 2;
}
/* An arbitrary transform for three byte characters. */
p[2] ^= 5;
return 3;
}
static BROTLI_NOINLINE int TransformDictionaryWord(
uint8_t* dst, const uint8_t* word, int len, int transform) {
int idx = 0;
{
const char* prefix = &kPrefixSuffix[kTransforms[transform].prefix_id];
while (*prefix) { dst[idx++] = (uint8_t)*prefix++; }
}
{
const int t = kTransforms[transform].transform;
int i = 0;
int skip = t - (kOmitFirst1 - 1);
if (skip > 0) {
word += skip;
len -= skip;
} else if (t <= kOmitLast9) {
len -= t;
}
while (i < len) { dst[idx++] = word[i++]; }
if (t == kUppercaseFirst) {
ToUpperCase(&dst[idx - len]);
} else if (t == kUppercaseAll) {
uint8_t* uppercase = &dst[idx - len];
while (len > 0) {
int step = ToUpperCase(uppercase);
uppercase += step;
len -= step;
}
}
}
{
const char* suffix = &kPrefixSuffix[kTransforms[transform].suffix_id];
while (*suffix) { dst[idx++] = (uint8_t)*suffix++; }
return idx;
}
}
#if defined(__cplusplus) || defined(c_plusplus)
} /* extern "C" */
#endif
#endif /* BROTLI_DEC_TRANSFORM_H_ */

View File

@ -8,208 +8,19 @@
#include "./backward_references.h"
#include <math.h> /* INFINITY */
#include <string.h> /* memcpy, memset */
#include "../common/constants.h"
#include "../common/types.h"
#include "../common/dictionary.h"
#include "../common/platform.h"
#include <brotli/types.h>
#include "./command.h"
#include "./fast_log.h"
#include "./find_match_length.h"
#include "./literal_cost.h"
#include "./dictionary_hash.h"
#include "./memory.h"
#include "./port.h"
#include "./prefix.h"
#include "./quality.h"
#if defined(__cplusplus) || defined(c_plusplus)
extern "C" {
#endif
#ifdef INFINITY
static const float kInfinity = INFINITY;
#else
static const float kInfinity = 3.4028e38f;
#endif
void BrotliInitZopfliNodes(ZopfliNode* array, size_t length) {
ZopfliNode stub;
size_t i;
stub.length = 1;
stub.distance = 0;
stub.insert_length = 0;
stub.u.cost = kInfinity;
for (i = 0; i < length; ++i) array[i] = stub;
}
static BROTLI_INLINE uint32_t ZopfliNodeCopyLength(const ZopfliNode* self) {
return self->length & 0xffffff;
}
static BROTLI_INLINE uint32_t ZopfliNodeLengthCode(const ZopfliNode* self) {
const uint32_t modifier = self->length >> 24;
return ZopfliNodeCopyLength(self) + 9u - modifier;
}
static BROTLI_INLINE uint32_t ZopfliNodeCopyDistance(const ZopfliNode* self) {
return self->distance & 0x1ffffff;
}
static BROTLI_INLINE uint32_t ZopfliNodeDistanceCode(const ZopfliNode* self) {
const uint32_t short_code = self->distance >> 25;
return short_code == 0 ? ZopfliNodeCopyDistance(self) + 15 : short_code - 1;
}
static BROTLI_INLINE uint32_t ZopfliNodeCommandLength(const ZopfliNode* self) {
return ZopfliNodeCopyLength(self) + self->insert_length;
}
/* Histogram based cost model for zopflification. */
typedef struct ZopfliCostModel {
/* The insert and copy length symbols. */
float cost_cmd_[BROTLI_NUM_COMMAND_SYMBOLS];
float cost_dist_[BROTLI_NUM_DISTANCE_SYMBOLS];
/* Cumulative costs of literals per position in the stream. */
float* literal_costs_;
float min_cost_cmd_;
size_t num_bytes_;
} ZopfliCostModel;
static void InitZopfliCostModel(
MemoryManager* m, ZopfliCostModel* self, size_t num_bytes) {
self->num_bytes_ = num_bytes;
self->literal_costs_ = BROTLI_ALLOC(m, float, num_bytes + 2);
if (BROTLI_IS_OOM(m)) return;
}
static void CleanupZopfliCostModel(MemoryManager* m, ZopfliCostModel* self) {
BROTLI_FREE(m, self->literal_costs_);
}
static void SetCost(const uint32_t* histogram, size_t histogram_size,
float* cost) {
size_t sum = 0;
float log2sum;
size_t i;
for (i = 0; i < histogram_size; i++) {
sum += histogram[i];
}
log2sum = (float)FastLog2(sum);
for (i = 0; i < histogram_size; i++) {
if (histogram[i] == 0) {
cost[i] = log2sum + 2;
continue;
}
/* Shannon bits for this symbol. */
cost[i] = log2sum - (float)FastLog2(histogram[i]);
/* Cannot be coded with less than 1 bit */
if (cost[i] < 1) cost[i] = 1;
}
}
static void ZopfliCostModelSetFromCommands(ZopfliCostModel* self,
size_t position,
const uint8_t* ringbuffer,
size_t ringbuffer_mask,
const Command* commands,
size_t num_commands,
size_t last_insert_len) {
uint32_t histogram_literal[BROTLI_NUM_LITERAL_SYMBOLS];
uint32_t histogram_cmd[BROTLI_NUM_COMMAND_SYMBOLS];
uint32_t histogram_dist[BROTLI_NUM_DISTANCE_SYMBOLS];
float cost_literal[BROTLI_NUM_LITERAL_SYMBOLS];
size_t pos = position - last_insert_len;
float min_cost_cmd = kInfinity;
size_t i;
float* cost_cmd = self->cost_cmd_;
memset(histogram_literal, 0, sizeof(histogram_literal));
memset(histogram_cmd, 0, sizeof(histogram_cmd));
memset(histogram_dist, 0, sizeof(histogram_dist));
for (i = 0; i < num_commands; i++) {
size_t inslength = commands[i].insert_len_;
size_t copylength = CommandCopyLen(&commands[i]);
size_t distcode = commands[i].dist_prefix_;
size_t cmdcode = commands[i].cmd_prefix_;
size_t j;
histogram_cmd[cmdcode]++;
if (cmdcode >= 128) histogram_dist[distcode]++;
for (j = 0; j < inslength; j++) {
histogram_literal[ringbuffer[(pos + j) & ringbuffer_mask]]++;
}
pos += inslength + copylength;
}
SetCost(histogram_literal, BROTLI_NUM_LITERAL_SYMBOLS, cost_literal);
SetCost(histogram_cmd, BROTLI_NUM_COMMAND_SYMBOLS, cost_cmd);
SetCost(histogram_dist, BROTLI_NUM_DISTANCE_SYMBOLS, self->cost_dist_);
for (i = 0; i < BROTLI_NUM_COMMAND_SYMBOLS; ++i) {
min_cost_cmd = BROTLI_MIN(float, min_cost_cmd, cost_cmd[i]);
}
self->min_cost_cmd_ = min_cost_cmd;
{
float* literal_costs = self->literal_costs_;
size_t num_bytes = self->num_bytes_;
literal_costs[0] = 0.0;
for (i = 0; i < num_bytes; ++i) {
literal_costs[i + 1] = literal_costs[i] +
cost_literal[ringbuffer[(position + i) & ringbuffer_mask]];
}
}
}
static void ZopfliCostModelSetFromLiteralCosts(ZopfliCostModel* self,
size_t position,
const uint8_t* ringbuffer,
size_t ringbuffer_mask) {
float* literal_costs = self->literal_costs_;
float* cost_dist = self->cost_dist_;
float* cost_cmd = self->cost_cmd_;
size_t num_bytes = self->num_bytes_;
size_t i;
BrotliEstimateBitCostsForLiterals(position, num_bytes, ringbuffer_mask,
ringbuffer, &literal_costs[1]);
literal_costs[0] = 0.0;
for (i = 0; i < num_bytes; ++i) {
literal_costs[i + 1] += literal_costs[i];
}
for (i = 0; i < BROTLI_NUM_COMMAND_SYMBOLS; ++i) {
cost_cmd[i] = (float)FastLog2(11 + (uint32_t)i);
}
for (i = 0; i < BROTLI_NUM_DISTANCE_SYMBOLS; ++i) {
cost_dist[i] = (float)FastLog2(20 + (uint32_t)i);
}
self->min_cost_cmd_ = (float)FastLog2(11);
}
static BROTLI_INLINE float ZopfliCostModelGetCommandCost(
const ZopfliCostModel* self, uint16_t cmdcode) {
return self->cost_cmd_[cmdcode];
}
static BROTLI_INLINE float ZopfliCostModelGetDistanceCost(
const ZopfliCostModel* self, size_t distcode) {
return self->cost_dist_[distcode];
}
static BROTLI_INLINE float ZopfliCostModelGetLiteralCosts(
const ZopfliCostModel* self, size_t from, size_t to) {
return self->literal_costs_[to] - self->literal_costs_[from];
}
static BROTLI_INLINE float ZopfliCostModelGetMinCostCmd(
const ZopfliCostModel* self) {
return self->min_cost_cmd_;
}
static BROTLI_INLINE size_t ComputeDistanceCode(size_t distance,
size_t max_distance,
const int* dist_cache) {
@ -231,440 +42,15 @@ static BROTLI_INLINE size_t ComputeDistanceCode(size_t distance,
return 3;
}
}
return distance + 15;
}
/* REQUIRES: len >= 2, start_pos <= pos */
/* REQUIRES: cost < kInfinity, nodes[start_pos].cost < kInfinity */
/* Maintains the "ZopfliNode array invariant". */
static BROTLI_INLINE void UpdateZopfliNode(ZopfliNode* nodes, size_t pos,
size_t start_pos, size_t len, size_t len_code, size_t dist,
size_t short_code, float cost) {
ZopfliNode* next = &nodes[pos + len];
next->length = (uint32_t)(len | ((len + 9u - len_code) << 24));
next->distance = (uint32_t)(dist | (short_code << 25));
next->insert_length = (uint32_t)(pos - start_pos);
next->u.cost = cost;
}
typedef struct PosData {
size_t pos;
int distance_cache[4];
float costdiff;
float cost;
} PosData;
/* Maintains the smallest 8 cost difference together with their positions */
typedef struct StartPosQueue {
PosData q_[8];
size_t idx_;
} StartPosQueue;
static BROTLI_INLINE void InitStartPosQueue(StartPosQueue* self) {
self->idx_ = 0;
}
static size_t StartPosQueueSize(const StartPosQueue* self) {
return BROTLI_MIN(size_t, self->idx_, 8);
}
static void StartPosQueuePush(StartPosQueue* self, const PosData* posdata) {
size_t offset = ~(self->idx_++) & 7;
size_t len = StartPosQueueSize(self);
size_t i;
PosData* q = self->q_;
q[offset] = *posdata;
/* Restore the sorted order. In the list of |len| items at most |len - 1|
adjacent element comparisons / swaps are required. */
for (i = 1; i < len; ++i) {
if (q[offset & 7].costdiff > q[(offset + 1) & 7].costdiff) {
BROTLI_SWAP(PosData, q, offset & 7, (offset + 1) & 7);
}
++offset;
}
}
static const PosData* StartPosQueueAt(const StartPosQueue* self, size_t k) {
return &self->q_[(k - self->idx_) & 7];
}
/* Returns the minimum possible copy length that can improve the cost of any */
/* future position. */
static size_t ComputeMinimumCopyLength(const float start_cost,
const ZopfliNode* nodes,
const size_t num_bytes,
const size_t pos) {
/* Compute the minimum possible cost of reaching any future position. */
float min_cost = start_cost;
size_t len = 2;
size_t next_len_bucket = 4;
size_t next_len_offset = 10;
while (pos + len <= num_bytes && nodes[pos + len].u.cost <= min_cost) {
/* We already reached (pos + len) with no more cost than the minimum
possible cost of reaching anything from this pos, so there is no point in
looking for lengths <= len. */
++len;
if (len == next_len_offset) {
/* We reached the next copy length code bucket, so we add one more
extra bit to the minimum cost. */
min_cost += 1.0f;
next_len_offset += next_len_bucket;
next_len_bucket *= 2;
}
}
return len;
}
/* REQUIRES: nodes[pos].cost < kInfinity
REQUIRES: nodes[0..pos] satisfies that "ZopfliNode array invariant". */
static uint32_t ComputeDistanceShortcut(const size_t block_start,
const size_t pos,
const size_t max_backward,
const ZopfliNode* nodes) {
const size_t clen = ZopfliNodeCopyLength(&nodes[pos]);
const size_t ilen = nodes[pos].insert_length;
const size_t dist = ZopfliNodeCopyDistance(&nodes[pos]);
/* Since |block_start + pos| is the end position of the command, the copy part
starts from |block_start + pos - clen|. Distances that are greater than
this or greater than |max_backward| are static dictionary references, and
do not update the last distances. Also distance code 0 (last distance)
does not update the last distances. */
if (pos == 0) {
return 0;
} else if (dist + clen <= block_start + pos &&
dist <= max_backward &&
ZopfliNodeDistanceCode(&nodes[pos]) > 0) {
return (uint32_t)pos;
} else {
return nodes[pos - clen - ilen].u.shortcut;
}
}
/* Fills in dist_cache[0..3] with the last four distances (as defined by
Section 4. of the Spec) that would be used at (block_start + pos) if we
used the shortest path of commands from block_start, computed from
nodes[0..pos]. The last four distances at block_start are in
starting_dist_cach[0..3].
REQUIRES: nodes[pos].cost < kInfinity
REQUIRES: nodes[0..pos] satisfies that "ZopfliNode array invariant". */
static void ComputeDistanceCache(const size_t pos,
const int* starting_dist_cache,
const ZopfliNode* nodes,
int* dist_cache) {
int idx = 0;
size_t p = nodes[pos].u.shortcut;
while (idx < 4 && p > 0) {
const size_t ilen = nodes[p].insert_length;
const size_t clen = ZopfliNodeCopyLength(&nodes[p]);
const size_t dist = ZopfliNodeCopyDistance(&nodes[p]);
dist_cache[idx++] = (int)dist;
/* Because of prerequisite, p >= clen + ilen >= 2. */
p = nodes[p - clen - ilen].u.shortcut;
}
for (; idx < 4; ++idx) {
dist_cache[idx] = *starting_dist_cache++;
}
}
static void UpdateNodes(const size_t num_bytes,
const size_t block_start,
const size_t pos,
const uint8_t* ringbuffer,
const size_t ringbuffer_mask,
const BrotliEncoderParams* params,
const size_t max_backward_limit,
const int* starting_dist_cache,
const size_t num_matches,
const BackwardMatch* matches,
const ZopfliCostModel* model,
StartPosQueue* queue,
ZopfliNode* nodes) {
const size_t cur_ix = block_start + pos;
const size_t cur_ix_masked = cur_ix & ringbuffer_mask;
const size_t max_distance = BROTLI_MIN(size_t, cur_ix, max_backward_limit);
const size_t max_len = num_bytes - pos;
const size_t max_zopfli_len = MaxZopfliLen(params);
const size_t max_iters = MaxZopfliCandidates(params);
size_t min_len;
size_t k;
{
/* Save cost, because ComputeDistanceCache invalidates it. */
float node_cost = nodes[pos].u.cost;
nodes[pos].u.shortcut = ComputeDistanceShortcut(
block_start, pos, max_backward_limit, nodes);
if (node_cost <= ZopfliCostModelGetLiteralCosts(model, 0, pos)) {
PosData posdata;
posdata.pos = pos;
posdata.cost = node_cost;
posdata.costdiff = node_cost -
ZopfliCostModelGetLiteralCosts(model, 0, pos);
ComputeDistanceCache(
pos, starting_dist_cache, nodes, posdata.distance_cache);
StartPosQueuePush(queue, &posdata);
}
}
{
const PosData* posdata = StartPosQueueAt(queue, 0);
float min_cost = (posdata->cost + ZopfliCostModelGetMinCostCmd(model) +
ZopfliCostModelGetLiteralCosts(model, posdata->pos, pos));
min_len = ComputeMinimumCopyLength(min_cost, nodes, num_bytes, pos);
}
/* Go over the command starting positions in order of increasing cost
difference. */
for (k = 0; k < max_iters && k < StartPosQueueSize(queue); ++k) {
const PosData* posdata = StartPosQueueAt(queue, k);
const size_t start = posdata->pos;
const uint16_t inscode = GetInsertLengthCode(pos - start);
const float start_costdiff = posdata->costdiff;
const float base_cost = start_costdiff + (float)GetInsertExtra(inscode) +
ZopfliCostModelGetLiteralCosts(model, 0, pos);
/* Look for last distance matches using the distance cache from this
starting position. */
size_t best_len = min_len - 1;
size_t j = 0;
for (; j < BROTLI_NUM_DISTANCE_SHORT_CODES && best_len < max_len; ++j) {
const size_t idx = kDistanceCacheIndex[j];
const size_t backward =
(size_t)(posdata->distance_cache[idx] + kDistanceCacheOffset[j]);
size_t prev_ix = cur_ix - backward;
if (prev_ix >= cur_ix) {
continue;
}
if (PREDICT_FALSE(backward > max_distance)) {
continue;
}
prev_ix &= ringbuffer_mask;
if (cur_ix_masked + best_len > ringbuffer_mask ||
prev_ix + best_len > ringbuffer_mask ||
ringbuffer[cur_ix_masked + best_len] !=
ringbuffer[prev_ix + best_len]) {
continue;
}
{
const size_t len =
FindMatchLengthWithLimit(&ringbuffer[prev_ix],
&ringbuffer[cur_ix_masked],
max_len);
const float dist_cost = base_cost +
ZopfliCostModelGetDistanceCost(model, j);
size_t l;
for (l = best_len + 1; l <= len; ++l) {
const uint16_t copycode = GetCopyLengthCode(l);
const uint16_t cmdcode =
CombineLengthCodes(inscode, copycode, j == 0);
const float cost = (cmdcode < 128 ? base_cost : dist_cost) +
(float)GetCopyExtra(copycode) +
ZopfliCostModelGetCommandCost(model, cmdcode);
if (cost < nodes[pos + l].u.cost) {
UpdateZopfliNode(nodes, pos, start, l, l, backward, j + 1, cost);
}
best_len = l;
}
}
}
/* At higher iterations look only for new last distance matches, since
looking only for new command start positions with the same distances
does not help much. */
if (k >= 2) continue;
{
/* Loop through all possible copy lengths at this position. */
size_t len = min_len;
for (j = 0; j < num_matches; ++j) {
BackwardMatch match = matches[j];
size_t dist = match.distance;
BROTLI_BOOL is_dictionary_match = TO_BROTLI_BOOL(dist > max_distance);
/* We already tried all possible last distance matches, so we can use
normal distance code here. */
size_t dist_code = dist + 15;
uint16_t dist_symbol;
uint32_t distextra;
uint32_t distnumextra;
float dist_cost;
size_t max_match_len;
PrefixEncodeCopyDistance(dist_code, 0, 0, &dist_symbol, &distextra);
distnumextra = distextra >> 24;
dist_cost = base_cost + (float)distnumextra +
ZopfliCostModelGetDistanceCost(model, dist_symbol);
/* Try all copy lengths up until the maximum copy length corresponding
to this distance. If the distance refers to the static dictionary, or
the maximum length is long enough, try only one maximum length. */
max_match_len = BackwardMatchLength(&match);
if (len < max_match_len &&
(is_dictionary_match || max_match_len > max_zopfli_len)) {
len = max_match_len;
}
for (; len <= max_match_len; ++len) {
const size_t len_code =
is_dictionary_match ? BackwardMatchLengthCode(&match) : len;
const uint16_t copycode = GetCopyLengthCode(len_code);
const uint16_t cmdcode = CombineLengthCodes(inscode, copycode, 0);
const float cost = dist_cost + (float)GetCopyExtra(copycode) +
ZopfliCostModelGetCommandCost(model, cmdcode);
if (cost < nodes[pos + len].u.cost) {
UpdateZopfliNode(nodes, pos, start, len, len_code, dist, 0, cost);
}
}
}
}
}
}
static size_t ComputeShortestPathFromNodes(size_t num_bytes,
ZopfliNode* nodes) {
size_t index = num_bytes;
size_t num_commands = 0;
while (nodes[index].insert_length == 0 && nodes[index].length == 1) --index;
nodes[index].u.next = BROTLI_UINT32_MAX;
while (index != 0) {
size_t len = ZopfliNodeCommandLength(&nodes[index]);
index -= len;
nodes[index].u.next = (uint32_t)len;
num_commands++;
}
return num_commands;
}
void BrotliZopfliCreateCommands(const size_t num_bytes,
const size_t block_start,
const size_t max_backward_limit,
const ZopfliNode* nodes,
int* dist_cache,
size_t* last_insert_len,
Command* commands,
size_t* num_literals) {
size_t pos = 0;
uint32_t offset = nodes[0].u.next;
size_t i;
for (i = 0; offset != BROTLI_UINT32_MAX; i++) {
const ZopfliNode* next = &nodes[pos + offset];
size_t copy_length = ZopfliNodeCopyLength(next);
size_t insert_length = next->insert_length;
pos += insert_length;
offset = next->u.next;
if (i == 0) {
insert_length += *last_insert_len;
*last_insert_len = 0;
}
{
size_t distance = ZopfliNodeCopyDistance(next);
size_t len_code = ZopfliNodeLengthCode(next);
size_t max_distance =
BROTLI_MIN(size_t, block_start + pos, max_backward_limit);
BROTLI_BOOL is_dictionary = TO_BROTLI_BOOL(distance > max_distance);
size_t dist_code = ZopfliNodeDistanceCode(next);
InitCommand(
&commands[i], insert_length, copy_length, len_code, dist_code);
if (!is_dictionary && dist_code > 0) {
dist_cache[3] = dist_cache[2];
dist_cache[2] = dist_cache[1];
dist_cache[1] = dist_cache[0];
dist_cache[0] = (int)distance;
}
}
*num_literals += insert_length;
pos += copy_length;
}
*last_insert_len += num_bytes - pos;
}
static size_t ZopfliIterate(size_t num_bytes,
size_t position,
const uint8_t* ringbuffer,
size_t ringbuffer_mask,
const BrotliEncoderParams* params,
const size_t max_backward_limit,
const int* dist_cache,
const ZopfliCostModel* model,
const uint32_t* num_matches,
const BackwardMatch* matches,
ZopfliNode* nodes) {
const size_t max_zopfli_len = MaxZopfliLen(params);
StartPosQueue queue;
size_t cur_match_pos = 0;
size_t i;
nodes[0].length = 0;
nodes[0].u.cost = 0;
InitStartPosQueue(&queue);
for (i = 0; i + 3 < num_bytes; i++) {
UpdateNodes(num_bytes, position, i, ringbuffer, ringbuffer_mask,
params, max_backward_limit, dist_cache, num_matches[i],
&matches[cur_match_pos], model, &queue, nodes);
cur_match_pos += num_matches[i];
/* The zopflification can be too slow in case of very long lengths, so in
such case skip it all, it does not cost a lot of compression ratio. */
if (num_matches[i] == 1 &&
BackwardMatchLength(&matches[cur_match_pos - 1]) > max_zopfli_len) {
i += BackwardMatchLength(&matches[cur_match_pos - 1]) - 1;
InitStartPosQueue(&queue);
}
}
return ComputeShortestPathFromNodes(num_bytes, nodes);
}
size_t BrotliZopfliComputeShortestPath(MemoryManager* m,
size_t num_bytes,
size_t position,
const uint8_t* ringbuffer,
size_t ringbuffer_mask,
const BrotliEncoderParams* params,
const size_t max_backward_limit,
const int* dist_cache,
H10* hasher,
ZopfliNode* nodes) {
const size_t max_zopfli_len = MaxZopfliLen(params);
ZopfliCostModel model;
StartPosQueue queue;
BackwardMatch matches[MAX_NUM_MATCHES_H10];
const size_t store_end = num_bytes >= StoreLookaheadH10() ?
position + num_bytes - StoreLookaheadH10() + 1 : position;
size_t i;
nodes[0].length = 0;
nodes[0].u.cost = 0;
InitZopfliCostModel(m, &model, num_bytes);
if (BROTLI_IS_OOM(m)) return 0;
ZopfliCostModelSetFromLiteralCosts(
&model, position, ringbuffer, ringbuffer_mask);
InitStartPosQueue(&queue);
for (i = 0; i + HashTypeLengthH10() - 1 < num_bytes; i++) {
const size_t pos = position + i;
const size_t max_distance = BROTLI_MIN(size_t, pos, max_backward_limit);
size_t num_matches = FindAllMatchesH10(hasher, ringbuffer, ringbuffer_mask,
pos, num_bytes - i, max_distance, params, matches);
if (num_matches > 0 &&
BackwardMatchLength(&matches[num_matches - 1]) > max_zopfli_len) {
matches[0] = matches[num_matches - 1];
num_matches = 1;
}
UpdateNodes(num_bytes, position, i, ringbuffer, ringbuffer_mask,
params, max_backward_limit, dist_cache, num_matches, matches,
&model, &queue, nodes);
if (num_matches == 1 && BackwardMatchLength(&matches[0]) > max_zopfli_len) {
/* Add the tail of the copy to the hasher. */
StoreRangeH10(hasher, ringbuffer, ringbuffer_mask, pos + 1, BROTLI_MIN(
size_t, pos + BackwardMatchLength(&matches[0]), store_end));
i += BackwardMatchLength(&matches[0]) - 1;
InitStartPosQueue(&queue);
}
}
CleanupZopfliCostModel(m, &model);
return ComputeShortestPathFromNodes(num_bytes, nodes);
return distance + BROTLI_NUM_DISTANCE_SHORT_CODES - 1;
}
#define EXPAND_CAT(a, b) CAT(a, b)
#define CAT(a, b) a ## b
#define FN(X) EXPAND_CAT(X, HASHER())
#define EXPORT_FN(X) EXPAND_CAT(X, EXPAND_CAT(PREFIX(), HASHER()))
#define PREFIX() N
#define HASHER() H2
/* NOLINTNEXTLINE(build/include) */
@ -691,21 +77,6 @@ size_t BrotliZopfliComputeShortestPath(MemoryManager* m,
#include "./backward_references_inc.h"
#undef HASHER
#define HASHER() H7
/* NOLINTNEXTLINE(build/include) */
#include "./backward_references_inc.h"
#undef HASHER
#define HASHER() H8
/* NOLINTNEXTLINE(build/include) */
#include "./backward_references_inc.h"
#undef HASHER
#define HASHER() H9
/* NOLINTNEXTLINE(build/include) */
#include "./backward_references_inc.h"
#undef HASHER
#define HASHER() H40
/* NOLINTNEXTLINE(build/include) */
#include "./backward_references_inc.h"
@ -721,170 +92,51 @@ size_t BrotliZopfliComputeShortestPath(MemoryManager* m,
#include "./backward_references_inc.h"
#undef HASHER
#define HASHER() H54
/* NOLINTNEXTLINE(build/include) */
#include "./backward_references_inc.h"
#undef HASHER
#define HASHER() H35
/* NOLINTNEXTLINE(build/include) */
#include "./backward_references_inc.h"
#undef HASHER
#define HASHER() H55
/* NOLINTNEXTLINE(build/include) */
#include "./backward_references_inc.h"
#undef HASHER
#define HASHER() H65
/* NOLINTNEXTLINE(build/include) */
#include "./backward_references_inc.h"
#undef HASHER
#undef PREFIX
#undef EXPORT_FN
#undef FN
#undef CAT
#undef EXPAND_CAT
static BROTLI_NOINLINE void CreateZopfliBackwardReferences(
MemoryManager* m, size_t num_bytes, size_t position, BROTLI_BOOL is_last,
const uint8_t* ringbuffer, size_t ringbuffer_mask,
const BrotliEncoderParams* params, H10* hasher, int* dist_cache,
size_t* last_insert_len, Command* commands, size_t* num_commands,
size_t* num_literals) {
const size_t max_backward_limit = MaxBackwardLimit(params->lgwin);
ZopfliNode* nodes;
InitH10(m, hasher, ringbuffer, params, position, num_bytes, is_last);
if (BROTLI_IS_OOM(m)) return;
StitchToPreviousBlockH10(hasher, num_bytes, position,
ringbuffer, ringbuffer_mask);
nodes = BROTLI_ALLOC(m, ZopfliNode, num_bytes + 1);
if (BROTLI_IS_OOM(m)) return;
BrotliInitZopfliNodes(nodes, num_bytes + 1);
*num_commands += BrotliZopfliComputeShortestPath(m, num_bytes, position,
ringbuffer, ringbuffer_mask, params, max_backward_limit,
dist_cache, hasher, nodes);
if (BROTLI_IS_OOM(m)) return;
BrotliZopfliCreateCommands(num_bytes, position, max_backward_limit, nodes,
dist_cache, last_insert_len, commands, num_literals);
BROTLI_FREE(m, nodes);
}
static BROTLI_NOINLINE void CreateHqZopfliBackwardReferences(
MemoryManager* m, size_t num_bytes, size_t position, BROTLI_BOOL is_last,
const uint8_t* ringbuffer, size_t ringbuffer_mask,
const BrotliEncoderParams* params, H10* hasher, int* dist_cache,
size_t* last_insert_len, Command* commands, size_t* num_commands,
size_t* num_literals) {
const size_t max_backward_limit = MaxBackwardLimit(params->lgwin);
uint32_t* num_matches = BROTLI_ALLOC(m, uint32_t, num_bytes);
size_t matches_size = 4 * num_bytes;
const size_t store_end = num_bytes >= StoreLookaheadH10() ?
position + num_bytes - StoreLookaheadH10() + 1 : position;
size_t cur_match_pos = 0;
size_t i;
size_t orig_num_literals;
size_t orig_last_insert_len;
int orig_dist_cache[4];
size_t orig_num_commands;
ZopfliCostModel model;
ZopfliNode* nodes;
BackwardMatch* matches = BROTLI_ALLOC(m, BackwardMatch, matches_size);
if (BROTLI_IS_OOM(m)) return;
InitH10(m, hasher, ringbuffer, params, position, num_bytes, is_last);
if (BROTLI_IS_OOM(m)) return;
StitchToPreviousBlockH10(hasher, num_bytes, position,
ringbuffer, ringbuffer_mask);
for (i = 0; i + HashTypeLengthH10() - 1 < num_bytes; ++i) {
const size_t pos = position + i;
size_t max_distance = BROTLI_MIN(size_t, pos, max_backward_limit);
size_t max_length = num_bytes - i;
size_t num_found_matches;
size_t cur_match_end;
size_t j;
/* Ensure that we have enough free slots. */
BROTLI_ENSURE_CAPACITY(m, BackwardMatch, matches, matches_size,
cur_match_pos + MAX_NUM_MATCHES_H10);
if (BROTLI_IS_OOM(m)) return;
num_found_matches = FindAllMatchesH10(hasher, ringbuffer, ringbuffer_mask,
pos, max_length, max_distance, params, &matches[cur_match_pos]);
cur_match_end = cur_match_pos + num_found_matches;
for (j = cur_match_pos; j + 1 < cur_match_end; ++j) {
assert(BackwardMatchLength(&matches[j]) <
BackwardMatchLength(&matches[j + 1]));
assert(matches[j].distance > max_distance ||
matches[j].distance <= matches[j + 1].distance);
}
num_matches[i] = (uint32_t)num_found_matches;
if (num_found_matches > 0) {
const size_t match_len = BackwardMatchLength(&matches[cur_match_end - 1]);
if (match_len > MAX_ZOPFLI_LEN_QUALITY_11) {
const size_t skip = match_len - 1;
matches[cur_match_pos++] = matches[cur_match_end - 1];
num_matches[i] = 1;
/* Add the tail of the copy to the hasher. */
StoreRangeH10(hasher, ringbuffer, ringbuffer_mask, pos + 1,
BROTLI_MIN(size_t, pos + match_len, store_end));
memset(&num_matches[i + 1], 0, skip * sizeof(num_matches[0]));
i += skip;
} else {
cur_match_pos = cur_match_end;
}
}
}
orig_num_literals = *num_literals;
orig_last_insert_len = *last_insert_len;
memcpy(orig_dist_cache, dist_cache, 4 * sizeof(dist_cache[0]));
orig_num_commands = *num_commands;
nodes = BROTLI_ALLOC(m, ZopfliNode, num_bytes + 1);
if (BROTLI_IS_OOM(m)) return;
InitZopfliCostModel(m, &model, num_bytes);
if (BROTLI_IS_OOM(m)) return;
for (i = 0; i < 2; i++) {
BrotliInitZopfliNodes(nodes, num_bytes + 1);
if (i == 0) {
ZopfliCostModelSetFromLiteralCosts(
&model, position, ringbuffer, ringbuffer_mask);
} else {
ZopfliCostModelSetFromCommands(&model, position, ringbuffer,
ringbuffer_mask, commands, *num_commands - orig_num_commands,
orig_last_insert_len);
}
*num_commands = orig_num_commands;
*num_literals = orig_num_literals;
*last_insert_len = orig_last_insert_len;
memcpy(dist_cache, orig_dist_cache, 4 * sizeof(dist_cache[0]));
*num_commands += ZopfliIterate(num_bytes, position, ringbuffer,
ringbuffer_mask, params, max_backward_limit, dist_cache,
&model, num_matches, matches, nodes);
BrotliZopfliCreateCommands(num_bytes, position, max_backward_limit,
nodes, dist_cache, last_insert_len, commands, num_literals);
}
CleanupZopfliCostModel(m, &model);
BROTLI_FREE(m, nodes);
BROTLI_FREE(m, matches);
BROTLI_FREE(m, num_matches);
}
void BrotliCreateBackwardReferences(MemoryManager* m,
size_t num_bytes,
size_t position,
BROTLI_BOOL is_last,
const uint8_t* ringbuffer,
size_t ringbuffer_mask,
const BrotliEncoderParams* params,
Hashers* hashers,
int* dist_cache,
size_t* last_insert_len,
Command* commands,
size_t* num_commands,
size_t* num_literals) {
if (params->quality == ZOPFLIFICATION_QUALITY) {
CreateZopfliBackwardReferences(
m, num_bytes, position, is_last, ringbuffer, ringbuffer_mask,
params, hashers->h10, dist_cache,
last_insert_len, commands, num_commands, num_literals);
return;
} else if (params->quality == HQ_ZOPFLIFICATION_QUALITY) {
CreateHqZopfliBackwardReferences(
m, num_bytes, position, is_last, ringbuffer, ringbuffer_mask,
params, hashers->h10, dist_cache,
last_insert_len, commands, num_commands, num_literals);
return;
}
switch (ChooseHasher(params)) {
#define _CASE(N) \
case N: \
CreateBackwardReferencesH ## N(m, num_bytes, position, is_last, \
ringbuffer, ringbuffer_mask, params, hashers->h ## N, dist_cache, \
last_insert_len, commands, num_commands, num_literals); \
break;
FOR_GENERIC_HASHERS(_CASE)
#undef _CASE
void BrotliCreateBackwardReferences(
size_t num_bytes, size_t position, const uint8_t* ringbuffer,
size_t ringbuffer_mask, const BrotliEncoderParams* params,
HasherHandle hasher, int* dist_cache, size_t* last_insert_len,
Command* commands, size_t* num_commands, size_t* num_literals) {
switch (params->hasher.type) {
#define CASE_(N) \
case N: \
CreateBackwardReferencesNH ## N( \
num_bytes, position, ringbuffer, \
ringbuffer_mask, params, hasher, dist_cache, \
last_insert_len, commands, num_commands, num_literals); \
return;
FOR_GENERIC_HASHERS(CASE_)
#undef CASE_
default:
break;
}
if (BROTLI_IS_OOM(m)) return;
}
#if defined(__cplusplus) || defined(c_plusplus)

View File

@ -9,11 +9,12 @@
#ifndef BROTLI_ENC_BACKWARD_REFERENCES_H_
#define BROTLI_ENC_BACKWARD_REFERENCES_H_
#include "../common/types.h"
#include "../common/constants.h"
#include "../common/dictionary.h"
#include "../common/platform.h"
#include <brotli/types.h>
#include "./command.h"
#include "./hash.h"
#include "./memory.h"
#include "./port.h"
#include "./quality.h"
#if defined(__cplusplus) || defined(c_plusplus)
@ -25,72 +26,10 @@ extern "C" {
CreateBackwardReferences calls, and must be incremented by the amount written
by this call. */
BROTLI_INTERNAL void BrotliCreateBackwardReferences(
MemoryManager* m, size_t num_bytes, size_t position, BROTLI_BOOL is_last,
const uint8_t* ringbuffer, size_t ringbuffer_mask,
const BrotliEncoderParams* params, Hashers* hashers, int* dist_cache,
size_t* last_insert_len, Command* commands, size_t* num_commands,
size_t* num_literals);
typedef struct ZopfliNode {
/* best length to get up to this byte (not including this byte itself)
highest 8 bit is used to reconstruct the length code */
uint32_t length;
/* distance associated with the length
highest 7 bit contains distance short code + 1 (or zero if no short code)
*/
uint32_t distance;
/* number of literal inserts before this copy */
uint32_t insert_length;
/* This union holds information used by dynamic-programming. During forward
pass |cost| it used to store the goal function. When node is processed its
|cost| is invalidated in favor of |shortcut|. On path backtracing pass
|next| is assigned the offset to next node on the path. */
union {
/* Smallest cost to get to this byte from the beginning, as found so far. */
float cost;
/* Offset to the next node on the path. Equals to command_length() of the
next node on the path. For last node equals to BROTLI_UINT32_MAX */
uint32_t next;
/* Node position that provides next distance for distance cache. */
uint32_t shortcut;
} u;
} ZopfliNode;
BROTLI_INTERNAL void BrotliInitZopfliNodes(ZopfliNode* array, size_t length);
/* Computes the shortest path of commands from position to at most
position + num_bytes.
On return, path->size() is the number of commands found and path[i] is the
length of the ith command (copy length plus insert length).
Note that the sum of the lengths of all commands can be less than num_bytes.
On return, the nodes[0..num_bytes] array will have the following
"ZopfliNode array invariant":
For each i in [1..num_bytes], if nodes[i].cost < kInfinity, then
(1) nodes[i].copy_length() >= 2
(2) nodes[i].command_length() <= i and
(3) nodes[i - nodes[i].command_length()].cost < kInfinity */
BROTLI_INTERNAL size_t BrotliZopfliComputeShortestPath(
MemoryManager* m, size_t num_bytes, size_t position,
const uint8_t* ringbuffer, size_t ringbuffer_mask,
const BrotliEncoderParams* params, const size_t max_backward_limit,
const int* dist_cache, H10* hasher, ZopfliNode* nodes);
BROTLI_INTERNAL void BrotliZopfliCreateCommands(const size_t num_bytes,
const size_t block_start,
const size_t max_backward_limit,
const ZopfliNode* nodes,
int* dist_cache,
size_t* last_insert_len,
Command* commands,
size_t* num_literals);
/* Maximum distance, see section 9.1. of the spec. */
static BROTLI_INLINE size_t MaxBackwardLimit(int lgwin) {
return (1u << lgwin) - 16;
}
size_t num_bytes, size_t position, const uint8_t* ringbuffer,
size_t ringbuffer_mask, const BrotliEncoderParams* params,
HasherHandle hasher, int* dist_cache, size_t* last_insert_len,
Command* commands, size_t* num_commands, size_t* num_literals);
#if defined(__cplusplus) || defined(c_plusplus)
} /* extern "C" */

View File

@ -0,0 +1,830 @@
/* Copyright 2013 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* Function to find backward reference copies. */
#include "./backward_references_hq.h"
#include <string.h> /* memcpy, memset */
#include "../common/constants.h"
#include "../common/platform.h"
#include <brotli/types.h>
#include "./command.h"
#include "./fast_log.h"
#include "./find_match_length.h"
#include "./literal_cost.h"
#include "./memory.h"
#include "./params.h"
#include "./prefix.h"
#include "./quality.h"
#if defined(__cplusplus) || defined(c_plusplus)
extern "C" {
#endif
#define BROTLI_MAX_EFFECTIVE_DISTANCE_ALPHABET_SIZE 544
static const float kInfinity = 1.7e38f; /* ~= 2 ^ 127 */
static const uint32_t kDistanceCacheIndex[] = {
0, 1, 2, 3, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1,
};
static const int kDistanceCacheOffset[] = {
0, 0, 0, 0, -1, 1, -2, 2, -3, 3, -1, 1, -2, 2, -3, 3
};
void BrotliInitZopfliNodes(ZopfliNode* array, size_t length) {
ZopfliNode stub;
size_t i;
stub.length = 1;
stub.distance = 0;
stub.dcode_insert_length = 0;
stub.u.cost = kInfinity;
for (i = 0; i < length; ++i) array[i] = stub;
}
static BROTLI_INLINE uint32_t ZopfliNodeCopyLength(const ZopfliNode* self) {
return self->length & 0x1FFFFFF;
}
static BROTLI_INLINE uint32_t ZopfliNodeLengthCode(const ZopfliNode* self) {
const uint32_t modifier = self->length >> 25;
return ZopfliNodeCopyLength(self) + 9u - modifier;
}
static BROTLI_INLINE uint32_t ZopfliNodeCopyDistance(const ZopfliNode* self) {
return self->distance;
}
static BROTLI_INLINE uint32_t ZopfliNodeDistanceCode(const ZopfliNode* self) {
const uint32_t short_code = self->dcode_insert_length >> 27;
return short_code == 0 ?
ZopfliNodeCopyDistance(self) + BROTLI_NUM_DISTANCE_SHORT_CODES - 1 :
short_code - 1;
}
static BROTLI_INLINE uint32_t ZopfliNodeCommandLength(const ZopfliNode* self) {
return ZopfliNodeCopyLength(self) + (self->dcode_insert_length & 0x7FFFFFF);
}
/* Histogram based cost model for zopflification. */
typedef struct ZopfliCostModel {
/* The insert and copy length symbols. */
float cost_cmd_[BROTLI_NUM_COMMAND_SYMBOLS];
float* cost_dist_;
uint32_t distance_histogram_size;
/* Cumulative costs of literals per position in the stream. */
float* literal_costs_;
float min_cost_cmd_;
size_t num_bytes_;
} ZopfliCostModel;
static void InitZopfliCostModel(
MemoryManager* m, ZopfliCostModel* self, const BrotliDistanceParams* dist,
size_t num_bytes) {
uint32_t distance_histogram_size = dist->alphabet_size;
if (distance_histogram_size > BROTLI_MAX_EFFECTIVE_DISTANCE_ALPHABET_SIZE) {
distance_histogram_size = BROTLI_MAX_EFFECTIVE_DISTANCE_ALPHABET_SIZE;
}
self->num_bytes_ = num_bytes;
self->literal_costs_ = BROTLI_ALLOC(m, float, num_bytes + 2);
self->cost_dist_ = BROTLI_ALLOC(m, float, dist->alphabet_size);
self->distance_histogram_size = distance_histogram_size;
if (BROTLI_IS_OOM(m)) return;
}
static void CleanupZopfliCostModel(MemoryManager* m, ZopfliCostModel* self) {
BROTLI_FREE(m, self->literal_costs_);
BROTLI_FREE(m, self->cost_dist_);
}
static void SetCost(const uint32_t* histogram, size_t histogram_size,
BROTLI_BOOL literal_histogram, float* cost) {
size_t sum = 0;
size_t missing_symbol_sum;
float log2sum;
float missing_symbol_cost;
size_t i;
for (i = 0; i < histogram_size; i++) {
sum += histogram[i];
}
log2sum = (float)FastLog2(sum);
missing_symbol_sum = sum;
if (!literal_histogram) {
for (i = 0; i < histogram_size; i++) {
if (histogram[i] == 0) missing_symbol_sum++;
}
}
missing_symbol_cost = (float)FastLog2(missing_symbol_sum) + 2;
for (i = 0; i < histogram_size; i++) {
if (histogram[i] == 0) {
cost[i] = missing_symbol_cost;
continue;
}
/* Shannon bits for this symbol. */
cost[i] = log2sum - (float)FastLog2(histogram[i]);
/* Cannot be coded with less than 1 bit */
if (cost[i] < 1) cost[i] = 1;
}
}
static void ZopfliCostModelSetFromCommands(ZopfliCostModel* self,
size_t position,
const uint8_t* ringbuffer,
size_t ringbuffer_mask,
const Command* commands,
size_t num_commands,
size_t last_insert_len) {
uint32_t histogram_literal[BROTLI_NUM_LITERAL_SYMBOLS];
uint32_t histogram_cmd[BROTLI_NUM_COMMAND_SYMBOLS];
uint32_t histogram_dist[BROTLI_MAX_EFFECTIVE_DISTANCE_ALPHABET_SIZE];
float cost_literal[BROTLI_NUM_LITERAL_SYMBOLS];
size_t pos = position - last_insert_len;
float min_cost_cmd = kInfinity;
size_t i;
float* cost_cmd = self->cost_cmd_;
memset(histogram_literal, 0, sizeof(histogram_literal));
memset(histogram_cmd, 0, sizeof(histogram_cmd));
memset(histogram_dist, 0, sizeof(histogram_dist));
for (i = 0; i < num_commands; i++) {
size_t inslength = commands[i].insert_len_;
size_t copylength = CommandCopyLen(&commands[i]);
size_t distcode = commands[i].dist_prefix_ & 0x3FF;
size_t cmdcode = commands[i].cmd_prefix_;
size_t j;
histogram_cmd[cmdcode]++;
if (cmdcode >= 128) histogram_dist[distcode]++;
for (j = 0; j < inslength; j++) {
histogram_literal[ringbuffer[(pos + j) & ringbuffer_mask]]++;
}
pos += inslength + copylength;
}
SetCost(histogram_literal, BROTLI_NUM_LITERAL_SYMBOLS, BROTLI_TRUE,
cost_literal);
SetCost(histogram_cmd, BROTLI_NUM_COMMAND_SYMBOLS, BROTLI_FALSE,
cost_cmd);
SetCost(histogram_dist, self->distance_histogram_size, BROTLI_FALSE,
self->cost_dist_);
for (i = 0; i < BROTLI_NUM_COMMAND_SYMBOLS; ++i) {
min_cost_cmd = BROTLI_MIN(float, min_cost_cmd, cost_cmd[i]);
}
self->min_cost_cmd_ = min_cost_cmd;
{
float* literal_costs = self->literal_costs_;
float literal_carry = 0.0;
size_t num_bytes = self->num_bytes_;
literal_costs[0] = 0.0;
for (i = 0; i < num_bytes; ++i) {
literal_carry +=
cost_literal[ringbuffer[(position + i) & ringbuffer_mask]];
literal_costs[i + 1] = literal_costs[i] + literal_carry;
literal_carry -= literal_costs[i + 1] - literal_costs[i];
}
}
}
static void ZopfliCostModelSetFromLiteralCosts(ZopfliCostModel* self,
size_t position,
const uint8_t* ringbuffer,
size_t ringbuffer_mask) {
float* literal_costs = self->literal_costs_;
float literal_carry = 0.0;
float* cost_dist = self->cost_dist_;
float* cost_cmd = self->cost_cmd_;
size_t num_bytes = self->num_bytes_;
size_t i;
BrotliEstimateBitCostsForLiterals(position, num_bytes, ringbuffer_mask,
ringbuffer, &literal_costs[1]);
literal_costs[0] = 0.0;
for (i = 0; i < num_bytes; ++i) {
literal_carry += literal_costs[i + 1];
literal_costs[i + 1] = literal_costs[i] + literal_carry;
literal_carry -= literal_costs[i + 1] - literal_costs[i];
}
for (i = 0; i < BROTLI_NUM_COMMAND_SYMBOLS; ++i) {
cost_cmd[i] = (float)FastLog2(11 + (uint32_t)i);
}
for (i = 0; i < self->distance_histogram_size; ++i) {
cost_dist[i] = (float)FastLog2(20 + (uint32_t)i);
}
self->min_cost_cmd_ = (float)FastLog2(11);
}
static BROTLI_INLINE float ZopfliCostModelGetCommandCost(
const ZopfliCostModel* self, uint16_t cmdcode) {
return self->cost_cmd_[cmdcode];
}
static BROTLI_INLINE float ZopfliCostModelGetDistanceCost(
const ZopfliCostModel* self, size_t distcode) {
return self->cost_dist_[distcode];
}
static BROTLI_INLINE float ZopfliCostModelGetLiteralCosts(
const ZopfliCostModel* self, size_t from, size_t to) {
return self->literal_costs_[to] - self->literal_costs_[from];
}
static BROTLI_INLINE float ZopfliCostModelGetMinCostCmd(
const ZopfliCostModel* self) {
return self->min_cost_cmd_;
}
/* REQUIRES: len >= 2, start_pos <= pos */
/* REQUIRES: cost < kInfinity, nodes[start_pos].cost < kInfinity */
/* Maintains the "ZopfliNode array invariant". */
static BROTLI_INLINE void UpdateZopfliNode(ZopfliNode* nodes, size_t pos,
size_t start_pos, size_t len, size_t len_code, size_t dist,
size_t short_code, float cost) {
ZopfliNode* next = &nodes[pos + len];
next->length = (uint32_t)(len | ((len + 9u - len_code) << 25));
next->distance = (uint32_t)dist;
next->dcode_insert_length = (uint32_t)(
(short_code << 27) | (pos - start_pos));
next->u.cost = cost;
}
typedef struct PosData {
size_t pos;
int distance_cache[4];
float costdiff;
float cost;
} PosData;
/* Maintains the smallest 8 cost difference together with their positions */
typedef struct StartPosQueue {
PosData q_[8];
size_t idx_;
} StartPosQueue;
static BROTLI_INLINE void InitStartPosQueue(StartPosQueue* self) {
self->idx_ = 0;
}
static size_t StartPosQueueSize(const StartPosQueue* self) {
return BROTLI_MIN(size_t, self->idx_, 8);
}
static void StartPosQueuePush(StartPosQueue* self, const PosData* posdata) {
size_t offset = ~(self->idx_++) & 7;
size_t len = StartPosQueueSize(self);
size_t i;
PosData* q = self->q_;
q[offset] = *posdata;
/* Restore the sorted order. In the list of |len| items at most |len - 1|
adjacent element comparisons / swaps are required. */
for (i = 1; i < len; ++i) {
if (q[offset & 7].costdiff > q[(offset + 1) & 7].costdiff) {
BROTLI_SWAP(PosData, q, offset & 7, (offset + 1) & 7);
}
++offset;
}
}
static const PosData* StartPosQueueAt(const StartPosQueue* self, size_t k) {
return &self->q_[(k - self->idx_) & 7];
}
/* Returns the minimum possible copy length that can improve the cost of any */
/* future position. */
static size_t ComputeMinimumCopyLength(const float start_cost,
const ZopfliNode* nodes,
const size_t num_bytes,
const size_t pos) {
/* Compute the minimum possible cost of reaching any future position. */
float min_cost = start_cost;
size_t len = 2;
size_t next_len_bucket = 4;
size_t next_len_offset = 10;
while (pos + len <= num_bytes && nodes[pos + len].u.cost <= min_cost) {
/* We already reached (pos + len) with no more cost than the minimum
possible cost of reaching anything from this pos, so there is no point in
looking for lengths <= len. */
++len;
if (len == next_len_offset) {
/* We reached the next copy length code bucket, so we add one more
extra bit to the minimum cost. */
min_cost += 1.0f;
next_len_offset += next_len_bucket;
next_len_bucket *= 2;
}
}
return len;
}
/* REQUIRES: nodes[pos].cost < kInfinity
REQUIRES: nodes[0..pos] satisfies that "ZopfliNode array invariant". */
static uint32_t ComputeDistanceShortcut(const size_t block_start,
const size_t pos,
const size_t max_backward,
const size_t gap,
const ZopfliNode* nodes) {
const size_t clen = ZopfliNodeCopyLength(&nodes[pos]);
const size_t ilen = nodes[pos].dcode_insert_length & 0x7FFFFFF;
const size_t dist = ZopfliNodeCopyDistance(&nodes[pos]);
/* Since |block_start + pos| is the end position of the command, the copy part
starts from |block_start + pos - clen|. Distances that are greater than
this or greater than |max_backward| are static dictionary references, and
do not update the last distances. Also distance code 0 (last distance)
does not update the last distances. */
if (pos == 0) {
return 0;
} else if (dist + clen <= block_start + pos + gap &&
dist <= max_backward + gap &&
ZopfliNodeDistanceCode(&nodes[pos]) > 0) {
return (uint32_t)pos;
} else {
return nodes[pos - clen - ilen].u.shortcut;
}
}
/* Fills in dist_cache[0..3] with the last four distances (as defined by
Section 4. of the Spec) that would be used at (block_start + pos) if we
used the shortest path of commands from block_start, computed from
nodes[0..pos]. The last four distances at block_start are in
starting_dist_cache[0..3].
REQUIRES: nodes[pos].cost < kInfinity
REQUIRES: nodes[0..pos] satisfies that "ZopfliNode array invariant". */
static void ComputeDistanceCache(const size_t pos,
const int* starting_dist_cache,
const ZopfliNode* nodes,
int* dist_cache) {
int idx = 0;
size_t p = nodes[pos].u.shortcut;
while (idx < 4 && p > 0) {
const size_t ilen = nodes[p].dcode_insert_length & 0x7FFFFFF;
const size_t clen = ZopfliNodeCopyLength(&nodes[p]);
const size_t dist = ZopfliNodeCopyDistance(&nodes[p]);
dist_cache[idx++] = (int)dist;
/* Because of prerequisite, p >= clen + ilen >= 2. */
p = nodes[p - clen - ilen].u.shortcut;
}
for (; idx < 4; ++idx) {
dist_cache[idx] = *starting_dist_cache++;
}
}
/* Maintains "ZopfliNode array invariant" and pushes node to the queue, if it
is eligible. */
static void EvaluateNode(
const size_t block_start, const size_t pos, const size_t max_backward_limit,
const size_t gap, const int* starting_dist_cache,
const ZopfliCostModel* model, StartPosQueue* queue, ZopfliNode* nodes) {
/* Save cost, because ComputeDistanceCache invalidates it. */
float node_cost = nodes[pos].u.cost;
nodes[pos].u.shortcut = ComputeDistanceShortcut(
block_start, pos, max_backward_limit, gap, nodes);
if (node_cost <= ZopfliCostModelGetLiteralCosts(model, 0, pos)) {
PosData posdata;
posdata.pos = pos;
posdata.cost = node_cost;
posdata.costdiff = node_cost -
ZopfliCostModelGetLiteralCosts(model, 0, pos);
ComputeDistanceCache(
pos, starting_dist_cache, nodes, posdata.distance_cache);
StartPosQueuePush(queue, &posdata);
}
}
/* Returns longest copy length. */
static size_t UpdateNodes(
const size_t num_bytes, const size_t block_start, const size_t pos,
const uint8_t* ringbuffer, const size_t ringbuffer_mask,
const BrotliEncoderParams* params, const size_t max_backward_limit,
const int* starting_dist_cache, const size_t num_matches,
const BackwardMatch* matches, const ZopfliCostModel* model,
StartPosQueue* queue, ZopfliNode* nodes) {
const size_t cur_ix = block_start + pos;
const size_t cur_ix_masked = cur_ix & ringbuffer_mask;
const size_t max_distance = BROTLI_MIN(size_t, cur_ix, max_backward_limit);
const size_t max_len = num_bytes - pos;
const size_t max_zopfli_len = MaxZopfliLen(params);
const size_t max_iters = MaxZopfliCandidates(params);
size_t min_len;
size_t result = 0;
size_t k;
size_t gap = 0;
EvaluateNode(block_start, pos, max_backward_limit, gap, starting_dist_cache,
model, queue, nodes);
{
const PosData* posdata = StartPosQueueAt(queue, 0);
float min_cost = (posdata->cost + ZopfliCostModelGetMinCostCmd(model) +
ZopfliCostModelGetLiteralCosts(model, posdata->pos, pos));
min_len = ComputeMinimumCopyLength(min_cost, nodes, num_bytes, pos);
}
/* Go over the command starting positions in order of increasing cost
difference. */
for (k = 0; k < max_iters && k < StartPosQueueSize(queue); ++k) {
const PosData* posdata = StartPosQueueAt(queue, k);
const size_t start = posdata->pos;
const uint16_t inscode = GetInsertLengthCode(pos - start);
const float start_costdiff = posdata->costdiff;
const float base_cost = start_costdiff + (float)GetInsertExtra(inscode) +
ZopfliCostModelGetLiteralCosts(model, 0, pos);
/* Look for last distance matches using the distance cache from this
starting position. */
size_t best_len = min_len - 1;
size_t j = 0;
for (; j < BROTLI_NUM_DISTANCE_SHORT_CODES && best_len < max_len; ++j) {
const size_t idx = kDistanceCacheIndex[j];
const size_t backward =
(size_t)(posdata->distance_cache[idx] + kDistanceCacheOffset[j]);
size_t prev_ix = cur_ix - backward;
size_t len = 0;
uint8_t continuation = ringbuffer[cur_ix_masked + best_len];
if (cur_ix_masked + best_len > ringbuffer_mask) {
break;
}
if (BROTLI_PREDICT_FALSE(backward > max_distance + gap)) {
continue;
}
if (backward <= max_distance) {
if (prev_ix >= cur_ix) {
continue;
}
prev_ix &= ringbuffer_mask;
if (prev_ix + best_len > ringbuffer_mask ||
continuation != ringbuffer[prev_ix + best_len]) {
continue;
}
len = FindMatchLengthWithLimit(&ringbuffer[prev_ix],
&ringbuffer[cur_ix_masked],
max_len);
} else {
continue;
}
{
const float dist_cost = base_cost +
ZopfliCostModelGetDistanceCost(model, j);
size_t l;
for (l = best_len + 1; l <= len; ++l) {
const uint16_t copycode = GetCopyLengthCode(l);
const uint16_t cmdcode =
CombineLengthCodes(inscode, copycode, j == 0);
const float cost = (cmdcode < 128 ? base_cost : dist_cost) +
(float)GetCopyExtra(copycode) +
ZopfliCostModelGetCommandCost(model, cmdcode);
if (cost < nodes[pos + l].u.cost) {
UpdateZopfliNode(nodes, pos, start, l, l, backward, j + 1, cost);
result = BROTLI_MAX(size_t, result, l);
}
best_len = l;
}
}
}
/* At higher iterations look only for new last distance matches, since
looking only for new command start positions with the same distances
does not help much. */
if (k >= 2) continue;
{
/* Loop through all possible copy lengths at this position. */
size_t len = min_len;
for (j = 0; j < num_matches; ++j) {
BackwardMatch match = matches[j];
size_t dist = match.distance;
BROTLI_BOOL is_dictionary_match =
TO_BROTLI_BOOL(dist > max_distance + gap);
/* We already tried all possible last distance matches, so we can use
normal distance code here. */
size_t dist_code = dist + BROTLI_NUM_DISTANCE_SHORT_CODES - 1;
uint16_t dist_symbol;
uint32_t distextra;
uint32_t distnumextra;
float dist_cost;
size_t max_match_len;
PrefixEncodeCopyDistance(
dist_code, params->dist.num_direct_distance_codes,
params->dist.distance_postfix_bits, &dist_symbol, &distextra);
distnumextra = dist_symbol >> 10;
dist_cost = base_cost + (float)distnumextra +
ZopfliCostModelGetDistanceCost(model, dist_symbol & 0x3FF);
/* Try all copy lengths up until the maximum copy length corresponding
to this distance. If the distance refers to the static dictionary, or
the maximum length is long enough, try only one maximum length. */
max_match_len = BackwardMatchLength(&match);
if (len < max_match_len &&
(is_dictionary_match || max_match_len > max_zopfli_len)) {
len = max_match_len;
}
for (; len <= max_match_len; ++len) {
const size_t len_code =
is_dictionary_match ? BackwardMatchLengthCode(&match) : len;
const uint16_t copycode = GetCopyLengthCode(len_code);
const uint16_t cmdcode = CombineLengthCodes(inscode, copycode, 0);
const float cost = dist_cost + (float)GetCopyExtra(copycode) +
ZopfliCostModelGetCommandCost(model, cmdcode);
if (cost < nodes[pos + len].u.cost) {
UpdateZopfliNode(nodes, pos, start, len, len_code, dist, 0, cost);
result = BROTLI_MAX(size_t, result, len);
}
}
}
}
}
return result;
}
static size_t ComputeShortestPathFromNodes(size_t num_bytes,
ZopfliNode* nodes) {
size_t index = num_bytes;
size_t num_commands = 0;
while ((nodes[index].dcode_insert_length & 0x7FFFFFF) == 0 &&
nodes[index].length == 1) --index;
nodes[index].u.next = BROTLI_UINT32_MAX;
while (index != 0) {
size_t len = ZopfliNodeCommandLength(&nodes[index]);
index -= len;
nodes[index].u.next = (uint32_t)len;
num_commands++;
}
return num_commands;
}
/* REQUIRES: nodes != NULL and len(nodes) >= num_bytes + 1 */
void BrotliZopfliCreateCommands(const size_t num_bytes,
const size_t block_start,
const size_t max_backward_limit,
const ZopfliNode* nodes,
int* dist_cache,
size_t* last_insert_len,
const BrotliEncoderParams* params,
Command* commands,
size_t* num_literals) {
size_t pos = 0;
uint32_t offset = nodes[0].u.next;
size_t i;
size_t gap = 0;
for (i = 0; offset != BROTLI_UINT32_MAX; i++) {
const ZopfliNode* next = &nodes[pos + offset];
size_t copy_length = ZopfliNodeCopyLength(next);
size_t insert_length = next->dcode_insert_length & 0x7FFFFFF;
pos += insert_length;
offset = next->u.next;
if (i == 0) {
insert_length += *last_insert_len;
*last_insert_len = 0;
}
{
size_t distance = ZopfliNodeCopyDistance(next);
size_t len_code = ZopfliNodeLengthCode(next);
size_t max_distance =
BROTLI_MIN(size_t, block_start + pos, max_backward_limit);
BROTLI_BOOL is_dictionary = TO_BROTLI_BOOL(distance > max_distance + gap);
size_t dist_code = ZopfliNodeDistanceCode(next);
InitCommand(&commands[i], &params->dist, insert_length,
copy_length, (int)len_code - (int)copy_length, dist_code);
if (!is_dictionary && dist_code > 0) {
dist_cache[3] = dist_cache[2];
dist_cache[2] = dist_cache[1];
dist_cache[1] = dist_cache[0];
dist_cache[0] = (int)distance;
}
}
*num_literals += insert_length;
pos += copy_length;
}
*last_insert_len += num_bytes - pos;
}
static size_t ZopfliIterate(size_t num_bytes,
size_t position,
const uint8_t* ringbuffer,
size_t ringbuffer_mask,
const BrotliEncoderParams* params,
const size_t max_backward_limit,
const size_t gap,
const int* dist_cache,
const ZopfliCostModel* model,
const uint32_t* num_matches,
const BackwardMatch* matches,
ZopfliNode* nodes) {
const size_t max_zopfli_len = MaxZopfliLen(params);
StartPosQueue queue;
size_t cur_match_pos = 0;
size_t i;
nodes[0].length = 0;
nodes[0].u.cost = 0;
InitStartPosQueue(&queue);
for (i = 0; i + 3 < num_bytes; i++) {
size_t skip = UpdateNodes(num_bytes, position, i, ringbuffer,
ringbuffer_mask, params, max_backward_limit, dist_cache,
num_matches[i], &matches[cur_match_pos], model, &queue, nodes);
if (skip < BROTLI_LONG_COPY_QUICK_STEP) skip = 0;
cur_match_pos += num_matches[i];
if (num_matches[i] == 1 &&
BackwardMatchLength(&matches[cur_match_pos - 1]) > max_zopfli_len) {
skip = BROTLI_MAX(size_t,
BackwardMatchLength(&matches[cur_match_pos - 1]), skip);
}
if (skip > 1) {
skip--;
while (skip) {
i++;
if (i + 3 >= num_bytes) break;
EvaluateNode(position, i, max_backward_limit, gap, dist_cache, model,
&queue, nodes);
cur_match_pos += num_matches[i];
skip--;
}
}
}
return ComputeShortestPathFromNodes(num_bytes, nodes);
}
/* REQUIRES: nodes != NULL and len(nodes) >= num_bytes + 1 */
size_t BrotliZopfliComputeShortestPath(MemoryManager* m,
size_t num_bytes, size_t position, const uint8_t* ringbuffer,
size_t ringbuffer_mask, const BrotliEncoderParams* params,
const size_t max_backward_limit, const int* dist_cache, HasherHandle hasher,
ZopfliNode* nodes) {
const size_t max_zopfli_len = MaxZopfliLen(params);
ZopfliCostModel model;
StartPosQueue queue;
BackwardMatch matches[2 * (MAX_NUM_MATCHES_H10 + 64)];
const size_t store_end = num_bytes >= StoreLookaheadH10() ?
position + num_bytes - StoreLookaheadH10() + 1 : position;
size_t i;
size_t gap = 0;
size_t lz_matches_offset = 0;
nodes[0].length = 0;
nodes[0].u.cost = 0;
InitZopfliCostModel(m, &model, &params->dist, num_bytes);
if (BROTLI_IS_OOM(m)) return 0;
ZopfliCostModelSetFromLiteralCosts(
&model, position, ringbuffer, ringbuffer_mask);
InitStartPosQueue(&queue);
for (i = 0; i + HashTypeLengthH10() - 1 < num_bytes; i++) {
const size_t pos = position + i;
const size_t max_distance = BROTLI_MIN(size_t, pos, max_backward_limit);
size_t skip;
size_t num_matches = FindAllMatchesH10(hasher, &params->dictionary,
ringbuffer, ringbuffer_mask, pos, num_bytes - i, max_distance, gap,
params, &matches[lz_matches_offset]);
if (num_matches > 0 &&
BackwardMatchLength(&matches[num_matches - 1]) > max_zopfli_len) {
matches[0] = matches[num_matches - 1];
num_matches = 1;
}
skip = UpdateNodes(num_bytes, position, i, ringbuffer, ringbuffer_mask,
params, max_backward_limit, dist_cache, num_matches, matches, &model,
&queue, nodes);
if (skip < BROTLI_LONG_COPY_QUICK_STEP) skip = 0;
if (num_matches == 1 && BackwardMatchLength(&matches[0]) > max_zopfli_len) {
skip = BROTLI_MAX(size_t, BackwardMatchLength(&matches[0]), skip);
}
if (skip > 1) {
/* Add the tail of the copy to the hasher. */
StoreRangeH10(hasher, ringbuffer, ringbuffer_mask, pos + 1, BROTLI_MIN(
size_t, pos + skip, store_end));
skip--;
while (skip) {
i++;
if (i + HashTypeLengthH10() - 1 >= num_bytes) break;
EvaluateNode(position, i, max_backward_limit, gap, dist_cache, &model,
&queue, nodes);
skip--;
}
}
}
CleanupZopfliCostModel(m, &model);
return ComputeShortestPathFromNodes(num_bytes, nodes);
}
void BrotliCreateZopfliBackwardReferences(MemoryManager* m,
size_t num_bytes, size_t position, const uint8_t* ringbuffer,
size_t ringbuffer_mask, const BrotliEncoderParams* params,
HasherHandle hasher, int* dist_cache, size_t* last_insert_len,
Command* commands, size_t* num_commands, size_t* num_literals) {
const size_t max_backward_limit = BROTLI_MAX_BACKWARD_LIMIT(params->lgwin);
ZopfliNode* nodes;
nodes = BROTLI_ALLOC(m, ZopfliNode, num_bytes + 1);
if (BROTLI_IS_OOM(m)) return;
BrotliInitZopfliNodes(nodes, num_bytes + 1);
*num_commands += BrotliZopfliComputeShortestPath(m,
num_bytes, position, ringbuffer, ringbuffer_mask,
params, max_backward_limit, dist_cache, hasher, nodes);
if (BROTLI_IS_OOM(m)) return;
BrotliZopfliCreateCommands(num_bytes, position, max_backward_limit, nodes,
dist_cache, last_insert_len, params, commands, num_literals);
BROTLI_FREE(m, nodes);
}
void BrotliCreateHqZopfliBackwardReferences(MemoryManager* m,
size_t num_bytes, size_t position, const uint8_t* ringbuffer,
size_t ringbuffer_mask, const BrotliEncoderParams* params,
HasherHandle hasher, int* dist_cache, size_t* last_insert_len,
Command* commands, size_t* num_commands, size_t* num_literals) {
const size_t max_backward_limit = BROTLI_MAX_BACKWARD_LIMIT(params->lgwin);
uint32_t* num_matches = BROTLI_ALLOC(m, uint32_t, num_bytes);
size_t matches_size = 4 * num_bytes;
const size_t store_end = num_bytes >= StoreLookaheadH10() ?
position + num_bytes - StoreLookaheadH10() + 1 : position;
size_t cur_match_pos = 0;
size_t i;
size_t orig_num_literals;
size_t orig_last_insert_len;
int orig_dist_cache[4];
size_t orig_num_commands;
ZopfliCostModel model;
ZopfliNode* nodes;
BackwardMatch* matches = BROTLI_ALLOC(m, BackwardMatch, matches_size);
size_t gap = 0;
size_t shadow_matches = 0;
if (BROTLI_IS_OOM(m)) return;
for (i = 0; i + HashTypeLengthH10() - 1 < num_bytes; ++i) {
const size_t pos = position + i;
size_t max_distance = BROTLI_MIN(size_t, pos, max_backward_limit);
size_t max_length = num_bytes - i;
size_t num_found_matches;
size_t cur_match_end;
size_t j;
/* Ensure that we have enough free slots. */
BROTLI_ENSURE_CAPACITY(m, BackwardMatch, matches, matches_size,
cur_match_pos + MAX_NUM_MATCHES_H10 + shadow_matches);
if (BROTLI_IS_OOM(m)) return;
num_found_matches = FindAllMatchesH10(hasher,
&params->dictionary, ringbuffer, ringbuffer_mask, pos, max_length,
max_distance, gap, params, &matches[cur_match_pos + shadow_matches]);
cur_match_end = cur_match_pos + num_found_matches;
for (j = cur_match_pos; j + 1 < cur_match_end; ++j) {
BROTLI_DCHECK(BackwardMatchLength(&matches[j]) <=
BackwardMatchLength(&matches[j + 1]));
}
num_matches[i] = (uint32_t)num_found_matches;
if (num_found_matches > 0) {
const size_t match_len = BackwardMatchLength(&matches[cur_match_end - 1]);
if (match_len > MAX_ZOPFLI_LEN_QUALITY_11) {
const size_t skip = match_len - 1;
matches[cur_match_pos++] = matches[cur_match_end - 1];
num_matches[i] = 1;
/* Add the tail of the copy to the hasher. */
StoreRangeH10(hasher, ringbuffer, ringbuffer_mask, pos + 1,
BROTLI_MIN(size_t, pos + match_len, store_end));
memset(&num_matches[i + 1], 0, skip * sizeof(num_matches[0]));
i += skip;
} else {
cur_match_pos = cur_match_end;
}
}
}
orig_num_literals = *num_literals;
orig_last_insert_len = *last_insert_len;
memcpy(orig_dist_cache, dist_cache, 4 * sizeof(dist_cache[0]));
orig_num_commands = *num_commands;
nodes = BROTLI_ALLOC(m, ZopfliNode, num_bytes + 1);
if (BROTLI_IS_OOM(m)) return;
InitZopfliCostModel(m, &model, &params->dist, num_bytes);
if (BROTLI_IS_OOM(m)) return;
for (i = 0; i < 2; i++) {
BrotliInitZopfliNodes(nodes, num_bytes + 1);
if (i == 0) {
ZopfliCostModelSetFromLiteralCosts(
&model, position, ringbuffer, ringbuffer_mask);
} else {
ZopfliCostModelSetFromCommands(&model, position, ringbuffer,
ringbuffer_mask, commands, *num_commands - orig_num_commands,
orig_last_insert_len);
}
*num_commands = orig_num_commands;
*num_literals = orig_num_literals;
*last_insert_len = orig_last_insert_len;
memcpy(dist_cache, orig_dist_cache, 4 * sizeof(dist_cache[0]));
*num_commands += ZopfliIterate(num_bytes, position, ringbuffer,
ringbuffer_mask, params, max_backward_limit, gap, dist_cache,
&model, num_matches, matches, nodes);
BrotliZopfliCreateCommands(num_bytes, position, max_backward_limit,
nodes, dist_cache, last_insert_len, params, commands, num_literals);
}
CleanupZopfliCostModel(m, &model);
BROTLI_FREE(m, nodes);
BROTLI_FREE(m, matches);
BROTLI_FREE(m, num_matches);
}
#if defined(__cplusplus) || defined(c_plusplus)
} /* extern "C" */
#endif

View File

@ -0,0 +1,93 @@
/* Copyright 2013 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* Function to find backward reference copies. */
#ifndef BROTLI_ENC_BACKWARD_REFERENCES_HQ_H_
#define BROTLI_ENC_BACKWARD_REFERENCES_HQ_H_
#include "../common/constants.h"
#include "../common/dictionary.h"
#include "../common/platform.h"
#include <brotli/types.h>
#include "./command.h"
#include "./hash.h"
#include "./memory.h"
#include "./quality.h"
#if defined(__cplusplus) || defined(c_plusplus)
extern "C" {
#endif
BROTLI_INTERNAL void BrotliCreateZopfliBackwardReferences(MemoryManager* m,
size_t num_bytes, size_t position, const uint8_t* ringbuffer,
size_t ringbuffer_mask, const BrotliEncoderParams* params,
HasherHandle hasher, int* dist_cache, size_t* last_insert_len,
Command* commands, size_t* num_commands, size_t* num_literals);
BROTLI_INTERNAL void BrotliCreateHqZopfliBackwardReferences(MemoryManager* m,
size_t num_bytes, size_t position, const uint8_t* ringbuffer,
size_t ringbuffer_mask, const BrotliEncoderParams* params,
HasherHandle hasher, int* dist_cache, size_t* last_insert_len,
Command* commands, size_t* num_commands, size_t* num_literals);
typedef struct ZopfliNode {
/* Best length to get up to this byte (not including this byte itself)
highest 7 bit is used to reconstruct the length code. */
uint32_t length;
/* Distance associated with the length. */
uint32_t distance;
/* Number of literal inserts before this copy; highest 5 bits contain
distance short code + 1 (or zero if no short code). */
uint32_t dcode_insert_length;
/* This union holds information used by dynamic-programming. During forward
pass |cost| it used to store the goal function. When node is processed its
|cost| is invalidated in favor of |shortcut|. On path back-tracing pass
|next| is assigned the offset to next node on the path. */
union {
/* Smallest cost to get to this byte from the beginning, as found so far. */
float cost;
/* Offset to the next node on the path. Equals to command_length() of the
next node on the path. For last node equals to BROTLI_UINT32_MAX */
uint32_t next;
/* Node position that provides next distance for distance cache. */
uint32_t shortcut;
} u;
} ZopfliNode;
BROTLI_INTERNAL void BrotliInitZopfliNodes(ZopfliNode* array, size_t length);
/* Computes the shortest path of commands from position to at most
position + num_bytes.
On return, path->size() is the number of commands found and path[i] is the
length of the i-th command (copy length plus insert length).
Note that the sum of the lengths of all commands can be less than num_bytes.
On return, the nodes[0..num_bytes] array will have the following
"ZopfliNode array invariant":
For each i in [1..num_bytes], if nodes[i].cost < kInfinity, then
(1) nodes[i].copy_length() >= 2
(2) nodes[i].command_length() <= i and
(3) nodes[i - nodes[i].command_length()].cost < kInfinity */
BROTLI_INTERNAL size_t BrotliZopfliComputeShortestPath(MemoryManager* m,
size_t num_bytes, size_t position, const uint8_t* ringbuffer,
size_t ringbuffer_mask, const BrotliEncoderParams* params,
const size_t max_backward_limit, const int* dist_cache, HasherHandle hasher,
ZopfliNode* nodes);
BROTLI_INTERNAL void BrotliZopfliCreateCommands(
const size_t num_bytes, const size_t block_start,
const size_t max_backward_limit, const ZopfliNode* nodes,
int* dist_cache, size_t* last_insert_len, const BrotliEncoderParams* params,
Command* commands, size_t* num_literals);
#if defined(__cplusplus) || defined(c_plusplus)
} /* extern "C" */
#endif
#endif /* BROTLI_ENC_BACKWARD_REFERENCES_HQ_H_ */

View File

@ -5,18 +5,16 @@
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* template parameters: FN */
/* template parameters: EXPORT_FN, FN */
#define Hasher HASHER()
static BROTLI_NOINLINE void FN(CreateBackwardReferences)(
MemoryManager* m, size_t num_bytes, size_t position, BROTLI_BOOL is_last,
static BROTLI_NOINLINE void EXPORT_FN(CreateBackwardReferences)(
size_t num_bytes, size_t position,
const uint8_t* ringbuffer, size_t ringbuffer_mask,
const BrotliEncoderParams* params, Hasher* hasher, int* dist_cache,
const BrotliEncoderParams* params, HasherHandle hasher, int* dist_cache,
size_t* last_insert_len, Command* commands, size_t* num_commands,
size_t* num_literals) {
/* Set maximum distance, see section 9.1. of the spec. */
const size_t max_backward_limit = MaxBackwardLimit(params->lgwin);
const size_t max_backward_limit = BROTLI_MAX_BACKWARD_LIMIT(params->lgwin);
const Command* const orig_commands = commands;
size_t insert_length = *last_insert_len;
@ -28,42 +26,42 @@ static BROTLI_NOINLINE void FN(CreateBackwardReferences)(
const size_t random_heuristics_window_size =
LiteralSpreeLengthForSparseSearch(params);
size_t apply_random_heuristics = position + random_heuristics_window_size;
const size_t gap = 0;
/* Minimum score to accept a backward reference. */
const score_t kMinScore = BROTLI_SCORE_BASE + 400;
const score_t kMinScore = BROTLI_SCORE_BASE + 100;
FN(Init)(m, hasher, ringbuffer, params, position, num_bytes, is_last);
if (BROTLI_IS_OOM(m)) return;
FN(StitchToPreviousBlock)(hasher, num_bytes, position,
ringbuffer, ringbuffer_mask);
FN(PrepareDistanceCache)(hasher, dist_cache);
while (position + FN(HashTypeLength)() < pos_end) {
size_t max_length = pos_end - position;
size_t max_distance = BROTLI_MIN(size_t, position, max_backward_limit);
HasherSearchResult sr;
sr.len = 0;
sr.len_x_code = 0;
sr.len_code_delta = 0;
sr.distance = 0;
sr.score = kMinScore;
if (FN(FindLongestMatch)(hasher, ringbuffer, ringbuffer_mask, dist_cache,
position, max_length, max_distance, &sr)) {
FN(FindLongestMatch)(hasher, &params->dictionary,
ringbuffer, ringbuffer_mask, dist_cache, position,
max_length, max_distance, gap,
params->dist.max_distance, &sr);
if (sr.score > kMinScore) {
/* Found a match. Let's look for something even better ahead. */
int delayed_backward_references_in_row = 0;
--max_length;
for (;; --max_length) {
const score_t cost_diff_lazy = 700;
BROTLI_BOOL is_match_found;
const score_t cost_diff_lazy = 175;
HasherSearchResult sr2;
sr2.len = params->quality < MIN_QUALITY_FOR_EXTENSIVE_REFERENCE_SEARCH ?
BROTLI_MIN(size_t, sr.len - 1, max_length) : 0;
sr2.len_x_code = 0;
sr2.len_code_delta = 0;
sr2.distance = 0;
sr2.score = kMinScore;
max_distance = BROTLI_MIN(size_t, position + 1, max_backward_limit);
is_match_found = FN(FindLongestMatch)(hasher, ringbuffer,
ringbuffer_mask, dist_cache, position + 1, max_length, max_distance,
&sr2);
if (is_match_found && sr2.score >= sr.score + cost_diff_lazy) {
FN(FindLongestMatch)(hasher, &params->dictionary,
ringbuffer, ringbuffer_mask, dist_cache, position + 1, max_length,
max_distance, gap, params->dist.max_distance, &sr2);
if (sr2.score >= sr.score + cost_diff_lazy) {
/* Ok, let's just write one byte for now and start a match from the
next byte. */
++position;
@ -80,26 +78,36 @@ static BROTLI_NOINLINE void FN(CreateBackwardReferences)(
position + 2 * sr.len + random_heuristics_window_size;
max_distance = BROTLI_MIN(size_t, position, max_backward_limit);
{
/* The first 16 codes are special shortcodes,
/* The first 16 codes are special short-codes,
and the minimum offset is 1. */
size_t distance_code =
ComputeDistanceCode(sr.distance, max_distance, dist_cache);
if (sr.distance <= max_distance && distance_code > 0) {
ComputeDistanceCode(sr.distance, max_distance + gap, dist_cache);
if ((sr.distance <= (max_distance + gap)) && distance_code > 0) {
dist_cache[3] = dist_cache[2];
dist_cache[2] = dist_cache[1];
dist_cache[1] = dist_cache[0];
dist_cache[0] = (int)sr.distance;
FN(PrepareDistanceCache)(hasher, dist_cache);
}
InitCommand(commands++, insert_length, sr.len, sr.len ^ sr.len_x_code,
distance_code);
InitCommand(commands++, &params->dist, insert_length,
sr.len, sr.len_code_delta, distance_code);
}
*num_literals += insert_length;
insert_length = 0;
/* Put the hash keys into the table, if there are enough bytes left.
Depending on the hasher implementation, it can push all positions
in the given range or only a subset of them. */
FN(StoreRange)(hasher, ringbuffer, ringbuffer_mask, position + 2,
BROTLI_MIN(size_t, position + sr.len, store_end));
in the given range or only a subset of them.
Avoid hash poisoning with RLE data. */
{
size_t range_start = position + 2;
size_t range_end = BROTLI_MIN(size_t, position + sr.len, store_end);
if (sr.distance < (sr.len >> 2)) {
range_start = BROTLI_MIN(size_t, range_end, BROTLI_MAX(size_t,
range_start, position + sr.len - (sr.distance << 2)));
}
FN(StoreRange)(hasher, ringbuffer, ringbuffer_mask, range_start,
range_end);
}
position += sr.len;
} else {
++insert_length;
@ -143,5 +151,3 @@ static BROTLI_NOINLINE void FN(CreateBackwardReferences)(
*last_insert_len = insert_length;
*num_commands += (size_t)(commands - orig_commands);
}
#undef Hasher

View File

@ -9,10 +9,10 @@
#include "./bit_cost.h"
#include "../common/constants.h"
#include "../common/types.h"
#include "../common/platform.h"
#include <brotli/types.h>
#include "./fast_log.h"
#include "./histogram.h"
#include "./port.h"
#if defined(__cplusplus) || defined(c_plusplus)
extern "C" {

View File

@ -9,20 +9,20 @@
#ifndef BROTLI_ENC_BIT_COST_H_
#define BROTLI_ENC_BIT_COST_H_
#include "../common/types.h"
#include "../common/platform.h"
#include <brotli/types.h>
#include "./fast_log.h"
#include "./histogram.h"
#include "./port.h"
#if defined(__cplusplus) || defined(c_plusplus)
extern "C" {
#endif
static BROTLI_INLINE double ShannonEntropy(const uint32_t *population,
size_t size, size_t *total) {
static BROTLI_INLINE double ShannonEntropy(
const uint32_t* population, size_t size, size_t* total) {
size_t sum = 0;
double retval = 0;
const uint32_t *population_end = population + size;
const uint32_t* population_end = population + size;
size_t p;
if (size & 1) {
goto odd_number_of_elements_left;
@ -42,7 +42,7 @@ static BROTLI_INLINE double ShannonEntropy(const uint32_t *population,
}
static BROTLI_INLINE double BitsEntropy(
const uint32_t *population, size_t size) {
const uint32_t* population, size_t size) {
size_t sum;
double retval = ShannonEntropy(population, size, &sum);
if (retval < sum) {

View File

@ -13,9 +13,9 @@
stream. */
static void FN(BuildAndStoreEntropyCodes)(MemoryManager* m, BlockEncoder* self,
const HistogramType* histograms, const size_t histograms_size,
HuffmanTree* tree, size_t* storage_ix, uint8_t* storage) {
const size_t alphabet_size = self->alphabet_size_;
const size_t table_size = histograms_size * alphabet_size;
const size_t alphabet_size, HuffmanTree* tree,
size_t* storage_ix, uint8_t* storage) {
const size_t table_size = histograms_size * self->histogram_length_;
self->depths_ = BROTLI_ALLOC(m, uint8_t, table_size);
self->bits_ = BROTLI_ALLOC(m, uint16_t, table_size);
if (BROTLI_IS_OOM(m)) return;
@ -23,9 +23,10 @@ static void FN(BuildAndStoreEntropyCodes)(MemoryManager* m, BlockEncoder* self,
{
size_t i;
for (i = 0; i < histograms_size; ++i) {
size_t ix = i * alphabet_size;
BuildAndStoreHuffmanTree(&histograms[i].data_[0], alphabet_size, tree,
&self->depths_[ix], &self->bits_[ix], storage_ix, storage);
size_t ix = i * self->histogram_length_;
BuildAndStoreHuffmanTree(&histograms[i].data_[0], self->histogram_length_,
alphabet_size, tree, &self->depths_[ix], &self->bits_[ix],
storage_ix, storage);
}
}
}

View File

@ -8,16 +8,15 @@
#include "./block_splitter.h"
#include <assert.h>
#include <string.h> /* memcpy, memset */
#include "../common/platform.h"
#include "./bit_cost.h"
#include "./cluster.h"
#include "./command.h"
#include "./fast_log.h"
#include "./histogram.h"
#include "./memory.h"
#include "./port.h"
#include "./quality.h"
#if defined(__cplusplus) || defined(c_plusplus)
@ -74,11 +73,9 @@ static void CopyLiteralsToByteArray(const Command* cmds,
}
}
static BROTLI_INLINE unsigned int MyRand(unsigned int* seed) {
static BROTLI_INLINE uint32_t MyRand(uint32_t* seed) {
/* Initial seed should be 7. In this case, loop length is (1 << 29). */
*seed *= 16807U;
if (*seed == 0) {
*seed = 1;
}
return *seed;
}
@ -177,7 +174,7 @@ void BrotliSplitBlock(MemoryManager* m,
for (i = 0; i < num_commands; ++i) {
const Command* cmd = &cmds[i];
if (CommandCopyLen(cmd) && cmd->cmd_prefix_ >= 128) {
distance_prefixes[j++] = cmd->dist_prefix_;
distance_prefixes[j++] = cmd->dist_prefix_ & 0x3FF;
}
}
/* Create the block split on the array of distance prefixes. */

View File

@ -9,10 +9,10 @@
#ifndef BROTLI_ENC_BLOCK_SPLITTER_H_
#define BROTLI_ENC_BLOCK_SPLITTER_H_
#include "../common/types.h"
#include "../common/platform.h"
#include <brotli/types.h>
#include "./command.h"
#include "./memory.h"
#include "./port.h"
#include "./quality.h"
#if defined(__cplusplus) || defined(c_plusplus)

View File

@ -13,7 +13,7 @@ static void FN(InitialEntropyCodes)(const DataType* data, size_t length,
size_t stride,
size_t num_histograms,
HistogramType* histograms) {
unsigned int seed = 7;
uint32_t seed = 7;
size_t block_length = length / num_histograms;
size_t i;
FN(ClearHistograms)(histograms, num_histograms);
@ -29,14 +29,13 @@ static void FN(InitialEntropyCodes)(const DataType* data, size_t length,
}
}
static void FN(RandomSample)(unsigned int* seed,
static void FN(RandomSample)(uint32_t* seed,
const DataType* data,
size_t length,
size_t stride,
HistogramType* sample) {
size_t pos = 0;
if (stride >= length) {
pos = 0;
stride = length;
} else {
pos = MyRand(seed) % (length - stride + 1);
@ -50,7 +49,7 @@ static void FN(RefineEntropyCodes)(const DataType* data, size_t length,
HistogramType* histograms) {
size_t iters =
kIterMulForRefining * length / stride + kMinItersForRefining;
unsigned int seed = 7;
uint32_t seed = 7;
size_t iter;
iters = ((iters + num_histograms - 1) / num_histograms) * num_histograms;
for (iter = 0; iter < iters; ++iter) {
@ -61,7 +60,7 @@ static void FN(RefineEntropyCodes)(const DataType* data, size_t length,
}
}
/* Assigns a block id from the range [0, vec.size()) to each data element
/* Assigns a block id from the range [0, num_histograms) to each data element
in data[0..length) and fills in block_id[0..length) with the assigned values.
Returns the number of blocks, i.e. one plus the number of block switches. */
static size_t FN(FindBlocks)(const DataType* data, const size_t length,
@ -71,13 +70,13 @@ static size_t FN(FindBlocks)(const DataType* data, const size_t length,
double* insert_cost,
double* cost,
uint8_t* switch_signal,
uint8_t *block_id) {
uint8_t* block_id) {
const size_t data_size = FN(HistogramDataSize)();
const size_t bitmaplen = (num_histograms + 7) >> 3;
size_t num_blocks = 1;
size_t i;
size_t j;
assert(num_histograms <= 256);
BROTLI_DCHECK(num_histograms <= 256);
if (num_histograms <= 1) {
for (i = 0; i < length; ++i) {
block_id[i] = 0;
@ -127,7 +126,7 @@ static size_t FN(FindBlocks)(const DataType* data, const size_t length,
if (cost[k] >= block_switch_cost) {
const uint8_t mask = (uint8_t)(1u << (k & 7));
cost[k] = block_switch_cost;
assert((k >> 3) < bitmaplen);
BROTLI_DCHECK((k >> 3) < bitmaplen);
switch_signal[ix + (k >> 3)] |= mask;
}
}
@ -138,7 +137,7 @@ static size_t FN(FindBlocks)(const DataType* data, const size_t length,
uint8_t cur_id = block_id[byte_ix];
while (byte_ix > 0) {
const uint8_t mask = (uint8_t)(1u << (cur_id & 7));
assert(((size_t)cur_id >> 3) < bitmaplen);
BROTLI_DCHECK(((size_t)cur_id >> 3) < bitmaplen);
--byte_ix;
ix -= bitmaplen;
if (switch_signal[ix + (cur_id >> 3)] & mask) {
@ -162,16 +161,16 @@ static size_t FN(RemapBlockIds)(uint8_t* block_ids, const size_t length,
new_id[i] = kInvalidId;
}
for (i = 0; i < length; ++i) {
assert(block_ids[i] < num_histograms);
BROTLI_DCHECK(block_ids[i] < num_histograms);
if (new_id[block_ids[i]] == kInvalidId) {
new_id[block_ids[i]] = next_id++;
}
}
for (i = 0; i < length; ++i) {
block_ids[i] = (uint8_t)new_id[block_ids[i]];
assert(block_ids[i] < num_histograms);
BROTLI_DCHECK(block_ids[i] < num_histograms);
}
assert(next_id <= num_histograms);
BROTLI_DCHECK(next_id <= num_histograms);
return next_id;
}
@ -214,7 +213,6 @@ static void FN(ClusterBlocks)(MemoryManager* m,
size_t num_final_clusters;
static const uint32_t kInvalidIndex = BROTLI_UINT32_MAX;
uint32_t* new_index;
uint8_t max_type = 0;
size_t i;
uint32_t sizes[HISTOGRAMS_PER_BATCH] = { 0 };
uint32_t new_clusters[HISTOGRAMS_PER_BATCH] = { 0 };
@ -228,13 +226,13 @@ static void FN(ClusterBlocks)(MemoryManager* m,
{
size_t block_idx = 0;
for (i = 0; i < length; ++i) {
assert(block_idx < num_blocks);
BROTLI_DCHECK(block_idx < num_blocks);
++block_lengths[block_idx];
if (i + 1 == length || block_ids[i] != block_ids[i + 1]) {
++block_idx;
}
}
assert(block_idx == num_blocks);
BROTLI_DCHECK(block_idx == num_blocks);
}
for (i = 0; i < num_blocks; i += HISTOGRAMS_PER_BATCH) {
@ -270,8 +268,8 @@ static void FN(ClusterBlocks)(MemoryManager* m,
histogram_symbols[i + j] = (uint32_t)num_clusters + remap[symbols[j]];
}
num_clusters += num_new_clusters;
assert(num_clusters == cluster_size_size);
assert(num_clusters == all_histograms_size);
BROTLI_DCHECK(num_clusters == cluster_size_size);
BROTLI_DCHECK(num_clusters == all_histograms_size);
}
BROTLI_FREE(m, histograms);
@ -337,6 +335,7 @@ static void FN(ClusterBlocks)(MemoryManager* m,
{
uint32_t cur_length = 0;
size_t block_idx = 0;
uint8_t max_type = 0;
for (i = 0; i < num_blocks; ++i) {
cur_length += block_lengths[i];
if (i + 1 == num_blocks ||
@ -398,7 +397,7 @@ static void FN(SplitByteVector)(MemoryManager* m,
{
/* Find a good path through literals with the good entropy codes. */
uint8_t* block_ids = BROTLI_ALLOC(m, uint8_t, length);
size_t num_blocks;
size_t num_blocks = 0;
const size_t bitmaplen = (num_histograms + 7) >> 3;
double* insert_cost = BROTLI_ALLOC(m, double, data_size * num_histograms);
double* cost = BROTLI_ALLOC(m, double, num_histograms);

View File

@ -13,13 +13,14 @@
#include <string.h> /* memcpy, memset */
#include "../common/constants.h"
#include "../common/types.h"
#include "./context.h"
#include "../common/context.h"
#include "../common/platform.h"
#include <brotli/types.h>
#include "./entropy_encode.h"
#include "./entropy_encode_static.h"
#include "./fast_log.h"
#include "./histogram.h"
#include "./memory.h"
#include "./port.h"
#include "./write_bits.h"
#if defined(__cplusplus) || defined(c_plusplus)
@ -27,6 +28,11 @@ extern "C" {
#endif
#define MAX_HUFFMAN_TREE_SIZE (2 * BROTLI_NUM_COMMAND_SYMBOLS + 1)
/* The maximum size of Huffman dictionary for distances assuming that
NPOSTFIX = 0 and NDIRECT = 0. */
#define MAX_SIMPLE_DISTANCE_ALPHABET_SIZE \
BROTLI_DISTANCE_ALPHABET_SIZE(0, 0, BROTLI_LARGE_MAX_DISTANCE_BITS)
/* MAX_SIMPLE_DISTANCE_ALPHABET_SIZE == 140 */
/* Represents the range of values belonging to a prefix code:
[offset, offset + 2^nbits) */
@ -76,16 +82,16 @@ static BROTLI_INLINE size_t NextBlockTypeCode(
return type_code;
}
/* nibblesbits represents the 2 bits to encode MNIBBLES (0-3)
/* |nibblesbits| represents the 2 bits to encode MNIBBLES (0-3)
REQUIRES: length > 0
REQUIRES: length <= (1 << 24) */
static void BrotliEncodeMlen(size_t length, uint64_t* bits,
size_t* numbits, uint64_t* nibblesbits) {
size_t lg = (length == 1) ? 1 : Log2FloorNonZero((uint32_t)(length - 1)) + 1;
size_t mnibbles = (lg < 16 ? 16 : (lg + 3)) / 4;
assert(length > 0);
assert(length <= (1 << 24));
assert(lg <= 24);
BROTLI_DCHECK(length > 0);
BROTLI_DCHECK(length <= (1 << 24));
BROTLI_DCHECK(lg <= 24);
*nibblesbits = mnibbles - 4;
*numbits = mnibbles * 4;
*bits = length - 1;
@ -252,7 +258,7 @@ static void StoreSimpleHuffmanTree(const uint8_t* depths,
size_t symbols[4],
size_t num_symbols,
size_t max_bits,
size_t *storage_ix, uint8_t *storage) {
size_t* storage_ix, uint8_t* storage) {
/* value of 1 indicates a simple Huffman code */
BrotliWriteBits(2, 1, storage_ix, storage);
BrotliWriteBits(2, num_symbols - 1, storage_ix, storage); /* NSYM - 1 */
@ -291,7 +297,7 @@ static void StoreSimpleHuffmanTree(const uint8_t* depths,
depths = symbol depths */
void BrotliStoreHuffmanTree(const uint8_t* depths, size_t num,
HuffmanTree* tree,
size_t *storage_ix, uint8_t *storage) {
size_t* storage_ix, uint8_t* storage) {
/* Write the Huffman tree into the brotli-representation.
The command alphabet is the largest, so this allocation will fit all
alphabets. */
@ -305,7 +311,7 @@ void BrotliStoreHuffmanTree(const uint8_t* depths, size_t num,
int num_codes = 0;
size_t code = 0;
assert(num <= BROTLI_NUM_COMMAND_SYMBOLS);
BROTLI_DCHECK(num <= BROTLI_NUM_COMMAND_SYMBOLS);
BrotliWriteHuffmanTree(depths, num, &huffman_tree_size, huffman_tree,
huffman_tree_extra_bits);
@ -343,7 +349,7 @@ void BrotliStoreHuffmanTree(const uint8_t* depths, size_t num,
code_length_bitdepth[code] = 0;
}
/* Store the real huffman tree now. */
/* Store the real Huffman tree now. */
BrotliStoreHuffmanTreeToBitMask(huffman_tree_size,
huffman_tree,
huffman_tree_extra_bits,
@ -354,8 +360,9 @@ void BrotliStoreHuffmanTree(const uint8_t* depths, size_t num,
/* Builds a Huffman tree from histogram[0:length] into depth[0:length] and
bits[0:length] and stores the encoded tree to the bit stream. */
static void BuildAndStoreHuffmanTree(const uint32_t *histogram,
const size_t length,
static void BuildAndStoreHuffmanTree(const uint32_t* histogram,
const size_t histogram_length,
const size_t alphabet_size,
HuffmanTree* tree,
uint8_t* depth,
uint16_t* bits,
@ -365,7 +372,7 @@ static void BuildAndStoreHuffmanTree(const uint32_t *histogram,
size_t s4[4] = { 0 };
size_t i;
size_t max_bits = 0;
for (i = 0; i < length; i++) {
for (i = 0; i < histogram_length; i++) {
if (histogram[i]) {
if (count < 4) {
s4[count] = i;
@ -377,7 +384,7 @@ static void BuildAndStoreHuffmanTree(const uint32_t *histogram,
}
{
size_t max_bits_counter = length - 1;
size_t max_bits_counter = alphabet_size - 1;
while (max_bits_counter) {
max_bits_counter >>= 1;
++max_bits;
@ -392,14 +399,14 @@ static void BuildAndStoreHuffmanTree(const uint32_t *histogram,
return;
}
memset(depth, 0, length * sizeof(depth[0]));
BrotliCreateHuffmanTree(histogram, length, 15, tree, depth);
BrotliConvertBitDepthsToSymbols(depth, length, bits);
memset(depth, 0, histogram_length * sizeof(depth[0]));
BrotliCreateHuffmanTree(histogram, histogram_length, 15, tree, depth);
BrotliConvertBitDepthsToSymbols(depth, histogram_length, bits);
if (count <= 4) {
StoreSimpleHuffmanTree(depth, s4, count, max_bits, storage_ix, storage);
} else {
BrotliStoreHuffmanTree(depth, length, tree, storage_ix, storage);
BrotliStoreHuffmanTree(depth, histogram_length, tree, storage_ix, storage);
}
}
@ -450,7 +457,7 @@ void BrotliBuildAndStoreHuffmanTreeFast(MemoryManager* m,
for (l = length; l != 0;) {
--l;
if (histogram[l]) {
if (PREDICT_TRUE(histogram[l] >= count_limit)) {
if (BROTLI_PREDICT_TRUE(histogram[l] >= count_limit)) {
InitHuffmanTree(node, histogram[l], -1, (int16_t)l);
} else {
InitHuffmanTree(node, count_limit, -1, (int16_t)l);
@ -548,7 +555,7 @@ void BrotliBuildAndStoreHuffmanTreeFast(MemoryManager* m,
/* Complex Huffman Tree */
StoreStaticCodeLengthCode(storage_ix, storage);
/* Actual rle coding. */
/* Actual RLE coding. */
for (i = 0; i < length;) {
const uint8_t value = depth[i];
size_t reps = 1;
@ -613,7 +620,7 @@ static void MoveToFrontTransform(const uint32_t* BROTLI_RESTRICT v_in,
for (i = 1; i < v_size; ++i) {
if (v_in[i] > max_value) max_value = v_in[i];
}
assert(max_value < 256u);
BROTLI_DCHECK(max_value < 256u);
for (i = 0; i <= max_value; ++i) {
mtf[i] = (uint8_t)i;
}
@ -621,7 +628,7 @@ static void MoveToFrontTransform(const uint32_t* BROTLI_RESTRICT v_in,
size_t mtf_size = max_value + 1;
for (i = 0; i < v_size; ++i) {
size_t index = IndexOf(mtf, mtf_size, (uint8_t)v_in[i]);
assert(index < mtf_size);
BROTLI_DCHECK(index < mtf_size);
v_out[i] = (uint32_t)index;
MoveToFront(mtf, index);
}
@ -653,7 +660,7 @@ static void RunLengthCodeZeros(const size_t in_size,
*max_run_length_prefix = max_prefix;
*out_size = 0;
for (i = 0; i < in_size;) {
assert(*out_size <= i);
BROTLI_DCHECK(*out_size <= i);
if (v[i] != 0) {
v[*out_size] = v[i] + *max_run_length_prefix;
++i;
@ -723,6 +730,7 @@ static void EncodeContextMap(MemoryManager* m,
}
}
BuildAndStoreHuffmanTree(histogram, num_clusters + max_run_length_prefix,
num_clusters + max_run_length_prefix,
tree, depths, bits, storage_ix, storage);
for (i = 0; i < num_rle_symbols; ++i) {
const uint32_t rle_symbol = rle_symbols[i] & kSymbolMask;
@ -782,10 +790,11 @@ static void BuildAndStoreBlockSplitCode(const uint8_t* types,
}
StoreVarLenUint8(num_types - 1, storage_ix, storage);
if (num_types > 1) { /* TODO: else? could StoreBlockSwitch occur? */
BuildAndStoreHuffmanTree(&type_histo[0], num_types + 2, tree,
BuildAndStoreHuffmanTree(&type_histo[0], num_types + 2, num_types + 2, tree,
&code->type_depths[0], &code->type_bits[0],
storage_ix, storage);
BuildAndStoreHuffmanTree(&length_histo[0], BROTLI_NUM_BLOCK_LEN_SYMBOLS,
BROTLI_NUM_BLOCK_LEN_SYMBOLS,
tree, &code->length_depths[0],
&code->length_bits[0], storage_ix, storage);
StoreBlockSwitch(code, lengths[0], types[0], 1, storage_ix, storage);
@ -816,8 +825,8 @@ static void StoreTrivialContextMap(size_t num_types,
for (i = context_bits; i < alphabet_size; ++i) {
histogram[i] = 1;
}
BuildAndStoreHuffmanTree(histogram, alphabet_size, tree,
depths, bits, storage_ix, storage);
BuildAndStoreHuffmanTree(histogram, alphabet_size, alphabet_size,
tree, depths, bits, storage_ix, storage);
for (i = 0; i < num_types; ++i) {
size_t code = (i == 0 ? 0 : i + context_bits - 1);
BrotliWriteBits(depths[code], bits[code], storage_ix, storage);
@ -832,7 +841,7 @@ static void StoreTrivialContextMap(size_t num_types,
/* Manages the encoding of one block category (literal, command or distance). */
typedef struct BlockEncoder {
size_t alphabet_size_;
size_t histogram_length_;
size_t num_block_types_;
const uint8_t* block_types_; /* Not owned. */
const uint32_t* block_lengths_; /* Not owned. */
@ -845,10 +854,10 @@ typedef struct BlockEncoder {
uint16_t* bits_;
} BlockEncoder;
static void InitBlockEncoder(BlockEncoder* self, size_t alphabet_size,
static void InitBlockEncoder(BlockEncoder* self, size_t histogram_length,
size_t num_block_types, const uint8_t* block_types,
const uint32_t* block_lengths, const size_t num_blocks) {
self->alphabet_size_ = alphabet_size;
self->histogram_length_ = histogram_length;
self->num_block_types_ = num_block_types;
self->block_types_ = block_types;
self->block_lengths_ = block_lengths;
@ -884,7 +893,7 @@ static void StoreSymbol(BlockEncoder* self, size_t symbol, size_t* storage_ix,
uint32_t block_len = self->block_lengths_[block_ix];
uint8_t block_type = self->block_types_[block_ix];
self->block_len_ = block_len;
self->entropy_ix_ = block_type * self->alphabet_size_;
self->entropy_ix_ = block_type * self->histogram_length_;
StoreBlockSwitch(&self->block_split_code_, block_len, block_type, 0,
storage_ix, storage);
}
@ -913,7 +922,7 @@ static void StoreSymbolWithContext(BlockEncoder* self, size_t symbol,
--self->block_len_;
{
size_t histo_ix = context_map[self->entropy_ix_ + context];
size_t ix = histo_ix * self->alphabet_size_ + symbol;
size_t ix = histo_ix * self->histogram_length_ + symbol;
BrotliWriteBits(self->depths_[ix], self->bits_[ix], storage_ix, storage);
}
}
@ -939,42 +948,38 @@ static void JumpToByteBoundary(size_t* storage_ix, uint8_t* storage) {
}
void BrotliStoreMetaBlock(MemoryManager* m,
const uint8_t* input,
size_t start_pos,
size_t length,
size_t mask,
uint8_t prev_byte,
uint8_t prev_byte2,
BROTLI_BOOL is_last,
uint32_t num_direct_distance_codes,
uint32_t distance_postfix_bits,
ContextType literal_context_mode,
const Command *commands,
size_t n_commands,
const MetaBlockSplit* mb,
size_t *storage_ix,
uint8_t *storage) {
const uint8_t* input, size_t start_pos, size_t length, size_t mask,
uint8_t prev_byte, uint8_t prev_byte2, BROTLI_BOOL is_last,
const BrotliEncoderParams* params, ContextType literal_context_mode,
const Command* commands, size_t n_commands, const MetaBlockSplit* mb,
size_t* storage_ix, uint8_t* storage) {
size_t pos = start_pos;
size_t i;
size_t num_distance_codes =
BROTLI_NUM_DISTANCE_SHORT_CODES + num_direct_distance_codes +
(48u << distance_postfix_bits);
uint32_t num_distance_symbols = params->dist.alphabet_size;
uint32_t num_effective_distance_symbols = num_distance_symbols;
HuffmanTree* tree;
ContextLut literal_context_lut = BROTLI_CONTEXT_LUT(literal_context_mode);
BlockEncoder literal_enc;
BlockEncoder command_enc;
BlockEncoder distance_enc;
const BrotliDistanceParams* dist = &params->dist;
if (params->large_window &&
num_effective_distance_symbols > BROTLI_NUM_HISTOGRAM_DISTANCE_SYMBOLS) {
num_effective_distance_symbols = BROTLI_NUM_HISTOGRAM_DISTANCE_SYMBOLS;
}
StoreCompressedMetaBlockHeader(is_last, length, storage_ix, storage);
tree = BROTLI_ALLOC(m, HuffmanTree, MAX_HUFFMAN_TREE_SIZE);
if (BROTLI_IS_OOM(m)) return;
InitBlockEncoder(&literal_enc, 256, mb->literal_split.num_types,
mb->literal_split.types, mb->literal_split.lengths,
mb->literal_split.num_blocks);
InitBlockEncoder(&literal_enc, BROTLI_NUM_LITERAL_SYMBOLS,
mb->literal_split.num_types, mb->literal_split.types,
mb->literal_split.lengths, mb->literal_split.num_blocks);
InitBlockEncoder(&command_enc, BROTLI_NUM_COMMAND_SYMBOLS,
mb->command_split.num_types, mb->command_split.types,
mb->command_split.lengths, mb->command_split.num_blocks);
InitBlockEncoder(&distance_enc, num_distance_codes,
InitBlockEncoder(&distance_enc, num_effective_distance_symbols,
mb->distance_split.num_types, mb->distance_split.types,
mb->distance_split.lengths, mb->distance_split.num_blocks);
@ -983,9 +988,10 @@ void BrotliStoreMetaBlock(MemoryManager* m,
BuildAndStoreBlockSwitchEntropyCodes(
&distance_enc, tree, storage_ix, storage);
BrotliWriteBits(2, distance_postfix_bits, storage_ix, storage);
BrotliWriteBits(4, num_direct_distance_codes >> distance_postfix_bits,
storage_ix, storage);
BrotliWriteBits(2, dist->distance_postfix_bits, storage_ix, storage);
BrotliWriteBits(
4, dist->num_direct_distance_codes >> dist->distance_postfix_bits,
storage_ix, storage);
for (i = 0; i < mb->literal_split.num_types; ++i) {
BrotliWriteBits(2, literal_context_mode, storage_ix, storage);
}
@ -1011,13 +1017,16 @@ void BrotliStoreMetaBlock(MemoryManager* m,
}
BuildAndStoreEntropyCodesLiteral(m, &literal_enc, mb->literal_histograms,
mb->literal_histograms_size, tree, storage_ix, storage);
mb->literal_histograms_size, BROTLI_NUM_LITERAL_SYMBOLS, tree,
storage_ix, storage);
if (BROTLI_IS_OOM(m)) return;
BuildAndStoreEntropyCodesCommand(m, &command_enc, mb->command_histograms,
mb->command_histograms_size, tree, storage_ix, storage);
mb->command_histograms_size, BROTLI_NUM_COMMAND_SYMBOLS, tree,
storage_ix, storage);
if (BROTLI_IS_OOM(m)) return;
BuildAndStoreEntropyCodesDistance(m, &distance_enc, mb->distance_histograms,
mb->distance_histograms_size, tree, storage_ix, storage);
mb->distance_histograms_size, num_distance_symbols, tree,
storage_ix, storage);
if (BROTLI_IS_OOM(m)) return;
BROTLI_FREE(m, tree);
@ -1035,7 +1044,8 @@ void BrotliStoreMetaBlock(MemoryManager* m,
} else {
size_t j;
for (j = cmd.insert_len_; j != 0; --j) {
size_t context = Context(prev_byte, prev_byte2, literal_context_mode);
size_t context =
BROTLI_CONTEXT(prev_byte, prev_byte2, literal_context_lut);
uint8_t literal = input[pos & mask];
StoreSymbolWithContext(&literal_enc, literal, context,
mb->literal_context_map, storage_ix, storage,
@ -1050,9 +1060,9 @@ void BrotliStoreMetaBlock(MemoryManager* m,
prev_byte2 = input[(pos - 2) & mask];
prev_byte = input[(pos - 1) & mask];
if (cmd.cmd_prefix_ >= 128) {
size_t dist_code = cmd.dist_prefix_;
uint32_t distnumextra = cmd.dist_extra_ >> 24;
uint64_t distextra = cmd.dist_extra_ & 0xffffff;
size_t dist_code = cmd.dist_prefix_ & 0x3FF;
uint32_t distnumextra = cmd.dist_prefix_ >> 10;
uint64_t distextra = cmd.dist_extra_;
if (mb->distance_context_map_size == 0) {
StoreSymbol(&distance_enc, dist_code, storage_ix, storage);
} else {
@ -1076,7 +1086,7 @@ void BrotliStoreMetaBlock(MemoryManager* m,
static void BuildHistograms(const uint8_t* input,
size_t start_pos,
size_t mask,
const Command *commands,
const Command* commands,
size_t n_commands,
HistogramLiteral* lit_histo,
HistogramCommand* cmd_histo,
@ -1093,7 +1103,7 @@ static void BuildHistograms(const uint8_t* input,
}
pos += CommandCopyLen(&cmd);
if (CommandCopyLen(&cmd) && cmd.cmd_prefix_ >= 128) {
HistogramAddDistance(dist_histo, cmd.dist_prefix_);
HistogramAddDistance(dist_histo, cmd.dist_prefix_ & 0x3FF);
}
}
}
@ -1101,7 +1111,7 @@ static void BuildHistograms(const uint8_t* input,
static void StoreDataWithHuffmanCodes(const uint8_t* input,
size_t start_pos,
size_t mask,
const Command *commands,
const Command* commands,
size_t n_commands,
const uint8_t* lit_depth,
const uint16_t* lit_bits,
@ -1128,9 +1138,9 @@ static void StoreDataWithHuffmanCodes(const uint8_t* input,
}
pos += CommandCopyLen(&cmd);
if (CommandCopyLen(&cmd) && cmd.cmd_prefix_ >= 128) {
const size_t dist_code = cmd.dist_prefix_;
const uint32_t distnumextra = cmd.dist_extra_ >> 24;
const uint32_t distextra = cmd.dist_extra_ & 0xffffff;
const size_t dist_code = cmd.dist_prefix_ & 0x3FF;
const uint32_t distnumextra = cmd.dist_prefix_ >> 10;
const uint32_t distextra = cmd.dist_extra_;
BrotliWriteBits(dist_depth[dist_code], dist_bits[dist_code],
storage_ix, storage);
BrotliWriteBits(distnumextra, distextra, storage_ix, storage);
@ -1139,25 +1149,21 @@ static void StoreDataWithHuffmanCodes(const uint8_t* input,
}
void BrotliStoreMetaBlockTrivial(MemoryManager* m,
const uint8_t* input,
size_t start_pos,
size_t length,
size_t mask,
BROTLI_BOOL is_last,
const Command *commands,
size_t n_commands,
size_t *storage_ix,
uint8_t *storage) {
const uint8_t* input, size_t start_pos, size_t length, size_t mask,
BROTLI_BOOL is_last, const BrotliEncoderParams* params,
const Command* commands, size_t n_commands,
size_t* storage_ix, uint8_t* storage) {
HistogramLiteral lit_histo;
HistogramCommand cmd_histo;
HistogramDistance dist_histo;
uint8_t lit_depth[256];
uint16_t lit_bits[256];
uint8_t lit_depth[BROTLI_NUM_LITERAL_SYMBOLS];
uint16_t lit_bits[BROTLI_NUM_LITERAL_SYMBOLS];
uint8_t cmd_depth[BROTLI_NUM_COMMAND_SYMBOLS];
uint16_t cmd_bits[BROTLI_NUM_COMMAND_SYMBOLS];
uint8_t dist_depth[64];
uint16_t dist_bits[64];
uint8_t dist_depth[MAX_SIMPLE_DISTANCE_ALPHABET_SIZE];
uint16_t dist_bits[MAX_SIMPLE_DISTANCE_ALPHABET_SIZE];
HuffmanTree* tree;
uint32_t num_distance_symbols = params->dist.alphabet_size;
StoreCompressedMetaBlockHeader(is_last, length, storage_ix, storage);
@ -1172,13 +1178,16 @@ void BrotliStoreMetaBlockTrivial(MemoryManager* m,
tree = BROTLI_ALLOC(m, HuffmanTree, MAX_HUFFMAN_TREE_SIZE);
if (BROTLI_IS_OOM(m)) return;
BuildAndStoreHuffmanTree(lit_histo.data_, 256, tree,
BuildAndStoreHuffmanTree(lit_histo.data_, BROTLI_NUM_LITERAL_SYMBOLS,
BROTLI_NUM_LITERAL_SYMBOLS, tree,
lit_depth, lit_bits,
storage_ix, storage);
BuildAndStoreHuffmanTree(cmd_histo.data_, BROTLI_NUM_COMMAND_SYMBOLS, tree,
BuildAndStoreHuffmanTree(cmd_histo.data_, BROTLI_NUM_COMMAND_SYMBOLS,
BROTLI_NUM_COMMAND_SYMBOLS, tree,
cmd_depth, cmd_bits,
storage_ix, storage);
BuildAndStoreHuffmanTree(dist_histo.data_, 64, tree,
BuildAndStoreHuffmanTree(dist_histo.data_, MAX_SIMPLE_DISTANCE_ALPHABET_SIZE,
num_distance_symbols, tree,
dist_depth, dist_bits,
storage_ix, storage);
BROTLI_FREE(m, tree);
@ -1193,15 +1202,14 @@ void BrotliStoreMetaBlockTrivial(MemoryManager* m,
}
void BrotliStoreMetaBlockFast(MemoryManager* m,
const uint8_t* input,
size_t start_pos,
size_t length,
size_t mask,
BROTLI_BOOL is_last,
const Command *commands,
size_t n_commands,
size_t *storage_ix,
uint8_t *storage) {
const uint8_t* input, size_t start_pos, size_t length, size_t mask,
BROTLI_BOOL is_last, const BrotliEncoderParams* params,
const Command* commands, size_t n_commands,
size_t* storage_ix, uint8_t* storage) {
uint32_t num_distance_symbols = params->dist.alphabet_size;
uint32_t distance_alphabet_bits =
Log2FloorNonZero(num_distance_symbols - 1) + 1;
StoreCompressedMetaBlockHeader(is_last, length, storage_ix, storage);
BrotliWriteBits(13, 0, storage_ix, storage);
@ -1245,8 +1253,8 @@ void BrotliStoreMetaBlockFast(MemoryManager* m,
uint16_t lit_bits[BROTLI_NUM_LITERAL_SYMBOLS];
uint8_t cmd_depth[BROTLI_NUM_COMMAND_SYMBOLS];
uint16_t cmd_bits[BROTLI_NUM_COMMAND_SYMBOLS];
uint8_t dist_depth[64];
uint16_t dist_bits[64];
uint8_t dist_depth[MAX_SIMPLE_DISTANCE_ALPHABET_SIZE];
uint16_t dist_bits[MAX_SIMPLE_DISTANCE_ALPHABET_SIZE];
HistogramClearLiteral(&lit_histo);
HistogramClearCommand(&cmd_histo);
HistogramClearDistance(&dist_histo);
@ -1266,7 +1274,8 @@ void BrotliStoreMetaBlockFast(MemoryManager* m,
if (BROTLI_IS_OOM(m)) return;
BrotliBuildAndStoreHuffmanTreeFast(m, dist_histo.data_,
dist_histo.total_count_,
/* max_bits = */ 6,
/* max_bits = */
distance_alphabet_bits,
dist_depth, dist_bits,
storage_ix, storage);
if (BROTLI_IS_OOM(m)) return;
@ -1285,11 +1294,11 @@ void BrotliStoreMetaBlockFast(MemoryManager* m,
/* This is for storing uncompressed blocks (simple raw storage of
bytes-as-bytes). */
void BrotliStoreUncompressedMetaBlock(BROTLI_BOOL is_final_block,
const uint8_t * BROTLI_RESTRICT input,
const uint8_t* BROTLI_RESTRICT input,
size_t position, size_t mask,
size_t len,
size_t * BROTLI_RESTRICT storage_ix,
uint8_t * BROTLI_RESTRICT storage) {
size_t* BROTLI_RESTRICT storage_ix,
uint8_t* BROTLI_RESTRICT storage) {
size_t masked_pos = position & mask;
BrotliStoreUncompressedMetaBlockHeader(len, storage_ix, storage);
JumpToByteBoundary(storage_ix, storage);
@ -1317,18 +1326,6 @@ void BrotliStoreUncompressedMetaBlock(BROTLI_BOOL is_final_block,
}
}
void BrotliStoreSyncMetaBlock(size_t* BROTLI_RESTRICT storage_ix,
uint8_t* BROTLI_RESTRICT storage) {
/* Empty metadata meta-block bit pattern:
1 bit: is_last (0)
2 bits: num nibbles (3)
1 bit: reserved (0)
2 bits: metadata length bytes (0) */
BrotliWriteBits(6, 6, storage_ix, storage);
JumpToByteBoundary(storage_ix, storage);
}
#if defined(__cplusplus) || defined(c_plusplus)
} /* extern "C" */
#endif

View File

@ -16,13 +16,13 @@
#ifndef BROTLI_ENC_BROTLI_BIT_STREAM_H_
#define BROTLI_ENC_BROTLI_BIT_STREAM_H_
#include "../common/types.h"
#include "../common/context.h"
#include "../common/platform.h"
#include <brotli/types.h>
#include "./command.h"
#include "./context.h"
#include "./entropy_encode.h"
#include "./memory.h"
#include "./metablock.h"
#include "./port.h"
#if defined(__cplusplus) || defined(c_plusplus)
extern "C" {
@ -32,7 +32,7 @@ extern "C" {
position for the current storage. */
BROTLI_INTERNAL void BrotliStoreHuffmanTree(const uint8_t* depths, size_t num,
HuffmanTree* tree, size_t *storage_ix, uint8_t *storage);
HuffmanTree* tree, size_t* storage_ix, uint8_t* storage);
BROTLI_INTERNAL void BrotliBuildAndStoreHuffmanTreeFast(
MemoryManager* m, const uint32_t* histogram, const size_t histogram_total,
@ -42,63 +42,40 @@ BROTLI_INTERNAL void BrotliBuildAndStoreHuffmanTreeFast(
/* REQUIRES: length > 0 */
/* REQUIRES: length <= (1 << 24) */
BROTLI_INTERNAL void BrotliStoreMetaBlock(MemoryManager* m,
const uint8_t* input,
size_t start_pos,
size_t length,
size_t mask,
uint8_t prev_byte,
uint8_t prev_byte2,
BROTLI_BOOL is_final_block,
uint32_t num_direct_distance_codes,
uint32_t distance_postfix_bits,
ContextType literal_context_mode,
const Command* commands,
size_t n_commands,
const MetaBlockSplit* mb,
size_t* storage_ix,
uint8_t* storage);
const uint8_t* input, size_t start_pos, size_t length, size_t mask,
uint8_t prev_byte, uint8_t prev_byte2, BROTLI_BOOL is_last,
const BrotliEncoderParams* params, ContextType literal_context_mode,
const Command* commands, size_t n_commands, const MetaBlockSplit* mb,
size_t* storage_ix, uint8_t* storage);
/* Stores the meta-block without doing any block splitting, just collects
one histogram per block category and uses that for entropy coding.
REQUIRES: length > 0
REQUIRES: length <= (1 << 24) */
BROTLI_INTERNAL void BrotliStoreMetaBlockTrivial(MemoryManager* m,
const uint8_t* input,
size_t start_pos,
size_t length,
size_t mask,
BROTLI_BOOL is_last,
const Command *commands,
size_t n_commands,
size_t* storage_ix,
uint8_t* storage);
const uint8_t* input, size_t start_pos, size_t length, size_t mask,
BROTLI_BOOL is_last, const BrotliEncoderParams* params,
const Command* commands, size_t n_commands,
size_t* storage_ix, uint8_t* storage);
/* Same as above, but uses static prefix codes for histograms with a only a few
symbols, and uses static code length prefix codes for all other histograms.
REQUIRES: length > 0
REQUIRES: length <= (1 << 24) */
BROTLI_INTERNAL void BrotliStoreMetaBlockFast(MemoryManager* m,
const uint8_t* input,
size_t start_pos,
size_t length,
size_t mask,
BROTLI_BOOL is_last,
const Command *commands,
size_t n_commands,
size_t* storage_ix,
uint8_t* storage);
const uint8_t* input, size_t start_pos, size_t length, size_t mask,
BROTLI_BOOL is_last, const BrotliEncoderParams* params,
const Command* commands, size_t n_commands,
size_t* storage_ix, uint8_t* storage);
/* This is for storing uncompressed blocks (simple raw storage of
bytes-as-bytes).
REQUIRES: length > 0
REQUIRES: length <= (1 << 24) */
BROTLI_INTERNAL void BrotliStoreUncompressedMetaBlock(
BROTLI_BOOL is_final_block, const uint8_t* input, size_t position,
size_t mask, size_t len, size_t* storage_ix, uint8_t* storage);
/* Stores an empty metadata meta-block and syncs to a byte boundary. */
BROTLI_INTERNAL void BrotliStoreSyncMetaBlock(size_t* storage_ix,
uint8_t* storage);
BROTLI_BOOL is_final_block, const uint8_t* BROTLI_RESTRICT input,
size_t position, size_t mask, size_t len,
size_t* BROTLI_RESTRICT storage_ix, uint8_t* BROTLI_RESTRICT storage);
#if defined(__cplusplus) || defined(c_plusplus)
} /* extern "C" */

View File

@ -8,12 +8,12 @@
#include "./cluster.h"
#include "../common/types.h"
#include "../common/platform.h"
#include <brotli/types.h>
#include "./bit_cost.h" /* BrotliPopulationCost */
#include "./fast_log.h"
#include "./histogram.h"
#include "./memory.h"
#include "./port.h"
#if defined(__cplusplus) || defined(c_plusplus)
extern "C" {

View File

@ -9,10 +9,10 @@
#ifndef BROTLI_ENC_CLUSTER_H_
#define BROTLI_ENC_CLUSTER_H_
#include "../common/types.h"
#include "../common/platform.h"
#include <brotli/types.h>
#include "./histogram.h"
#include "./memory.h"
#include "./port.h"
#if defined(__cplusplus) || defined(c_plusplus)
extern "C" {

View File

@ -17,6 +17,8 @@ BROTLI_INTERNAL void FN(BrotliCompareAndPushToQueue)(
size_t* num_pairs) CODE({
BROTLI_BOOL is_good_pair = BROTLI_FALSE;
HistogramPair p;
p.idx1 = p.idx2 = 0;
p.cost_diff = p.cost_combo = 0;
if (idx1 == idx2) {
return;
}

View File

@ -9,9 +9,11 @@
#ifndef BROTLI_ENC_COMMAND_H_
#define BROTLI_ENC_COMMAND_H_
#include "../common/types.h"
#include "../common/port.h"
#include "../common/constants.h"
#include "../common/platform.h"
#include <brotli/types.h>
#include "./fast_log.h"
#include "./params.h"
#include "./prefix.h"
#if defined(__cplusplus) || defined(c_plusplus)
@ -60,15 +62,21 @@ static BROTLI_INLINE uint16_t GetCopyLengthCode(size_t copylen) {
static BROTLI_INLINE uint16_t CombineLengthCodes(
uint16_t inscode, uint16_t copycode, BROTLI_BOOL use_last_distance) {
uint16_t bits64 =
(uint16_t)((copycode & 0x7u) | ((inscode & 0x7u) << 3));
if (use_last_distance && inscode < 8 && copycode < 16) {
return (copycode < 8) ? bits64 : (bits64 | 64);
(uint16_t)((copycode & 0x7u) | ((inscode & 0x7u) << 3u));
if (use_last_distance && inscode < 8u && copycode < 16u) {
return (copycode < 8u) ? bits64 : (bits64 | 64u);
} else {
/* "To convert an insert-and-copy length code to an insert length code and
a copy length code, the following table can be used" */
static const uint16_t cells[9] = { 128u, 192u, 384u, 256u, 320u, 512u,
448u, 576u, 640u };
return cells[(copycode >> 3) + 3 * (inscode >> 3)] | bits64;
/* Specification: 5 Encoding of ... (last table) */
/* offset = 2 * index, where index is in range [0..8] */
uint32_t offset = 2u * ((copycode >> 3u) + 3u * (inscode >> 3u));
/* All values in specification are K * 64,
where K = [2, 3, 6, 4, 5, 8, 7, 9, 10],
i + 1 = [1, 2, 3, 4, 5, 6, 7, 8, 9],
K - i - 1 = [1, 1, 3, 0, 0, 2, 0, 1, 2] = D.
All values in D require only 2 bits to encode.
Magic constant is shifted 6 bits left, to avoid final multiplication. */
offset = (offset << 5u) + 0x40u + ((0x520D40u >> offset) & 0xC0u);
return (uint16_t)(offset | bits64);
}
}
@ -98,44 +106,61 @@ static BROTLI_INLINE uint32_t GetCopyExtra(uint16_t copycode) {
typedef struct Command {
uint32_t insert_len_;
/* Stores copy_len in low 24 bits and copy_len XOR copy_code in high 8 bit. */
/* Stores copy_len in low 25 bits and copy_code - copy_len in high 7 bit. */
uint32_t copy_len_;
/* Stores distance extra bits. */
uint32_t dist_extra_;
uint16_t cmd_prefix_;
/* Stores distance code in low 10 bits
and number of extra bits in high 6 bits. */
uint16_t dist_prefix_;
} Command;
/* distance_code is e.g. 0 for same-as-last short code, or 16 for offset 1. */
static BROTLI_INLINE void InitCommand(Command* self, size_t insertlen,
size_t copylen, size_t copylen_code, size_t distance_code) {
static BROTLI_INLINE void InitCommand(Command* self,
const BrotliDistanceParams* dist, size_t insertlen,
size_t copylen, int copylen_code_delta, size_t distance_code) {
/* Don't rely on signed int representation, use honest casts. */
uint32_t delta = (uint8_t)((int8_t)copylen_code_delta);
self->insert_len_ = (uint32_t)insertlen;
self->copy_len_ = (uint32_t)(copylen | ((copylen_code ^ copylen) << 24));
self->copy_len_ = (uint32_t)(copylen | (delta << 25));
/* The distance prefix and extra bits are stored in this Command as if
npostfix and ndirect were 0, they are only recomputed later after the
clustering if needed. */
PrefixEncodeCopyDistance(
distance_code, 0, 0, &self->dist_prefix_, &self->dist_extra_);
distance_code, dist->num_direct_distance_codes,
dist->distance_postfix_bits, &self->dist_prefix_, &self->dist_extra_);
GetLengthCode(
insertlen, copylen_code, TO_BROTLI_BOOL(self->dist_prefix_ == 0),
&self->cmd_prefix_);
insertlen, (size_t)((int)copylen + copylen_code_delta),
TO_BROTLI_BOOL((self->dist_prefix_ & 0x3FF) == 0), &self->cmd_prefix_);
}
static BROTLI_INLINE void InitInsertCommand(Command* self, size_t insertlen) {
self->insert_len_ = (uint32_t)insertlen;
self->copy_len_ = 4 << 24;
self->copy_len_ = 4 << 25;
self->dist_extra_ = 0;
self->dist_prefix_ = 16;
self->dist_prefix_ = BROTLI_NUM_DISTANCE_SHORT_CODES;
GetLengthCode(insertlen, 4, BROTLI_FALSE, &self->cmd_prefix_);
}
static BROTLI_INLINE uint32_t CommandDistanceCode(const Command* self) {
if (self->dist_prefix_ < 16) {
return self->dist_prefix_;
static BROTLI_INLINE uint32_t CommandRestoreDistanceCode(
const Command* self, const BrotliDistanceParams* dist) {
if ((self->dist_prefix_ & 0x3FFu) <
BROTLI_NUM_DISTANCE_SHORT_CODES + dist->num_direct_distance_codes) {
return self->dist_prefix_ & 0x3FFu;
} else {
uint32_t nbits = self->dist_extra_ >> 24;
uint32_t extra = self->dist_extra_ & 0xffffff;
uint32_t prefix = self->dist_prefix_ - 12u - 2u * nbits;
return (prefix << nbits) + extra + 12;
uint32_t dcode = self->dist_prefix_ & 0x3FFu;
uint32_t nbits = self->dist_prefix_ >> 10;
uint32_t extra = self->dist_extra_;
uint32_t postfix_mask = (1U << dist->distance_postfix_bits) - 1U;
uint32_t hcode = (dcode - dist->num_direct_distance_codes -
BROTLI_NUM_DISTANCE_SHORT_CODES) >>
dist->distance_postfix_bits;
uint32_t lcode = (dcode - dist->num_direct_distance_codes -
BROTLI_NUM_DISTANCE_SHORT_CODES) & postfix_mask;
uint32_t offset = ((2U + (hcode & 1U)) << nbits) - 4U;
return ((offset + extra) << dist->distance_postfix_bits) + lcode +
dist->num_direct_distance_codes + BROTLI_NUM_DISTANCE_SHORT_CODES;
}
}
@ -149,11 +174,13 @@ static BROTLI_INLINE uint32_t CommandDistanceContext(const Command* self) {
}
static BROTLI_INLINE uint32_t CommandCopyLen(const Command* self) {
return self->copy_len_ & 0xFFFFFF;
return self->copy_len_ & 0x1FFFFFF;
}
static BROTLI_INLINE uint32_t CommandCopyLenCode(const Command* self) {
return (self->copy_len_ & 0xFFFFFF) ^ (self->copy_len_ >> 24);
uint32_t modifier = self->copy_len_ >> 25;
int32_t delta = (int8_t)((uint8_t)(modifier | ((modifier & 0x40) << 1)));
return (uint32_t)((int32_t)(self->copy_len_ & 0x1FFFFFF) + delta);
}
#if defined(__cplusplus) || defined(c_plusplus)

View File

@ -16,37 +16,39 @@
#include <string.h> /* memcmp, memcpy, memset */
#include "../common/types.h"
#include "../common/constants.h"
#include "../common/platform.h"
#include <brotli/types.h>
#include "./brotli_bit_stream.h"
#include "./entropy_encode.h"
#include "./fast_log.h"
#include "./find_match_length.h"
#include "./memory.h"
#include "./port.h"
#include "./write_bits.h"
#if defined(__cplusplus) || defined(c_plusplus)
extern "C" {
#endif
#define MAX_DISTANCE (long)BROTLI_MAX_BACKWARD_LIMIT(18)
/* kHashMul32 multiplier has these properties:
* The multiplier must be odd. Otherwise we may lose the highest bit.
* No long streaks of 1s or 0s.
* No long streaks of ones or zeros.
* There is no effort to ensure that it is a prime, the oddity is enough
for this use.
* The number has been tuned heuristically against compression benchmarks. */
static const uint32_t kHashMul32 = 0x1e35a7bd;
static const uint32_t kHashMul32 = 0x1E35A7BD;
static BROTLI_INLINE uint32_t Hash(const uint8_t* p, size_t shift) {
const uint64_t h = (BROTLI_UNALIGNED_LOAD64(p) << 24) * kHashMul32;
const uint64_t h = (BROTLI_UNALIGNED_LOAD64LE(p) << 24) * kHashMul32;
return (uint32_t)(h >> shift);
}
static BROTLI_INLINE uint32_t HashBytesAtOffset(
uint64_t v, int offset, size_t shift) {
assert(offset >= 0);
assert(offset <= 3);
BROTLI_DCHECK(offset >= 0);
BROTLI_DCHECK(offset <= 3);
{
const uint64_t h = ((v >> (8 * offset)) << 24) * kHashMul32;
return (uint32_t)(h >> shift);
@ -55,7 +57,7 @@ static BROTLI_INLINE uint32_t HashBytesAtOffset(
static BROTLI_INLINE BROTLI_BOOL IsMatch(const uint8_t* p1, const uint8_t* p2) {
return TO_BROTLI_BOOL(
BROTLI_UNALIGNED_LOAD32(p1) == BROTLI_UNALIGNED_LOAD32(p2) &&
BrotliUnalignedRead32(p1) == BrotliUnalignedRead32(p2) &&
p1[4] == p2[4]);
}
@ -132,7 +134,7 @@ static void BuildAndStoreCommandPrefixCode(const uint32_t histogram[128],
BrotliCreateHuffmanTree(histogram, 64, 15, tree, depth);
BrotliCreateHuffmanTree(&histogram[64], 64, 14, tree, &depth[64]);
/* We have to jump through a few hoopes here in order to compute
/* We have to jump through a few hoops here in order to compute
the command bits because the symbols are in a different order than in
the full alphabet. This looks complicated, but having the symbols
in this order in the command bits saves a few branches in the Emit*
@ -200,7 +202,7 @@ static BROTLI_INLINE void EmitInsertLen(size_t insertlen,
} else {
BrotliWriteBits(depth[61], bits[61], storage_ix, storage);
BrotliWriteBits(12, insertlen - 2114, storage_ix, storage);
++histo[21];
++histo[61];
}
}
@ -213,11 +215,11 @@ static BROTLI_INLINE void EmitLongInsertLen(size_t insertlen,
if (insertlen < 22594) {
BrotliWriteBits(depth[62], bits[62], storage_ix, storage);
BrotliWriteBits(14, insertlen - 6210, storage_ix, storage);
++histo[22];
++histo[62];
} else {
BrotliWriteBits(depth[63], bits[63], storage_ix, storage);
BrotliWriteBits(24, insertlen - 22594, storage_ix, storage);
++histo[23];
++histo[63];
}
}
@ -249,7 +251,7 @@ static BROTLI_INLINE void EmitCopyLen(size_t copylen,
} else {
BrotliWriteBits(depth[39], bits[39], storage_ix, storage);
BrotliWriteBits(24, copylen - 2118, storage_ix, storage);
++histo[47];
++histo[39];
}
}
@ -291,7 +293,7 @@ static BROTLI_INLINE void EmitCopyLenLastDistance(size_t copylen,
BrotliWriteBits(depth[39], bits[39], storage_ix, storage);
BrotliWriteBits(24, copylen - 2120, storage_ix, storage);
BrotliWriteBits(depth[64], bits[64], storage_ix, storage);
++histo[47];
++histo[39];
++histo[64];
}
}
@ -322,27 +324,26 @@ static BROTLI_INLINE void EmitLiterals(const uint8_t* input, const size_t len,
}
}
/* REQUIRES: len <= 1 << 20. */
/* REQUIRES: len <= 1 << 24. */
static void BrotliStoreMetaBlockHeader(
size_t len, BROTLI_BOOL is_uncompressed, size_t* storage_ix,
uint8_t* storage) {
size_t nibbles = 6;
/* ISLAST */
BrotliWriteBits(1, 0, storage_ix, storage);
if (len <= (1U << 16)) {
/* MNIBBLES is 4 */
BrotliWriteBits(2, 0, storage_ix, storage);
BrotliWriteBits(16, len - 1, storage_ix, storage);
} else {
/* MNIBBLES is 5 */
BrotliWriteBits(2, 1, storage_ix, storage);
BrotliWriteBits(20, len - 1, storage_ix, storage);
nibbles = 4;
} else if (len <= (1U << 20)) {
nibbles = 5;
}
BrotliWriteBits(2, nibbles - 4, storage_ix, storage);
BrotliWriteBits(nibbles * 4, len - 1, storage_ix, storage);
/* ISUNCOMPRESSED */
BrotliWriteBits(1, (uint64_t)is_uncompressed, storage_ix, storage);
}
static void UpdateBits(size_t n_bits, uint32_t bits, size_t pos,
uint8_t *array) {
uint8_t* array) {
while (n_bits > 0) {
size_t byte_pos = pos >> 3;
size_t n_unchanged_bits = pos & 7;
@ -421,13 +422,11 @@ static uint32_t kCmdHistoSeed[128] = {
1, 1, 1, 1, 0, 0, 0, 0,
};
void BrotliCompressFragmentFast(MemoryManager* m,
const uint8_t* input, size_t input_size,
BROTLI_BOOL is_last,
int* table, size_t table_size,
uint8_t cmd_depth[128], uint16_t cmd_bits[128],
size_t* cmd_code_numbits, uint8_t* cmd_code,
size_t* storage_ix, uint8_t* storage) {
static BROTLI_INLINE void BrotliCompressFragmentFastImpl(
MemoryManager* m, const uint8_t* input, size_t input_size,
BROTLI_BOOL is_last, int* table, size_t table_bits, uint8_t cmd_depth[128],
uint16_t cmd_bits[128], size_t* cmd_code_numbits, uint8_t* cmd_code,
size_t* storage_ix, uint8_t* storage) {
uint32_t cmd_histo[128];
const uint8_t* ip_end;
@ -442,7 +441,7 @@ void BrotliCompressFragmentFast(MemoryManager* m,
static const size_t kFirstBlockSize = 3 << 15;
static const size_t kMergeBlockSize = 1 << 16;
const size_t kInputMarginBytes = 16;
const size_t kInputMarginBytes = BROTLI_WINDOW_GAP;
const size_t kMinMatchLen = 5;
const uint8_t* metablock_start = input;
@ -460,21 +459,7 @@ void BrotliCompressFragmentFast(MemoryManager* m,
const uint8_t* ip;
int last_distance;
const size_t shift = 64u - Log2FloorNonZero(table_size);
assert(table_size);
assert(table_size <= (1u << 31));
/* table must be power of two */
assert((table_size & (table_size - 1)) == 0);
assert(table_size - 1 ==
(size_t)(MAKE_UINT64_T(0xFFFFFFFF, 0xFFFFFF) >> shift));
if (input_size == 0) {
assert(is_last);
BrotliWriteBits(1, 1, storage_ix, storage); /* islast */
BrotliWriteBits(1, 1, storage_ix, storage); /* isempty */
*storage_ix = (*storage_ix + 7u) & ~7u;
return;
}
const size_t shift = 64u - table_bits;
BrotliStoreMetaBlockHeader(block_size, 0, storage_ix, storage);
/* No block splits, no contexts. */
@ -506,7 +491,7 @@ void BrotliCompressFragmentFast(MemoryManager* m,
last_distance = -1;
ip_end = input + block_size;
if (PREDICT_TRUE(block_size >= kInputMarginBytes)) {
if (BROTLI_PREDICT_TRUE(block_size >= kInputMarginBytes)) {
/* For the last block, we need to keep a 16 bytes margin so that we can be
sure that all distances are at most window size - 16.
For all other blocks, we only need to keep a margin of 5 bytes so that
@ -530,40 +515,44 @@ void BrotliCompressFragmentFast(MemoryManager* m,
and doesn't bother looking for matches everywhere.
The "skip" variable keeps track of how many bytes there are since the
last match; dividing it by 32 (ie. right-shifting by five) gives the
last match; dividing it by 32 (i.e. right-shifting by five) gives the
number of bytes to move ahead for each iteration. */
uint32_t skip = 32;
const uint8_t* next_ip = ip;
const uint8_t* candidate;
assert(next_emit < ip);
BROTLI_DCHECK(next_emit < ip);
trawl:
do {
uint32_t hash = next_hash;
uint32_t bytes_between_hash_lookups = skip++ >> 5;
assert(hash == Hash(next_ip, shift));
BROTLI_DCHECK(hash == Hash(next_ip, shift));
ip = next_ip;
next_ip = ip + bytes_between_hash_lookups;
if (PREDICT_FALSE(next_ip > ip_limit)) {
if (BROTLI_PREDICT_FALSE(next_ip > ip_limit)) {
goto emit_remainder;
}
next_hash = Hash(next_ip, shift);
candidate = ip - last_distance;
if (IsMatch(ip, candidate)) {
if (PREDICT_TRUE(candidate < ip)) {
if (BROTLI_PREDICT_TRUE(candidate < ip)) {
table[hash] = (int)(ip - base_ip);
break;
}
}
candidate = base_ip + table[hash];
assert(candidate >= base_ip);
assert(candidate < ip);
BROTLI_DCHECK(candidate >= base_ip);
BROTLI_DCHECK(candidate < ip);
table[hash] = (int)(ip - base_ip);
} while (PREDICT_TRUE(!IsMatch(ip, candidate)));
} while (BROTLI_PREDICT_TRUE(!IsMatch(ip, candidate)));
/* Check copy distance. If candidate is not feasible, continue search.
Checking is done outside of hot loop to reduce overhead. */
if (ip - candidate > MAX_DISTANCE) goto trawl;
/* Step 2: Emit the found match together with the literal bytes from
"next_emit" to the bit stream, and then see if we can find a next macth
"next_emit" to the bit stream, and then see if we can find a next match
immediately afterwards. Repeat until we find no match for the input
without emitting some literal bytes. */
@ -576,8 +565,8 @@ void BrotliCompressFragmentFast(MemoryManager* m,
int distance = (int)(base - candidate); /* > 0 */
size_t insert = (size_t)(base - next_emit);
ip += matched;
assert(0 == memcmp(base, candidate, matched));
if (PREDICT_TRUE(insert < 6210)) {
BROTLI_DCHECK(0 == memcmp(base, candidate, matched));
if (BROTLI_PREDICT_TRUE(insert < 6210)) {
EmitInsertLen(insert, cmd_depth, cmd_bits, cmd_histo,
storage_ix, storage);
} else if (ShouldUseUncompressedMode(metablock_start, next_emit, insert,
@ -606,14 +595,14 @@ void BrotliCompressFragmentFast(MemoryManager* m,
storage_ix, storage);
next_emit = ip;
if (PREDICT_FALSE(ip >= ip_limit)) {
if (BROTLI_PREDICT_FALSE(ip >= ip_limit)) {
goto emit_remainder;
}
/* We could immediately start working at ip now, but to improve
compression we first update "table" with the hashes of some positions
within the last copy. */
{
uint64_t input_bytes = BROTLI_UNALIGNED_LOAD64(ip - 3);
uint64_t input_bytes = BROTLI_UNALIGNED_LOAD64LE(ip - 3);
uint32_t prev_hash = HashBytesAtOffset(input_bytes, 0, shift);
uint32_t cur_hash = HashBytesAtOffset(input_bytes, 3, shift);
table[prev_hash] = (int)(ip - base_ip - 3);
@ -633,23 +622,24 @@ void BrotliCompressFragmentFast(MemoryManager* m,
const uint8_t* base = ip;
size_t matched = 5 + FindMatchLengthWithLimit(
candidate + 5, ip + 5, (size_t)(ip_end - ip) - 5);
if (ip - candidate > MAX_DISTANCE) break;
ip += matched;
last_distance = (int)(base - candidate); /* > 0 */
assert(0 == memcmp(base, candidate, matched));
BROTLI_DCHECK(0 == memcmp(base, candidate, matched));
EmitCopyLen(matched, cmd_depth, cmd_bits, cmd_histo,
storage_ix, storage);
EmitDistance((size_t)last_distance, cmd_depth, cmd_bits,
cmd_histo, storage_ix, storage);
next_emit = ip;
if (PREDICT_FALSE(ip >= ip_limit)) {
if (BROTLI_PREDICT_FALSE(ip >= ip_limit)) {
goto emit_remainder;
}
/* We could immediately start working at ip now, but to improve
compression we first update "table" with the hashes of some positions
within the last copy. */
{
uint64_t input_bytes = BROTLI_UNALIGNED_LOAD64(ip - 3);
uint64_t input_bytes = BROTLI_UNALIGNED_LOAD64LE(ip - 3);
uint32_t prev_hash = HashBytesAtOffset(input_bytes, 0, shift);
uint32_t cur_hash = HashBytesAtOffset(input_bytes, 3, shift);
table[prev_hash] = (int)(ip - base_ip - 3);
@ -668,7 +658,7 @@ void BrotliCompressFragmentFast(MemoryManager* m,
}
emit_remainder:
assert(next_emit <= ip_end);
BROTLI_DCHECK(next_emit <= ip_end);
input += block_size;
input_size -= block_size;
block_size = BROTLI_MIN(size_t, input_size, kMergeBlockSize);
@ -678,7 +668,7 @@ void BrotliCompressFragmentFast(MemoryManager* m,
if (input_size > 0 &&
total_block_size + block_size <= (1 << 20) &&
ShouldMergeBlock(input, block_size, lit_depth)) {
assert(total_block_size > (1 << 16));
BROTLI_DCHECK(total_block_size > (1 << 16));
/* Update the size of the current meta-block and continue emitting commands.
We can do this because the current size and the new size both have 5
nibbles. */
@ -690,7 +680,7 @@ void BrotliCompressFragmentFast(MemoryManager* m,
/* Emit the remaining bytes as literals. */
if (next_emit < ip_end) {
const size_t insert = (size_t)(ip_end - next_emit);
if (PREDICT_TRUE(insert < 6210)) {
if (BROTLI_PREDICT_TRUE(insert < 6210)) {
EmitInsertLen(insert, cmd_depth, cmd_bits, cmd_histo,
storage_ix, storage);
EmitLiterals(next_emit, insert, lit_depth, lit_bits, storage_ix, storage);
@ -728,11 +718,7 @@ next_block:
goto emit_commands;
}
if (is_last) {
BrotliWriteBits(1, 1, storage_ix, storage); /* islast */
BrotliWriteBits(1, 1, storage_ix, storage); /* isempty */
*storage_ix = (*storage_ix + 7u) & ~7u;
} else {
if (!is_last) {
/* If this is not the last block, update the command and distance prefix
codes for the next block and store the compressed forms. */
cmd_code[0] = 0;
@ -742,6 +728,63 @@ next_block:
}
}
#define FOR_TABLE_BITS_(X) X(9) X(11) X(13) X(15)
#define BAKE_METHOD_PARAM_(B) \
static BROTLI_NOINLINE void BrotliCompressFragmentFastImpl ## B( \
MemoryManager* m, const uint8_t* input, size_t input_size, \
BROTLI_BOOL is_last, int* table, uint8_t cmd_depth[128], \
uint16_t cmd_bits[128], size_t* cmd_code_numbits, uint8_t* cmd_code, \
size_t* storage_ix, uint8_t* storage) { \
BrotliCompressFragmentFastImpl(m, input, input_size, is_last, table, B, \
cmd_depth, cmd_bits, cmd_code_numbits, cmd_code, storage_ix, storage); \
}
FOR_TABLE_BITS_(BAKE_METHOD_PARAM_)
#undef BAKE_METHOD_PARAM_
void BrotliCompressFragmentFast(
MemoryManager* m, const uint8_t* input, size_t input_size,
BROTLI_BOOL is_last, int* table, size_t table_size, uint8_t cmd_depth[128],
uint16_t cmd_bits[128], size_t* cmd_code_numbits, uint8_t* cmd_code,
size_t* storage_ix, uint8_t* storage) {
const size_t initial_storage_ix = *storage_ix;
const size_t table_bits = Log2FloorNonZero(table_size);
if (input_size == 0) {
BROTLI_DCHECK(is_last);
BrotliWriteBits(1, 1, storage_ix, storage); /* islast */
BrotliWriteBits(1, 1, storage_ix, storage); /* isempty */
*storage_ix = (*storage_ix + 7u) & ~7u;
return;
}
switch (table_bits) {
#define CASE_(B) \
case B: \
BrotliCompressFragmentFastImpl ## B( \
m, input, input_size, is_last, table, cmd_depth, cmd_bits, \
cmd_code_numbits, cmd_code, storage_ix, storage); \
break;
FOR_TABLE_BITS_(CASE_)
#undef CASE_
default: BROTLI_DCHECK(0); break;
}
/* If output is larger than single uncompressed block, rewrite it. */
if (*storage_ix - initial_storage_ix > 31 + (input_size << 3)) {
EmitUncompressedMetaBlock(input, input + input_size, initial_storage_ix,
storage_ix, storage);
}
if (is_last) {
BrotliWriteBits(1, 1, storage_ix, storage); /* islast */
BrotliWriteBits(1, 1, storage_ix, storage); /* isempty */
*storage_ix = (*storage_ix + 7u) & ~7u;
}
}
#undef FOR_TABLE_BITS_
#if defined(__cplusplus) || defined(c_plusplus)
} /* extern "C" */
#endif

View File

@ -12,9 +12,9 @@
#ifndef BROTLI_ENC_COMPRESS_FRAGMENT_H_
#define BROTLI_ENC_COMPRESS_FRAGMENT_H_
#include "../common/types.h"
#include "../common/platform.h"
#include <brotli/types.h>
#include "./memory.h"
#include "./port.h"
#if defined(__cplusplus) || defined(c_plusplus)
extern "C" {
@ -37,8 +37,11 @@ extern "C" {
updated to represent the updated "cmd_depth" and "cmd_bits".
REQUIRES: "input_size" is greater than zero, or "is_last" is 1.
REQUIRES: "input_size" is less or equal to maximal metablock size (1 << 24).
REQUIRES: All elements in "table[0..table_size-1]" are initialized to zero.
REQUIRES: "table_size" is a power of two */
REQUIRES: "table_size" is an odd (9, 11, 13, 15) power of two
OUTPUT: maximal copy distance <= |input_size|
OUTPUT: maximal copy distance <= BROTLI_MAX_BACKWARD_LIMIT(18) */
BROTLI_INTERNAL void BrotliCompressFragmentFast(MemoryManager* m,
const uint8_t* input,
size_t input_size,

View File

@ -14,49 +14,54 @@
#include <string.h> /* memcmp, memcpy, memset */
#include "../common/types.h"
#include "../common/constants.h"
#include "../common/platform.h"
#include <brotli/types.h>
#include "./bit_cost.h"
#include "./brotli_bit_stream.h"
#include "./entropy_encode.h"
#include "./fast_log.h"
#include "./find_match_length.h"
#include "./memory.h"
#include "./port.h"
#include "./write_bits.h"
#if defined(__cplusplus) || defined(c_plusplus)
extern "C" {
#endif
#define MAX_DISTANCE (long)BROTLI_MAX_BACKWARD_LIMIT(18)
/* kHashMul32 multiplier has these properties:
* The multiplier must be odd. Otherwise we may lose the highest bit.
* No long streaks of 1s or 0s.
* No long streaks of ones or zeros.
* There is no effort to ensure that it is a prime, the oddity is enough
for this use.
* The number has been tuned heuristically against compression benchmarks. */
static const uint32_t kHashMul32 = 0x1e35a7bd;
static const uint32_t kHashMul32 = 0x1E35A7BD;
static BROTLI_INLINE uint32_t Hash(const uint8_t* p, size_t shift) {
const uint64_t h = (BROTLI_UNALIGNED_LOAD64(p) << 16) * kHashMul32;
static BROTLI_INLINE uint32_t Hash(const uint8_t* p,
size_t shift, size_t length) {
const uint64_t h =
(BROTLI_UNALIGNED_LOAD64LE(p) << ((8 - length) * 8)) * kHashMul32;
return (uint32_t)(h >> shift);
}
static BROTLI_INLINE uint32_t HashBytesAtOffset(
uint64_t v, int offset, size_t shift) {
assert(offset >= 0);
assert(offset <= 2);
static BROTLI_INLINE uint32_t HashBytesAtOffset(uint64_t v, size_t offset,
size_t shift, size_t length) {
BROTLI_DCHECK(offset <= 8 - length);
{
const uint64_t h = ((v >> (8 * offset)) << 16) * kHashMul32;
const uint64_t h = ((v >> (8 * offset)) << ((8 - length) * 8)) * kHashMul32;
return (uint32_t)(h >> shift);
}
}
static BROTLI_INLINE BROTLI_BOOL IsMatch(const uint8_t* p1, const uint8_t* p2) {
return TO_BROTLI_BOOL(
BROTLI_UNALIGNED_LOAD32(p1) == BROTLI_UNALIGNED_LOAD32(p2) &&
p1[4] == p2[4] &&
p1[5] == p2[5]);
static BROTLI_INLINE BROTLI_BOOL IsMatch(const uint8_t* p1, const uint8_t* p2,
size_t length) {
if (BrotliUnalignedRead32(p1) == BrotliUnalignedRead32(p2)) {
if (length == 4) return BROTLI_TRUE;
return TO_BROTLI_BOOL(p1[4] == p2[4] && p1[5] == p2[5]);
}
return BROTLI_FALSE;
}
/* Builds a command and distance prefix code (each 64 symbols) into "depth" and
@ -71,7 +76,7 @@ static void BuildAndStoreCommandPrefixCode(
uint16_t cmd_bits[64];
BrotliCreateHuffmanTree(histogram, 64, 15, tree, depth);
BrotliCreateHuffmanTree(&histogram[64], 64, 14, tree, &depth[64]);
/* We have to jump through a few hoopes here in order to compute
/* We have to jump through a few hoops here in order to compute
the command bits because the symbols are in a different order than in
the full alphabet. This looks complicated, but having the symbols
in this order in the command bits saves a few branches in the Emit*
@ -213,31 +218,31 @@ static BROTLI_INLINE void EmitDistance(uint32_t distance, uint32_t** commands) {
++(*commands);
}
/* REQUIRES: len <= 1 << 20. */
/* REQUIRES: len <= 1 << 24. */
static void BrotliStoreMetaBlockHeader(
size_t len, BROTLI_BOOL is_uncompressed, size_t* storage_ix,
uint8_t* storage) {
size_t nibbles = 6;
/* ISLAST */
BrotliWriteBits(1, 0, storage_ix, storage);
if (len <= (1U << 16)) {
/* MNIBBLES is 4 */
BrotliWriteBits(2, 0, storage_ix, storage);
BrotliWriteBits(16, len - 1, storage_ix, storage);
} else {
/* MNIBBLES is 5 */
BrotliWriteBits(2, 1, storage_ix, storage);
BrotliWriteBits(20, len - 1, storage_ix, storage);
nibbles = 4;
} else if (len <= (1U << 20)) {
nibbles = 5;
}
BrotliWriteBits(2, nibbles - 4, storage_ix, storage);
BrotliWriteBits(nibbles * 4, len - 1, storage_ix, storage);
/* ISUNCOMPRESSED */
BrotliWriteBits(1, (uint64_t)is_uncompressed, storage_ix, storage);
}
static void CreateCommands(const uint8_t* input, size_t block_size,
size_t input_size, const uint8_t* base_ip, int* table, size_t table_size,
static BROTLI_INLINE void CreateCommands(const uint8_t* input,
size_t block_size, size_t input_size, const uint8_t* base_ip, int* table,
size_t table_bits, size_t min_match,
uint8_t** literals, uint32_t** commands) {
/* "ip" is the input pointer. */
const uint8_t* ip = input;
const size_t shift = 64u - Log2FloorNonZero(table_size);
const size_t shift = 64u - table_bits;
const uint8_t* ip_end = input + block_size;
/* "next_emit" is a pointer to the first byte that is not covered by a
previous copy. Bytes between "next_emit" and the start of the next copy or
@ -245,27 +250,19 @@ static void CreateCommands(const uint8_t* input, size_t block_size,
const uint8_t* next_emit = input;
int last_distance = -1;
const size_t kInputMarginBytes = 16;
const size_t kMinMatchLen = 6;
const size_t kInputMarginBytes = BROTLI_WINDOW_GAP;
assert(table_size);
assert(table_size <= (1u << 31));
/* table must be power of two */
assert((table_size & (table_size - 1)) == 0);
assert(table_size - 1 ==
(size_t)(MAKE_UINT64_T(0xFFFFFFFF, 0xFFFFFF) >> shift));
if (PREDICT_TRUE(block_size >= kInputMarginBytes)) {
if (BROTLI_PREDICT_TRUE(block_size >= kInputMarginBytes)) {
/* For the last block, we need to keep a 16 bytes margin so that we can be
sure that all distances are at most window size - 16.
For all other blocks, we only need to keep a margin of 5 bytes so that
we don't go over the block size with a copy. */
const size_t len_limit = BROTLI_MIN(size_t, block_size - kMinMatchLen,
const size_t len_limit = BROTLI_MIN(size_t, block_size - min_match,
input_size - kInputMarginBytes);
const uint8_t* ip_limit = input + len_limit;
uint32_t next_hash;
for (next_hash = Hash(++ip, shift); ; ) {
for (next_hash = Hash(++ip, shift, min_match); ; ) {
/* Step 1: Scan forward in the input looking for a 6-byte-long match.
If we get close to exhausting the input then goto emit_remainder.
@ -286,34 +283,38 @@ static void CreateCommands(const uint8_t* input, size_t block_size,
const uint8_t* next_ip = ip;
const uint8_t* candidate;
assert(next_emit < ip);
BROTLI_DCHECK(next_emit < ip);
trawl:
do {
uint32_t hash = next_hash;
uint32_t bytes_between_hash_lookups = skip++ >> 5;
ip = next_ip;
assert(hash == Hash(ip, shift));
BROTLI_DCHECK(hash == Hash(ip, shift, min_match));
next_ip = ip + bytes_between_hash_lookups;
if (PREDICT_FALSE(next_ip > ip_limit)) {
if (BROTLI_PREDICT_FALSE(next_ip > ip_limit)) {
goto emit_remainder;
}
next_hash = Hash(next_ip, shift);
next_hash = Hash(next_ip, shift, min_match);
candidate = ip - last_distance;
if (IsMatch(ip, candidate)) {
if (PREDICT_TRUE(candidate < ip)) {
if (IsMatch(ip, candidate, min_match)) {
if (BROTLI_PREDICT_TRUE(candidate < ip)) {
table[hash] = (int)(ip - base_ip);
break;
}
}
candidate = base_ip + table[hash];
assert(candidate >= base_ip);
assert(candidate < ip);
BROTLI_DCHECK(candidate >= base_ip);
BROTLI_DCHECK(candidate < ip);
table[hash] = (int)(ip - base_ip);
} while (PREDICT_TRUE(!IsMatch(ip, candidate)));
} while (BROTLI_PREDICT_TRUE(!IsMatch(ip, candidate, min_match)));
/* Check copy distance. If candidate is not feasible, continue search.
Checking is done outside of hot loop to reduce overhead. */
if (ip - candidate > MAX_DISTANCE) goto trawl;
/* Step 2: Emit the found match together with the literal bytes from
"next_emit", and then see if we can find a next macth immediately
"next_emit", and then see if we can find a next match immediately
afterwards. Repeat until we find no match for the input
without emitting some literal bytes. */
@ -321,12 +322,13 @@ static void CreateCommands(const uint8_t* input, size_t block_size,
/* We have a 6-byte match at ip, and we need to emit bytes in
[next_emit, ip). */
const uint8_t* base = ip;
size_t matched = 6 + FindMatchLengthWithLimit(
candidate + 6, ip + 6, (size_t)(ip_end - ip) - 6);
size_t matched = min_match + FindMatchLengthWithLimit(
candidate + min_match, ip + min_match,
(size_t)(ip_end - ip) - min_match);
int distance = (int)(base - candidate); /* > 0 */
int insert = (int)(base - next_emit);
ip += matched;
assert(0 == memcmp(base, candidate, matched));
BROTLI_DCHECK(0 == memcmp(base, candidate, matched));
EmitInsertLen((uint32_t)insert, commands);
memcpy(*literals, next_emit, (size_t)insert);
*literals += insert;
@ -340,79 +342,107 @@ static void CreateCommands(const uint8_t* input, size_t block_size,
EmitCopyLenLastDistance(matched, commands);
next_emit = ip;
if (PREDICT_FALSE(ip >= ip_limit)) {
if (BROTLI_PREDICT_FALSE(ip >= ip_limit)) {
goto emit_remainder;
}
{
/* We could immediately start working at ip now, but to improve
compression we first update "table" with the hashes of some
positions within the last copy. */
uint64_t input_bytes = BROTLI_UNALIGNED_LOAD64(ip - 5);
uint32_t prev_hash = HashBytesAtOffset(input_bytes, 0, shift);
uint64_t input_bytes;
uint32_t cur_hash;
table[prev_hash] = (int)(ip - base_ip - 5);
prev_hash = HashBytesAtOffset(input_bytes, 1, shift);
table[prev_hash] = (int)(ip - base_ip - 4);
prev_hash = HashBytesAtOffset(input_bytes, 2, shift);
table[prev_hash] = (int)(ip - base_ip - 3);
input_bytes = BROTLI_UNALIGNED_LOAD64(ip - 2);
cur_hash = HashBytesAtOffset(input_bytes, 2, shift);
prev_hash = HashBytesAtOffset(input_bytes, 0, shift);
table[prev_hash] = (int)(ip - base_ip - 2);
prev_hash = HashBytesAtOffset(input_bytes, 1, shift);
table[prev_hash] = (int)(ip - base_ip - 1);
uint32_t prev_hash;
if (min_match == 4) {
input_bytes = BROTLI_UNALIGNED_LOAD64LE(ip - 3);
cur_hash = HashBytesAtOffset(input_bytes, 3, shift, min_match);
prev_hash = HashBytesAtOffset(input_bytes, 0, shift, min_match);
table[prev_hash] = (int)(ip - base_ip - 3);
prev_hash = HashBytesAtOffset(input_bytes, 1, shift, min_match);
table[prev_hash] = (int)(ip - base_ip - 2);
prev_hash = HashBytesAtOffset(input_bytes, 0, shift, min_match);
table[prev_hash] = (int)(ip - base_ip - 1);
} else {
input_bytes = BROTLI_UNALIGNED_LOAD64LE(ip - 5);
prev_hash = HashBytesAtOffset(input_bytes, 0, shift, min_match);
table[prev_hash] = (int)(ip - base_ip - 5);
prev_hash = HashBytesAtOffset(input_bytes, 1, shift, min_match);
table[prev_hash] = (int)(ip - base_ip - 4);
prev_hash = HashBytesAtOffset(input_bytes, 2, shift, min_match);
table[prev_hash] = (int)(ip - base_ip - 3);
input_bytes = BROTLI_UNALIGNED_LOAD64LE(ip - 2);
cur_hash = HashBytesAtOffset(input_bytes, 2, shift, min_match);
prev_hash = HashBytesAtOffset(input_bytes, 0, shift, min_match);
table[prev_hash] = (int)(ip - base_ip - 2);
prev_hash = HashBytesAtOffset(input_bytes, 1, shift, min_match);
table[prev_hash] = (int)(ip - base_ip - 1);
}
candidate = base_ip + table[cur_hash];
table[cur_hash] = (int)(ip - base_ip);
}
}
while (IsMatch(ip, candidate)) {
while (ip - candidate <= MAX_DISTANCE &&
IsMatch(ip, candidate, min_match)) {
/* We have a 6-byte match at ip, and no need to emit any
literal bytes prior to ip. */
const uint8_t* base = ip;
size_t matched = 6 + FindMatchLengthWithLimit(
candidate + 6, ip + 6, (size_t)(ip_end - ip) - 6);
size_t matched = min_match + FindMatchLengthWithLimit(
candidate + min_match, ip + min_match,
(size_t)(ip_end - ip) - min_match);
ip += matched;
last_distance = (int)(base - candidate); /* > 0 */
assert(0 == memcmp(base, candidate, matched));
BROTLI_DCHECK(0 == memcmp(base, candidate, matched));
EmitCopyLen(matched, commands);
EmitDistance((uint32_t)last_distance, commands);
next_emit = ip;
if (PREDICT_FALSE(ip >= ip_limit)) {
if (BROTLI_PREDICT_FALSE(ip >= ip_limit)) {
goto emit_remainder;
}
{
/* We could immediately start working at ip now, but to improve
compression we first update "table" with the hashes of some
positions within the last copy. */
uint64_t input_bytes = BROTLI_UNALIGNED_LOAD64(ip - 5);
uint32_t prev_hash = HashBytesAtOffset(input_bytes, 0, shift);
uint64_t input_bytes;
uint32_t cur_hash;
table[prev_hash] = (int)(ip - base_ip - 5);
prev_hash = HashBytesAtOffset(input_bytes, 1, shift);
table[prev_hash] = (int)(ip - base_ip - 4);
prev_hash = HashBytesAtOffset(input_bytes, 2, shift);
table[prev_hash] = (int)(ip - base_ip - 3);
input_bytes = BROTLI_UNALIGNED_LOAD64(ip - 2);
cur_hash = HashBytesAtOffset(input_bytes, 2, shift);
prev_hash = HashBytesAtOffset(input_bytes, 0, shift);
table[prev_hash] = (int)(ip - base_ip - 2);
prev_hash = HashBytesAtOffset(input_bytes, 1, shift);
table[prev_hash] = (int)(ip - base_ip - 1);
uint32_t prev_hash;
if (min_match == 4) {
input_bytes = BROTLI_UNALIGNED_LOAD64LE(ip - 3);
cur_hash = HashBytesAtOffset(input_bytes, 3, shift, min_match);
prev_hash = HashBytesAtOffset(input_bytes, 0, shift, min_match);
table[prev_hash] = (int)(ip - base_ip - 3);
prev_hash = HashBytesAtOffset(input_bytes, 1, shift, min_match);
table[prev_hash] = (int)(ip - base_ip - 2);
prev_hash = HashBytesAtOffset(input_bytes, 2, shift, min_match);
table[prev_hash] = (int)(ip - base_ip - 1);
} else {
input_bytes = BROTLI_UNALIGNED_LOAD64LE(ip - 5);
prev_hash = HashBytesAtOffset(input_bytes, 0, shift, min_match);
table[prev_hash] = (int)(ip - base_ip - 5);
prev_hash = HashBytesAtOffset(input_bytes, 1, shift, min_match);
table[prev_hash] = (int)(ip - base_ip - 4);
prev_hash = HashBytesAtOffset(input_bytes, 2, shift, min_match);
table[prev_hash] = (int)(ip - base_ip - 3);
input_bytes = BROTLI_UNALIGNED_LOAD64LE(ip - 2);
cur_hash = HashBytesAtOffset(input_bytes, 2, shift, min_match);
prev_hash = HashBytesAtOffset(input_bytes, 0, shift, min_match);
table[prev_hash] = (int)(ip - base_ip - 2);
prev_hash = HashBytesAtOffset(input_bytes, 1, shift, min_match);
table[prev_hash] = (int)(ip - base_ip - 1);
}
candidate = base_ip + table[cur_hash];
table[cur_hash] = (int)(ip - base_ip);
}
}
next_hash = Hash(++ip, shift);
next_hash = Hash(++ip, shift, min_match);
}
}
emit_remainder:
assert(next_emit <= ip_end);
BROTLI_DCHECK(next_emit <= ip_end);
/* Emit the remaining bytes as literals. */
if (next_emit < ip_end) {
const uint32_t insert = (uint32_t)(ip_end - next_emit);
@ -457,7 +487,9 @@ static void StoreCommands(MemoryManager* m,
if (BROTLI_IS_OOM(m)) return;
for (i = 0; i < num_commands; ++i) {
++cmd_histo[commands[i] & 0xff];
const uint32_t code = commands[i] & 0xFF;
BROTLI_DCHECK(code < 128);
++cmd_histo[code];
}
cmd_histo[1] += 1;
cmd_histo[2] += 1;
@ -468,8 +500,9 @@ static void StoreCommands(MemoryManager* m,
for (i = 0; i < num_commands; ++i) {
const uint32_t cmd = commands[i];
const uint32_t code = cmd & 0xff;
const uint32_t code = cmd & 0xFF;
const uint32_t extra = cmd >> 8;
BROTLI_DCHECK(code < 128);
BrotliWriteBits(cmd_depths[code], cmd_bits[code], storage_ix, storage);
BrotliWriteBits(kNumExtraBits[code], extra, storage_ix, storage);
if (code < 24) {
@ -504,15 +537,32 @@ static BROTLI_BOOL ShouldCompress(
}
}
void BrotliCompressFragmentTwoPass(MemoryManager* m,
const uint8_t* input, size_t input_size,
BROTLI_BOOL is_last,
uint32_t* command_buf, uint8_t* literal_buf,
int* table, size_t table_size,
size_t* storage_ix, uint8_t* storage) {
static void RewindBitPosition(const size_t new_storage_ix,
size_t* storage_ix, uint8_t* storage) {
const size_t bitpos = new_storage_ix & 7;
const size_t mask = (1u << bitpos) - 1;
storage[new_storage_ix >> 3] &= (uint8_t)mask;
*storage_ix = new_storage_ix;
}
static void EmitUncompressedMetaBlock(const uint8_t* input, size_t input_size,
size_t* storage_ix, uint8_t* storage) {
BrotliStoreMetaBlockHeader(input_size, 1, storage_ix, storage);
*storage_ix = (*storage_ix + 7u) & ~7u;
memcpy(&storage[*storage_ix >> 3], input, input_size);
*storage_ix += input_size << 3;
storage[*storage_ix >> 3] = 0;
}
static BROTLI_INLINE void BrotliCompressFragmentTwoPassImpl(
MemoryManager* m, const uint8_t* input, size_t input_size,
BROTLI_BOOL is_last, uint32_t* command_buf, uint8_t* literal_buf,
int* table, size_t table_bits, size_t min_match,
size_t* storage_ix, uint8_t* storage) {
/* Save the start of the first block for position and distance computations.
*/
const uint8_t* base_ip = input;
BROTLI_UNUSED(is_last);
while (input_size > 0) {
size_t block_size =
@ -520,8 +570,8 @@ void BrotliCompressFragmentTwoPass(MemoryManager* m,
uint32_t* commands = command_buf;
uint8_t* literals = literal_buf;
size_t num_literals;
CreateCommands(input, block_size, input_size, base_ip, table, table_size,
&literals, &commands);
CreateCommands(input, block_size, input_size, base_ip, table,
table_bits, min_match, &literals, &commands);
num_literals = (size_t)(literals - literal_buf);
if (ShouldCompress(input, block_size, num_literals)) {
const size_t num_commands = (size_t)(commands - command_buf);
@ -535,15 +585,51 @@ void BrotliCompressFragmentTwoPass(MemoryManager* m,
/* Since we did not find many backward references and the entropy of
the data is close to 8 bits, we can simply emit an uncompressed block.
This makes compression speed of uncompressible data about 3x faster. */
BrotliStoreMetaBlockHeader(block_size, 1, storage_ix, storage);
*storage_ix = (*storage_ix + 7u) & ~7u;
memcpy(&storage[*storage_ix >> 3], input, block_size);
*storage_ix += block_size << 3;
storage[*storage_ix >> 3] = 0;
EmitUncompressedMetaBlock(input, block_size, storage_ix, storage);
}
input += block_size;
input_size -= block_size;
}
}
#define FOR_TABLE_BITS_(X) \
X(8) X(9) X(10) X(11) X(12) X(13) X(14) X(15) X(16) X(17)
#define BAKE_METHOD_PARAM_(B) \
static BROTLI_NOINLINE void BrotliCompressFragmentTwoPassImpl ## B( \
MemoryManager* m, const uint8_t* input, size_t input_size, \
BROTLI_BOOL is_last, uint32_t* command_buf, uint8_t* literal_buf, \
int* table, size_t* storage_ix, uint8_t* storage) { \
size_t min_match = (B <= 15) ? 4 : 6; \
BrotliCompressFragmentTwoPassImpl(m, input, input_size, is_last, command_buf,\
literal_buf, table, B, min_match, storage_ix, storage); \
}
FOR_TABLE_BITS_(BAKE_METHOD_PARAM_)
#undef BAKE_METHOD_PARAM_
void BrotliCompressFragmentTwoPass(
MemoryManager* m, const uint8_t* input, size_t input_size,
BROTLI_BOOL is_last, uint32_t* command_buf, uint8_t* literal_buf,
int* table, size_t table_size, size_t* storage_ix, uint8_t* storage) {
const size_t initial_storage_ix = *storage_ix;
const size_t table_bits = Log2FloorNonZero(table_size);
switch (table_bits) {
#define CASE_(B) \
case B: \
BrotliCompressFragmentTwoPassImpl ## B( \
m, input, input_size, is_last, command_buf, \
literal_buf, table, storage_ix, storage); \
break;
FOR_TABLE_BITS_(CASE_)
#undef CASE_
default: BROTLI_DCHECK(0); break;
}
/* If output is larger than single uncompressed block, rewrite it. */
if (*storage_ix - initial_storage_ix > 31 + (input_size << 3)) {
RewindBitPosition(initial_storage_ix, storage_ix, storage);
EmitUncompressedMetaBlock(input, input_size, storage_ix, storage);
}
if (is_last) {
BrotliWriteBits(1, 1, storage_ix, storage); /* islast */
@ -552,6 +638,8 @@ void BrotliCompressFragmentTwoPass(MemoryManager* m,
}
}
#undef FOR_TABLE_BITS_
#if defined(__cplusplus) || defined(c_plusplus)
} /* extern "C" */
#endif

View File

@ -13,9 +13,9 @@
#ifndef BROTLI_ENC_COMPRESS_FRAGMENT_TWO_PASS_H_
#define BROTLI_ENC_COMPRESS_FRAGMENT_TWO_PASS_H_
#include "../common/types.h"
#include "../common/platform.h"
#include <brotli/types.h>
#include "./memory.h"
#include "./port.h"
#if defined(__cplusplus) || defined(c_plusplus)
extern "C" {
@ -29,10 +29,13 @@ static const size_t kCompressFragmentTwoPassBlockSize = 1 << 17;
If "is_last" is 1, emits an additional empty last meta-block.
REQUIRES: "input_size" is greater than zero, or "is_last" is 1.
REQUIRES: "input_size" is less or equal to maximal metablock size (1 << 24).
REQUIRES: "command_buf" and "literal_buf" point to at least
kCompressFragmentTwoPassBlockSize long arrays.
REQUIRES: All elements in "table[0..table_size-1]" are initialized to zero.
REQUIRES: "table_size" is a power of two */
REQUIRES: "table_size" is a power of two
OUTPUT: maximal copy distance <= |input_size|
OUTPUT: maximal copy distance <= BROTLI_MAX_BACKWARD_LIMIT(18) */
BROTLI_INTERNAL void BrotliCompressFragmentTwoPass(MemoryManager* m,
const uint8_t* input,
size_t input_size,

View File

@ -1,161 +0,0 @@
/* Copyright 2016 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* C++ API for Brotli compression. */
#ifndef BROTLI_ENC_COMPRESSOR_H_
#define BROTLI_ENC_COMPRESSOR_H_
#include "./encode.h"
#include "./streams.h"
namespace brotli {
static const int kMinWindowBits = kBrotliMinWindowBits;
static const int kMaxWindowBits = kBrotliMaxWindowBits;
static const int kMinInputBlockBits = kBrotliMinInputBlockBits;
static const int kMaxInputBlockBits = kBrotliMaxInputBlockBits;
struct BrotliParams {
BrotliParams(void)
: mode(MODE_GENERIC),
quality(11),
lgwin(22),
lgblock(0),
enable_dictionary(true),
enable_transforms(false),
greedy_block_split(false),
enable_context_modeling(true) {}
enum Mode {
/* Default compression mode. The compressor does not know anything in
advance about the properties of the input. */
MODE_GENERIC = 0,
/* Compression mode for UTF-8 format text input. */
MODE_TEXT = 1,
/* Compression mode used in WOFF 2.0. */
MODE_FONT = 2
};
Mode mode;
/* Controls the compression-speed vs compression-density tradeoffs. The higher
the |quality|, the slower the compression. Range is 0 to 11. */
int quality;
/* Base 2 logarithm of the sliding window size. Range is 10 to 24. */
int lgwin;
/* Base 2 logarithm of the maximum input block size. Range is 16 to 24.
If set to 0, the value will be set based on the quality. */
int lgblock;
/* These settings are deprecated and will be ignored.
All speed vs. size compromises are controlled by the |quality| param. */
bool enable_dictionary;
bool enable_transforms;
bool greedy_block_split;
bool enable_context_modeling;
};
/* An instance can not be reused for multiple brotli streams. */
class BrotliCompressor {
public:
explicit BrotliCompressor(BrotliParams params);
~BrotliCompressor(void);
/* The maximum input size that can be processed at once. */
size_t input_block_size(void) const {
return BrotliEncoderInputBlockSize(state_);
}
/* Encodes the data in |input_buffer| as a meta-block and writes it to
|encoded_buffer| (|*encoded_size should| be set to the size of
|encoded_buffer|) and sets |*encoded_size| to the number of bytes that
was written. The |input_size| must not be greater than input_block_size().
Returns false if there was an error and true otherwise. */
bool WriteMetaBlock(const size_t input_size,
const uint8_t* input_buffer,
const bool is_last,
size_t* encoded_size,
uint8_t* encoded_buffer);
/* Writes a metadata meta-block containing the given input to encoded_buffer.
|*encoded_size| should be set to the size of the encoded_buffer.
Sets |*encoded_size| to the number of bytes that was written.
Note that the given input data will not be part of the sliding window and
thus no backward references can be made to this data from subsequent
metablocks. |input_size| must not be greater than 2^24 and provided
|*encoded_size| must not be less than |input_size| + 6.
Returns false if there was an error and true otherwise. */
bool WriteMetadata(const size_t input_size,
const uint8_t* input_buffer,
const bool is_last,
size_t* encoded_size,
uint8_t* encoded_buffer);
/* Writes a zero-length meta-block with end-of-input bit set to the
internal output buffer and copies the output buffer to |encoded_buffer|
(|*encoded_size| should be set to the size of |encoded_buffer|) and sets
|*encoded_size| to the number of bytes written.
Returns false if there was an error and true otherwise. */
bool FinishStream(size_t* encoded_size, uint8_t* encoded_buffer);
/* Copies the given input data to the internal ring buffer of the compressor.
No processing of the data occurs at this time and this function can be
called multiple times before calling WriteBrotliData() to process the
accumulated input. At most input_block_size() bytes of input data can be
copied to the ring buffer, otherwise the next WriteBrotliData() will fail.
*/
void CopyInputToRingBuffer(const size_t input_size,
const uint8_t* input_buffer);
/* Processes the accumulated input data and sets |*out_size| to the length of
the new output meta-block, or to zero if no new output meta-block has been
created (in this case the processed input data is buffered internally).
If |*out_size| is positive, |*output| points to the start of the output
data. If |is_last| or |force_flush| is true, an output meta-block is always
created. However, until |is_last| is true encoder may retain up to 7 bits
of the last byte of output. To force encoder to dump the remaining bits
use WriteMetadata() to append an empty meta-data block.
Returns false if the size of the input data is larger than
input_block_size(). */
bool WriteBrotliData(const bool is_last, const bool force_flush,
size_t* out_size, uint8_t** output);
/* Fills the new state with a dictionary for LZ77, warming up the ringbuffer,
e.g. for custom static dictionaries for data formats.
Not to be confused with the built-in transformable dictionary of Brotli.
To decode, use BrotliSetCustomDictionary() of the decoder with the same
dictionary. */
void BrotliSetCustomDictionary(size_t size, const uint8_t* dict);
/* No-op, but we keep it here for API backward-compatibility. */
void WriteStreamHeader(void) {}
private:
BrotliEncoderState* state_;
};
/* Compresses the data in |input_buffer| into |encoded_buffer|, and sets
|*encoded_size| to the compressed length.
Returns 0 if there was an error and 1 otherwise. */
int BrotliCompressBuffer(BrotliParams params,
size_t input_size,
const uint8_t* input_buffer,
size_t* encoded_size,
uint8_t* encoded_buffer);
/* Same as above, but uses the specified input and output classes instead
of reading from and writing to pre-allocated memory buffers. */
int BrotliCompress(BrotliParams params, BrotliIn* in, BrotliOut* out);
/* Before compressing the data, sets a custom LZ77 dictionary with
BrotliCompressor::BrotliSetCustomDictionary. */
int BrotliCompressWithCustomDictionary(size_t dictsize, const uint8_t* dict,
BrotliParams params,
BrotliIn* in, BrotliOut* out);
} /* namespace brotli */
#endif /* BROTLI_ENC_COMPRESSOR_H_ */

View File

@ -1,184 +0,0 @@
/* Copyright 2013 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* Functions to map previous bytes into a context id. */
#ifndef BROTLI_ENC_CONTEXT_H_
#define BROTLI_ENC_CONTEXT_H_
#include "../common/types.h"
#include "../common/port.h"
#if defined(__cplusplus) || defined(c_plusplus)
extern "C" {
#endif
/* Second-order context lookup table for UTF8 byte streams.
If p1 and p2 are the previous two bytes, we calculate the context as
context = kUTF8ContextLookup[p1] | kUTF8ContextLookup[p2 + 256].
If the previous two bytes are ASCII characters (i.e. < 128), this will be
equivalent to
context = 4 * context1(p1) + context2(p2),
where context1 is based on the previous byte in the following way:
0 : non-ASCII control
1 : \t, \n, \r
2 : space
3 : other punctuation
4 : " '
5 : %
6 : ( < [ {
7 : ) > ] }
8 : , ; :
9 : .
10 : =
11 : number
12 : upper-case vowel
13 : upper-case consonant
14 : lower-case vowel
15 : lower-case consonant
and context2 is based on the second last byte:
0 : control, space
1 : punctuation
2 : upper-case letter, number
3 : lower-case letter
If the last byte is ASCII, and the second last byte is not (in a valid UTF8
stream it will be a continuation byte, value between 128 and 191), the
context is the same as if the second last byte was an ASCII control or space.
If the last byte is a UTF8 lead byte (value >= 192), then the next byte will
be a continuation byte and the context id is 2 or 3 depending on the LSB of
the last byte and to a lesser extent on the second last byte if it is ASCII.
If the last byte is a UTF8 continuation byte, the second last byte can be:
- continuation byte: the next byte is probably ASCII or lead byte (assuming
4-byte UTF8 characters are rare) and the context id is 0 or 1.
- lead byte (192 - 207): next byte is ASCII or lead byte, context is 0 or 1
- lead byte (208 - 255): next byte is continuation byte, context is 2 or 3
The possible value combinations of the previous two bytes, the range of
context ids and the type of the next byte is summarized in the table below:
|--------\-----------------------------------------------------------------|
| \ Last byte |
| Second \---------------------------------------------------------------|
| last byte \ ASCII | cont. byte | lead byte |
| \ (0-127) | (128-191) | (192-) |
|=============|===================|=====================|==================|
| ASCII | next: ASCII/lead | not valid | next: cont. |
| (0-127) | context: 4 - 63 | | context: 2 - 3 |
|-------------|-------------------|---------------------|------------------|
| cont. byte | next: ASCII/lead | next: ASCII/lead | next: cont. |
| (128-191) | context: 4 - 63 | context: 0 - 1 | context: 2 - 3 |
|-------------|-------------------|---------------------|------------------|
| lead byte | not valid | next: ASCII/lead | not valid |
| (192-207) | | context: 0 - 1 | |
|-------------|-------------------|---------------------|------------------|
| lead byte | not valid | next: cont. | not valid |
| (208-) | | context: 2 - 3 | |
|-------------|-------------------|---------------------|------------------|
*/
static const uint8_t kUTF8ContextLookup[512] = {
/* Last byte. */
/* */
/* ASCII range. */
0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 4, 0, 0, 4, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
8, 12, 16, 12, 12, 20, 12, 16, 24, 28, 12, 12, 32, 12, 36, 12,
44, 44, 44, 44, 44, 44, 44, 44, 44, 44, 32, 32, 24, 40, 28, 12,
12, 48, 52, 52, 52, 48, 52, 52, 52, 48, 52, 52, 52, 52, 52, 48,
52, 52, 52, 52, 52, 48, 52, 52, 52, 52, 52, 24, 12, 28, 12, 12,
12, 56, 60, 60, 60, 56, 60, 60, 60, 56, 60, 60, 60, 60, 60, 56,
60, 60, 60, 60, 60, 56, 60, 60, 60, 60, 60, 24, 12, 28, 12, 0,
/* UTF8 continuation byte range. */
0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1,
0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1,
0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1,
0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1,
/* UTF8 lead byte range. */
2, 3, 2, 3, 2, 3, 2, 3, 2, 3, 2, 3, 2, 3, 2, 3,
2, 3, 2, 3, 2, 3, 2, 3, 2, 3, 2, 3, 2, 3, 2, 3,
2, 3, 2, 3, 2, 3, 2, 3, 2, 3, 2, 3, 2, 3, 2, 3,
2, 3, 2, 3, 2, 3, 2, 3, 2, 3, 2, 3, 2, 3, 2, 3,
/* Second last byte. */
/* */
/* ASCII range. */
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1,
1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1,
1, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 1, 1, 1, 1, 0,
/* UTF8 continuation byte range. */
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
/* UTF8 lead byte range. */
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
};
/* Context lookup table for small signed integers. */
static const uint8_t kSigned3BitContextLookup[] = {
0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5,
5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5,
5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5,
6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 7,
};
typedef enum ContextType {
CONTEXT_LSB6 = 0,
CONTEXT_MSB6 = 1,
CONTEXT_UTF8 = 2,
CONTEXT_SIGNED = 3
} ContextType;
static BROTLI_INLINE uint8_t Context(uint8_t p1, uint8_t p2, ContextType mode) {
switch (mode) {
case CONTEXT_LSB6:
return p1 & 0x3f;
case CONTEXT_MSB6:
return (uint8_t)(p1 >> 2);
case CONTEXT_UTF8:
return kUTF8ContextLookup[p1] | kUTF8ContextLookup[p2 + 256];
case CONTEXT_SIGNED:
return (uint8_t)((kSigned3BitContextLookup[p1] << 3) +
kSigned3BitContextLookup[p2]);
default:
return 0;
}
}
#if defined(__cplusplus) || defined(c_plusplus)
} /* extern "C" */
#endif
#endif /* BROTLI_ENC_CONTEXT_H_ */

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@ -1,221 +0,0 @@
/* Copyright 2013 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* API for Brotli compression. */
#ifndef BROTLI_ENC_ENCODE_H_
#define BROTLI_ENC_ENCODE_H_
#include "../common/types.h"
#if defined(__cplusplus) || defined(c_plusplus)
extern "C" {
#endif
static const int kBrotliMaxWindowBits = 24;
static const int kBrotliMinWindowBits = 10;
static const int kBrotliMinInputBlockBits = 16;
static const int kBrotliMaxInputBlockBits = 24;
#define BROTLI_MIN_QUALITY 0
#define BROTLI_MAX_QUALITY 11
typedef enum BrotliEncoderMode {
/* Default compression mode. The compressor does not know anything in
advance about the properties of the input. */
BROTLI_MODE_GENERIC = 0,
/* Compression mode for UTF-8 format text input. */
BROTLI_MODE_TEXT = 1,
/* Compression mode used in WOFF 2.0. */
BROTLI_MODE_FONT = 2
} BrotliEncoderMode;
#define BROTLI_DEFAULT_QUALITY 11
#define BROTLI_DEFAULT_WINDOW 22
#define BROTLI_DEFAULT_MODE BROTLI_MODE_GENERIC
typedef enum BrotliEncoderOperation {
BROTLI_OPERATION_PROCESS = 0,
/* Request output stream to flush. Performed when input stream is depleted
and there is enough space in output stream. */
BROTLI_OPERATION_FLUSH = 1,
/* Request output stream to finish. Performed when input stream is depleted
and there is enough space in output stream. */
BROTLI_OPERATION_FINISH = 2
} BrotliEncoderOperation;
typedef enum BrotliEncoderParameter {
BROTLI_PARAM_MODE = 0,
/* Controls the compression-speed vs compression-density tradeoffs. The higher
the quality, the slower the compression. Range is 0 to 11. */
BROTLI_PARAM_QUALITY = 1,
/* Base 2 logarithm of the sliding window size. Range is 10 to 24. */
BROTLI_PARAM_LGWIN = 2,
/* Base 2 logarithm of the maximum input block size. Range is 16 to 24.
If set to 0, the value will be set based on the quality. */
BROTLI_PARAM_LGBLOCK = 3
} BrotliEncoderParameter;
/* A state can not be reused for multiple brotli streams. */
typedef struct BrotliEncoderStateStruct BrotliEncoderState;
BROTLI_BOOL BrotliEncoderSetParameter(
BrotliEncoderState* state, BrotliEncoderParameter p, uint32_t value);
/* Creates the instance of BrotliEncoderState and initializes it.
|alloc_func| and |free_func| MUST be both zero or both non-zero. In the case
they are both zero, default memory allocators are used. |opaque| is passed to
|alloc_func| and |free_func| when they are called. */
BrotliEncoderState* BrotliEncoderCreateInstance(brotli_alloc_func alloc_func,
brotli_free_func free_func,
void* opaque);
/* Deinitializes and frees BrotliEncoderState instance. */
void BrotliEncoderDestroyInstance(BrotliEncoderState* state);
/* The maximum input size that can be processed at once. */
size_t BrotliEncoderInputBlockSize(BrotliEncoderState* state);
/* Encodes the data in |input_buffer| as a meta-block and writes it to
|encoded_buffer| (|*encoded_size should| be set to the size of
|encoded_buffer|) and sets |*encoded_size| to the number of bytes that
was written. The |input_size| must not be greater than input_block_size().
Returns false if there was an error and true otherwise. */
BROTLI_BOOL BrotliEncoderWriteMetaBlock(
BrotliEncoderState* state, const size_t input_size,
const uint8_t* input_buffer, const BROTLI_BOOL is_last,
size_t* encoded_size, uint8_t* encoded_buffer);
/* Writes a metadata meta-block containing the given input to encoded_buffer.
|*encoded_size| should be set to the size of the encoded_buffer.
Sets |*encoded_size| to the number of bytes that was written.
Note that the given input data will not be part of the sliding window and
thus no backward references can be made to this data from subsequent
metablocks. |input_size| must not be greater than 2^24 and provided
|*encoded_size| must not be less than |input_size| + 6.
Returns false if there was an error and true otherwise. */
BROTLI_BOOL BrotliEncoderWriteMetadata(
BrotliEncoderState* state, const size_t input_size,
const uint8_t* input_buffer, const BROTLI_BOOL is_last,
size_t* encoded_size, uint8_t* encoded_buffer);
/* Writes a zero-length meta-block with end-of-input bit set to the
internal output buffer and copies the output buffer to |encoded_buffer|
(|*encoded_size| should be set to the size of |encoded_buffer|) and sets
|*encoded_size| to the number of bytes written.
Returns false if there was an error and true otherwise. */
BROTLI_BOOL BrotliEncoderFinishStream(
BrotliEncoderState* state, size_t* encoded_size, uint8_t* encoded_buffer);
/* Copies the given input data to the internal ring buffer of the compressor.
No processing of the data occurs at this time and this function can be
called multiple times before calling WriteBrotliData() to process the
accumulated input. At most input_block_size() bytes of input data can be
copied to the ring buffer, otherwise the next WriteBrotliData() will fail.
*/
void BrotliEncoderCopyInputToRingBuffer(BrotliEncoderState* state,
const size_t input_size,
const uint8_t* input_buffer);
/* Processes the accumulated input data and sets |*out_size| to the length of
the new output meta-block, or to zero if no new output meta-block has been
created (in this case the processed input data is buffered internally).
If |*out_size| is positive, |*output| points to the start of the output
data. If |is_last| or |force_flush| is 1, an output meta-block is always
created. However, until |is_last| is 1 encoder may retain up to 7 bits
of the last byte of output. To force encoder to dump the remaining bits
use WriteMetadata() to append an empty meta-data block.
Returns false if the size of the input data is larger than
input_block_size(). */
BROTLI_BOOL BrotliEncoderWriteData(
BrotliEncoderState* state, const BROTLI_BOOL is_last,
const BROTLI_BOOL force_flush, size_t* out_size, uint8_t** output);
/* Fills the new state with a dictionary for LZ77, warming up the ringbuffer,
e.g. for custom static dictionaries for data formats.
Not to be confused with the built-in transformable dictionary of Brotli.
To decode, use BrotliSetCustomDictionary() of the decoder with the same
dictionary. */
void BrotliEncoderSetCustomDictionary(BrotliEncoderState* state, size_t size,
const uint8_t* dict);
/* Returns buffer size that is large enough to contain BrotliEncoderCompress
output for any input.
Returns 0 if result does not fit size_t. */
size_t BrotliEncoderMaxCompressedSize(size_t input_size);
/* Compresses the data in |input_buffer| into |encoded_buffer|, and sets
|*encoded_size| to the compressed length.
BROTLI_DEFAULT_QUALITY, BROTLI_DEFAULT_WINDOW and BROTLI_DEFAULT_MODE should
be used as |quality|, |lgwin| and |mode| if there are no specific
requirements to encoder speed and compression ratio.
If compression fails, |*encoded_size| is set to 0.
If BrotliEncoderMaxCompressedSize(|input_size|) is not zero, then
|*encoded_size| is never set to the bigger value.
Returns false if there was an error and true otherwise. */
BROTLI_BOOL BrotliEncoderCompress(
int quality, int lgwin, BrotliEncoderMode mode, size_t input_size,
const uint8_t* input_buffer, size_t* encoded_size, uint8_t* encoded_buffer);
/* Progressively compress input stream and push produced bytes to output stream.
Internally workflow consists of 3 tasks:
* (optional) copy input data to internal buffer
* actually compress data and (optionally) store it to internal buffer
* (optional) copy compressed bytes from internal buffer to output stream
Whenever all 3 tasks can't move forward anymore, or error occurs, this
method returns.
|available_in| and |next_in| represent input stream; when X bytes of input
are consumed, X is subtracted from |available_in| and added to |next_in|.
|available_out| and |next_out| represent output stream; when Y bytes are
pushed to output, Y is subtracted from |available_out| and added to
|next_out|. |total_out|, if it is not a null-pointer, is assigned to the
total amount of bytes pushed by the instance of encoder to output.
|op| is used to perform flush or finish the stream.
Flushing the stream means forcing encoding of all input passed to encoder and
completing the current output block, so it could be fully decoded by stream
decoder. To perform flush |op| must be set to BROTLI_OPERATION_FLUSH. Under
some circumstances (e.g. lack of output stream capacity) this operation would
require several calls to BrotliEncoderCompressStream. The method must be
called again until both input stream is depleted and encoder has no more
output (see BrotliEncoderHasMoreOutput) after the method is called.
Finishing the stream means encoding of all input passed to encoder and
adding specific "final" marks, so stream decoder could determine that stream
is complete. To perform finish |op| must be set to BROTLI_OPERATION_FINISH.
Under some circumstances (e.g. lack of output stream capacity) this operation
would require several calls to BrotliEncoderCompressStream. The method must
be called again until both input stream is depleted and encoder has no more
output (see BrotliEncoderHasMoreOutput) after the method is called.
WARNING: when flushing and finishing, |op| should not change until operation
is complete; input stream should not be refilled as well.
Returns false if there was an error and true otherwise.
*/
BROTLI_BOOL BrotliEncoderCompressStream(
BrotliEncoderState* s, BrotliEncoderOperation op, size_t* available_in,
const uint8_t** next_in, size_t* available_out, uint8_t** next_out,
size_t* total_out);
/* Check if encoder is in "finished" state, i.e. no more input is acceptable and
no more output will be produced.
Works only with BrotliEncoderCompressStream workflow.
Returns 1 if stream is finished and 0 otherwise. */
BROTLI_BOOL BrotliEncoderIsFinished(BrotliEncoderState* s);
/* Check if encoder has more output bytes in internal buffer.
Works only with BrotliEncoderCompressStream workflow.
Returns 1 if has more output (in internal buffer) and 0 otherwise. */
BROTLI_BOOL BrotliEncoderHasMoreOutput(BrotliEncoderState* s);
#if defined(__cplusplus) || defined(c_plusplus)
} /* extern "C" */
#endif
#endif /* BROTLI_ENC_ENCODE_H_ */

View File

@ -1,27 +0,0 @@
/* Copyright 2013 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* API for parallel Brotli compression
Note that this is only a proof of concept currently and not part of the
final API yet. */
#ifndef BROTLI_ENC_ENCODE_PARALLEL_H_
#define BROTLI_ENC_ENCODE_PARALLEL_H_
#include "../common/types.h"
#include "./compressor.h"
namespace brotli {
int BrotliCompressBufferParallel(BrotliParams params,
size_t input_size,
const uint8_t* input_buffer,
size_t* encoded_size,
uint8_t* encoded_buffer);
} /* namespace brotli */
#endif /* BROTLI_ENC_ENCODE_PARALLEL_H_ */

View File

@ -0,0 +1,32 @@
/* Copyright 2017 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
#include "./encoder_dict.h"
#include "../common/dictionary.h"
#include "../common/transform.h"
#include "./dictionary_hash.h"
#include "./hash.h"
#if defined(__cplusplus) || defined(c_plusplus)
extern "C" {
#endif
void BrotliInitEncoderDictionary(BrotliEncoderDictionary* dict) {
dict->words = BrotliGetDictionary();
dict->hash_table = kStaticDictionaryHash;
dict->buckets = kStaticDictionaryBuckets;
dict->dict_words = kStaticDictionaryWords;
dict->cutoffTransformsCount = kCutoffTransformsCount;
dict->cutoffTransforms = kCutoffTransforms;
}
#if defined(__cplusplus) || defined(c_plusplus)
} /* extern "C" */
#endif

View File

@ -0,0 +1,41 @@
/* Copyright 2017 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
#ifndef BROTLI_ENC_ENCODER_DICT_H_
#define BROTLI_ENC_ENCODER_DICT_H_
#include "../common/dictionary.h"
#include "../common/platform.h"
#include <brotli/types.h>
#include "./static_dict_lut.h"
#if defined(__cplusplus) || defined(c_plusplus)
extern "C" {
#endif
/* Dictionary data (words and transforms) for 1 possible context */
typedef struct BrotliEncoderDictionary {
const BrotliDictionary* words;
/* cut off for fast encoder */
uint32_t cutoffTransformsCount;
uint64_t cutoffTransforms;
/* from dictionary_hash.h, for fast encoder */
const uint16_t* hash_table;
/* from static_dict_lut.h, for slow encoder */
const uint16_t* buckets;
const DictWord* dict_words;
} BrotliEncoderDictionary;
BROTLI_INTERNAL void BrotliInitEncoderDictionary(BrotliEncoderDictionary* dict);
#if defined(__cplusplus) || defined(c_plusplus)
} /* extern "C" */
#endif
#endif /* BROTLI_ENC_ENCODER_DICT_H_ */

View File

@ -11,8 +11,8 @@
#include <string.h> /* memset */
#include "../common/constants.h"
#include "../common/types.h"
#include "./port.h"
#include "../common/platform.h"
#include <brotli/types.h>
#if defined(__cplusplus) || defined(c_plusplus)
extern "C" {
@ -23,7 +23,7 @@ BROTLI_BOOL BrotliSetDepth(
int stack[16];
int level = 0;
int p = p0;
assert(max_depth <= 15);
BROTLI_DCHECK(max_depth <= 15);
stack[0] = -1;
while (BROTLI_TRUE) {
if (pool[p].index_left_ >= 0) {
@ -66,11 +66,11 @@ static BROTLI_INLINE BROTLI_BOOL SortHuffmanTree(
we are not planning to use this with extremely long blocks.
See http://en.wikipedia.org/wiki/Huffman_coding */
void BrotliCreateHuffmanTree(const uint32_t *data,
void BrotliCreateHuffmanTree(const uint32_t* data,
const size_t length,
const int tree_limit,
HuffmanTree* tree,
uint8_t *depth) {
uint8_t* depth) {
uint32_t count_limit;
HuffmanTree sentinel;
InitHuffmanTree(&sentinel, BROTLI_UINT32_MAX, -1, -1);
@ -165,7 +165,7 @@ static void BrotliWriteHuffmanTreeRepetitions(
size_t* tree_size,
uint8_t* tree,
uint8_t* extra_bits_data) {
assert(repetitions > 0);
BROTLI_DCHECK(repetitions > 0);
if (previous_value != value) {
tree[*tree_size] = value;
extra_bits_data[*tree_size] = 0;
@ -246,7 +246,7 @@ void BrotliOptimizeHuffmanCountsForRle(size_t length, uint32_t* counts,
size_t limit;
size_t sum;
const size_t streak_limit = 1240;
/* Let's make the Huffman code more compatible with rle encoding. */
/* Let's make the Huffman code more compatible with RLE encoding. */
size_t i;
for (i = 0; i < length; i++) {
if (counts[i]) {
@ -293,10 +293,10 @@ void BrotliOptimizeHuffmanCountsForRle(size_t length, uint32_t* counts,
}
}
/* 2) Let's mark all population counts that already can be encoded
with an rle code. */
with an RLE code. */
memset(good_for_rle, 0, length);
{
/* Let's not spoil any of the existing good rle codes.
/* Let's not spoil any of the existing good RLE codes.
Mark any seq of 0's that is longer as 5 as a good_for_rle.
Mark any seq of non-0's that is longer as 7 as a good_for_rle. */
uint32_t symbol = counts[0];
@ -319,7 +319,7 @@ void BrotliOptimizeHuffmanCountsForRle(size_t length, uint32_t* counts,
}
}
}
/* 3) Let's replace those population counts that lead to more rle codes.
/* 3) Let's replace those population counts that lead to more RLE codes.
Math here is in 24.8 fixed point representation. */
stride = 0;
limit = 256 * (counts[0] + counts[1] + counts[2]) / 3 + 420;
@ -371,8 +371,8 @@ void BrotliOptimizeHuffmanCountsForRle(size_t length, uint32_t* counts,
}
static void DecideOverRleUse(const uint8_t* depth, const size_t length,
BROTLI_BOOL *use_rle_for_non_zero,
BROTLI_BOOL *use_rle_for_zero) {
BROTLI_BOOL* use_rle_for_non_zero,
BROTLI_BOOL* use_rle_for_zero) {
size_t total_reps_zero = 0;
size_t total_reps_non_zero = 0;
size_t count_reps_zero = 1;
@ -420,15 +420,15 @@ void BrotliWriteHuffmanTree(const uint8_t* depth,
}
}
/* First gather statistics on if it is a good idea to do rle. */
/* First gather statistics on if it is a good idea to do RLE. */
if (length > 50) {
/* Find rle coding for longer codes.
Shorter codes seem not to benefit from rle. */
/* Find RLE coding for longer codes.
Shorter codes seem not to benefit from RLE. */
DecideOverRleUse(depth, new_length,
&use_rle_for_non_zero, &use_rle_for_zero);
}
/* Actual rle coding. */
/* Actual RLE coding. */
for (i = 0; i < new_length;) {
const uint8_t value = depth[i];
size_t reps = 1;
@ -454,26 +454,26 @@ void BrotliWriteHuffmanTree(const uint8_t* depth,
static uint16_t BrotliReverseBits(size_t num_bits, uint16_t bits) {
static const size_t kLut[16] = { /* Pre-reversed 4-bit values. */
0x0, 0x8, 0x4, 0xc, 0x2, 0xa, 0x6, 0xe,
0x1, 0x9, 0x5, 0xd, 0x3, 0xb, 0x7, 0xf
0x00, 0x08, 0x04, 0x0C, 0x02, 0x0A, 0x06, 0x0E,
0x01, 0x09, 0x05, 0x0D, 0x03, 0x0B, 0x07, 0x0F
};
size_t retval = kLut[bits & 0xf];
size_t retval = kLut[bits & 0x0F];
size_t i;
for (i = 4; i < num_bits; i += 4) {
retval <<= 4;
bits = (uint16_t)(bits >> 4);
retval |= kLut[bits & 0xf];
retval |= kLut[bits & 0x0F];
}
retval >>= ((0 - num_bits) & 0x3);
retval >>= ((0 - num_bits) & 0x03);
return (uint16_t)retval;
}
/* 0..15 are values for bits */
#define MAX_HUFFMAN_BITS 16
void BrotliConvertBitDepthsToSymbols(const uint8_t *depth,
void BrotliConvertBitDepthsToSymbols(const uint8_t* depth,
size_t len,
uint16_t *bits) {
uint16_t* bits) {
/* In Brotli, all bit depths are [1..15]
0 bit depth means that the symbol does not exist. */
uint16_t bl_count[MAX_HUFFMAN_BITS] = { 0 };

View File

@ -9,8 +9,8 @@
#ifndef BROTLI_ENC_ENTROPY_ENCODE_H_
#define BROTLI_ENC_ENTROPY_ENCODE_H_
#include "../common/types.h"
#include "./port.h"
#include "../common/platform.h"
#include <brotli/types.h>
#if defined(__cplusplus) || defined(c_plusplus)
extern "C" {
@ -30,7 +30,7 @@ static BROTLI_INLINE void InitHuffmanTree(HuffmanTree* self, uint32_t count,
self->index_right_or_value_ = right;
}
/* Returns 1 is assignment of depths succeded, otherwise 0. */
/* Returns 1 is assignment of depths succeeded, otherwise 0. */
BROTLI_INTERNAL BROTLI_BOOL BrotliSetDepth(
int p, HuffmanTree* pool, uint8_t* depth, int max_depth);
@ -46,14 +46,14 @@ BROTLI_INTERNAL BROTLI_BOOL BrotliSetDepth(
be at least 2 * length + 1 long.
See http://en.wikipedia.org/wiki/Huffman_coding */
BROTLI_INTERNAL void BrotliCreateHuffmanTree(const uint32_t *data,
BROTLI_INTERNAL void BrotliCreateHuffmanTree(const uint32_t* data,
const size_t length,
const int tree_limit,
HuffmanTree* tree,
uint8_t *depth);
uint8_t* depth);
/* Change the population counts in a way that the consequent
Huffman tree compression, especially its rle-part will be more
Huffman tree compression, especially its RLE-part will be more
likely to compress this data more efficiently.
length contains the size of the histogram.
@ -62,7 +62,7 @@ BROTLI_INTERNAL void BrotliCreateHuffmanTree(const uint32_t *data,
BROTLI_INTERNAL void BrotliOptimizeHuffmanCountsForRle(
size_t length, uint32_t* counts, uint8_t* good_for_rle);
/* Write a Huffman tree from bit depths into the bitstream representation
/* Write a Huffman tree from bit depths into the bit-stream representation
of a Huffman tree. The generated Huffman tree is to be compressed once
more using a Huffman tree */
BROTLI_INTERNAL void BrotliWriteHuffmanTree(const uint8_t* depth,
@ -72,9 +72,9 @@ BROTLI_INTERNAL void BrotliWriteHuffmanTree(const uint8_t* depth,
uint8_t* extra_bits_data);
/* Get the actual bit values for a tree of bit depths. */
BROTLI_INTERNAL void BrotliConvertBitDepthsToSymbols(const uint8_t *depth,
BROTLI_INTERNAL void BrotliConvertBitDepthsToSymbols(const uint8_t* depth,
size_t len,
uint16_t *bits);
uint16_t* bits);
/* Input size optimized Shell sort. */
typedef BROTLI_BOOL (*HuffmanTreeComparator)(

View File

@ -10,8 +10,8 @@
#define BROTLI_ENC_ENTROPY_ENCODE_STATIC_H_
#include "../common/constants.h"
#include "../common/port.h"
#include "../common/types.h"
#include "../common/platform.h"
#include <brotli/types.h>
#include "./write_bits.h"
#if defined(__cplusplus) || defined(c_plusplus)
@ -83,7 +83,7 @@ static const uint32_t kCodeLengthBits[18] = {
static BROTLI_INLINE void StoreStaticCodeLengthCode(
size_t* storage_ix, uint8_t* storage) {
BrotliWriteBits(
40, MAKE_UINT64_T(0x0000ffU, 0x55555554U), storage_ix, storage);
40, BROTLI_MAKE_UINT64_T(0x0000FFu, 0x55555554u), storage_ix, storage);
}
static const uint64_t kZeroRepsBits[BROTLI_NUM_COMMAND_SYMBOLS] = {
@ -516,7 +516,7 @@ static const uint16_t kStaticCommandCodeBits[BROTLI_NUM_COMMAND_SYMBOLS] = {
static BROTLI_INLINE void StoreStaticCommandHuffmanTree(
size_t* storage_ix, uint8_t* storage) {
BrotliWriteBits(
56, MAKE_UINT64_T(0x926244U, 0x16307003U), storage_ix, storage);
56, BROTLI_MAKE_UINT64_T(0x926244U, 0x16307003U), storage_ix, storage);
BrotliWriteBits(3, 0x00000000U, storage_ix, storage);
}
@ -529,7 +529,7 @@ static const uint16_t kStaticDistanceCodeBits[64] = {
static BROTLI_INLINE void StoreStaticDistanceHuffmanTree(
size_t* storage_ix, uint8_t* storage) {
BrotliWriteBits(28, 0x0369dc03U, storage_ix, storage);
BrotliWriteBits(28, 0x0369DC03u, storage_ix, storage);
}
#if defined(__cplusplus) || defined(c_plusplus)

View File

@ -11,15 +11,17 @@
#include <math.h>
#include "../common/types.h"
#include "../common/port.h"
#include "../common/platform.h"
#include <brotli/types.h>
#if defined(__cplusplus) || defined(c_plusplus)
extern "C" {
#endif
static BROTLI_INLINE uint32_t Log2FloorNonZero(size_t n) {
#ifdef __GNUC__
/* TODO: generalize and move to platform.h */
#if BROTLI_GNUC_HAS_BUILTIN(__builtin_clz, 3, 4, 0) || \
BROTLI_INTEL_VERSION_CHECK(16, 0, 0)
return 31u ^ (uint32_t)__builtin_clz((uint32_t)n);
#else
uint32_t result = 0;

View File

@ -9,29 +9,29 @@
#ifndef BROTLI_ENC_FIND_MATCH_LENGTH_H_
#define BROTLI_ENC_FIND_MATCH_LENGTH_H_
#include "../common/types.h"
#include "./port.h"
#include "../common/platform.h"
#include <brotli/types.h>
#if defined(__cplusplus) || defined(c_plusplus)
extern "C" {
#endif
/* Separate implementation for little-endian 64-bit targets, for speed. */
#if defined(__GNUC__) && defined(_LP64) && defined(IS_LITTLE_ENDIAN)
#if defined(__GNUC__) && defined(_LP64) && defined(BROTLI_LITTLE_ENDIAN)
static BROTLI_INLINE size_t FindMatchLengthWithLimit(const uint8_t* s1,
const uint8_t* s2,
size_t limit) {
size_t matched = 0;
size_t limit2 = (limit >> 3) + 1; /* + 1 is for pre-decrement in while */
while (PREDICT_TRUE(--limit2)) {
if (PREDICT_FALSE(BROTLI_UNALIGNED_LOAD64(s2) ==
BROTLI_UNALIGNED_LOAD64(s1 + matched))) {
while (BROTLI_PREDICT_TRUE(--limit2)) {
if (BROTLI_PREDICT_FALSE(BROTLI_UNALIGNED_LOAD64LE(s2) ==
BROTLI_UNALIGNED_LOAD64LE(s1 + matched))) {
s2 += 8;
matched += 8;
} else {
uint64_t x =
BROTLI_UNALIGNED_LOAD64(s2) ^ BROTLI_UNALIGNED_LOAD64(s1 + matched);
uint64_t x = BROTLI_UNALIGNED_LOAD64LE(s2) ^
BROTLI_UNALIGNED_LOAD64LE(s1 + matched);
size_t matching_bits = (size_t)__builtin_ctzll(x);
matched += matching_bits >> 3;
return matched;
@ -39,7 +39,7 @@ static BROTLI_INLINE size_t FindMatchLengthWithLimit(const uint8_t* s1,
}
limit = (limit & 7) + 1; /* + 1 is for pre-decrement in while */
while (--limit) {
if (PREDICT_TRUE(s1[matched] == *s2)) {
if (BROTLI_PREDICT_TRUE(s1[matched] == *s2)) {
++s2;
++matched;
} else {
@ -60,8 +60,8 @@ static BROTLI_INLINE size_t FindMatchLengthWithLimit(const uint8_t* s1,
the first non-matching bit and use that to calculate the total
length of the match. */
while (s2_ptr <= s2_limit - 4 &&
BROTLI_UNALIGNED_LOAD32(s2_ptr) ==
BROTLI_UNALIGNED_LOAD32(s1 + matched)) {
BrotliUnalignedRead32(s2_ptr) ==
BrotliUnalignedRead32(s1 + matched)) {
s2_ptr += 4;
matched += 4;
}

View File

@ -14,12 +14,12 @@
#include "../common/constants.h"
#include "../common/dictionary.h"
#include "../common/types.h"
#include "./dictionary_hash.h"
#include "../common/platform.h"
#include <brotli/types.h>
#include "./encoder_dict.h"
#include "./fast_log.h"
#include "./find_match_length.h"
#include "./memory.h"
#include "./port.h"
#include "./quality.h"
#include "./static_dict.h"
@ -27,51 +27,91 @@
extern "C" {
#endif
#define MAX_TREE_SEARCH_DEPTH 64
#define MAX_TREE_COMP_LENGTH 128
/* Pointer to hasher data.
*
* Excluding initialization and destruction, hasher can be passed as
* HasherHandle by value.
*
* Typically hasher data consists of 3 sections:
* * HasherCommon structure
* * private structured hasher data, depending on hasher type
* * private dynamic hasher data, depending on hasher type and parameters
*
* Using "define" instead of "typedef", because on MSVC __restrict does not work
* on typedef pointer types. */
#define HasherHandle uint8_t*
typedef struct {
BrotliHasherParams params;
/* False if hasher needs to be "prepared" before use. */
BROTLI_BOOL is_prepared_;
size_t dict_num_lookups;
size_t dict_num_matches;
} HasherCommon;
static BROTLI_INLINE HasherCommon* GetHasherCommon(HasherHandle handle) {
return (HasherCommon*)handle;
}
#define score_t size_t
static const uint32_t kDistanceCacheIndex[] = {
0, 1, 2, 3, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1,
};
static const int kDistanceCacheOffset[] = {
0, 0, 0, 0, -1, 1, -2, 2, -3, 3, -1, 1, -2, 2, -3, 3
};
static const uint32_t kCutoffTransformsCount = 10;
static const uint8_t kCutoffTransforms[] = {
0, 12, 27, 23, 42, 63, 56, 48, 59, 64
};
/* 0, 12, 27, 23, 42, 63, 56, 48, 59, 64 */
/* 0+0, 4+8, 8+19, 12+11, 16+26, 20+43, 24+32, 28+20, 32+27, 36+28 */
static const uint64_t kCutoffTransforms =
BROTLI_MAKE_UINT64_T(0x071B520A, 0xDA2D3200);
typedef struct HasherSearchResult {
size_t len;
size_t len_x_code; /* == len ^ len_code */
size_t distance;
score_t score;
int len_code_delta; /* == len_code - len */
} HasherSearchResult;
typedef struct DictionarySearchStatictics {
size_t num_lookups;
size_t num_matches;
} DictionarySearchStatictics;
/* kHashMul32 multiplier has these properties:
* The multiplier must be odd. Otherwise we may lose the highest bit.
* No long streaks of 1s or 0s.
* No long streaks of ones or zeros.
* There is no effort to ensure that it is a prime, the oddity is enough
for this use.
* The number has been tuned heuristically against compression benchmarks. */
static const uint32_t kHashMul32 = 0x1e35a7bd;
static const uint32_t kHashMul32 = 0x1E35A7BD;
static const uint64_t kHashMul64 = BROTLI_MAKE_UINT64_T(0x1E35A7BD, 0x1E35A7BD);
static const uint64_t kHashMul64Long =
BROTLI_MAKE_UINT64_T(0x1FE35A7Bu, 0xD3579BD3u);
static BROTLI_INLINE uint32_t Hash14(const uint8_t* data) {
uint32_t h = BROTLI_UNALIGNED_LOAD32(data) * kHashMul32;
uint32_t h = BROTLI_UNALIGNED_LOAD32LE(data) * kHashMul32;
/* The higher bits contain more mixture from the multiplication,
so we take our results from there. */
return h >> (32 - 14);
}
#define BROTLI_LITERAL_BYTE_SCORE 540
#define BROTLI_DISTANCE_BIT_PENALTY 120
static BROTLI_INLINE void PrepareDistanceCache(
int* BROTLI_RESTRICT distance_cache, const int num_distances) {
if (num_distances > 4) {
int last_distance = distance_cache[0];
distance_cache[4] = last_distance - 1;
distance_cache[5] = last_distance + 1;
distance_cache[6] = last_distance - 2;
distance_cache[7] = last_distance + 2;
distance_cache[8] = last_distance - 3;
distance_cache[9] = last_distance + 3;
if (num_distances > 10) {
int next_last_distance = distance_cache[1];
distance_cache[10] = next_last_distance - 1;
distance_cache[11] = next_last_distance + 1;
distance_cache[12] = next_last_distance - 2;
distance_cache[13] = next_last_distance + 2;
distance_cache[14] = next_last_distance - 3;
distance_cache[15] = next_last_distance + 3;
}
}
}
#define BROTLI_LITERAL_BYTE_SCORE 135
#define BROTLI_DISTANCE_BIT_PENALTY 30
/* Score must be positive after applying maximal penalty. */
#define BROTLI_SCORE_BASE (BROTLI_DISTANCE_BIT_PENALTY * 8 * sizeof(size_t))
@ -97,97 +137,83 @@ static BROTLI_INLINE score_t BackwardReferenceScore(
BROTLI_DISTANCE_BIT_PENALTY * Log2FloorNonZero(backward_reference_offset);
}
static const score_t kDistanceShortCodeCost[BROTLI_NUM_DISTANCE_SHORT_CODES] = {
/* Repeat last */
BROTLI_SCORE_BASE + 60,
/* 2nd, 3rd, 4th last */
BROTLI_SCORE_BASE - 95,
BROTLI_SCORE_BASE - 117,
BROTLI_SCORE_BASE - 127,
/* Last with offset */
BROTLI_SCORE_BASE - 93,
BROTLI_SCORE_BASE - 93,
BROTLI_SCORE_BASE - 96,
BROTLI_SCORE_BASE - 96,
BROTLI_SCORE_BASE - 99,
BROTLI_SCORE_BASE - 99,
/* 2nd last with offset */
BROTLI_SCORE_BASE - 105,
BROTLI_SCORE_BASE - 105,
BROTLI_SCORE_BASE - 115,
BROTLI_SCORE_BASE - 115,
BROTLI_SCORE_BASE - 125,
BROTLI_SCORE_BASE - 125
};
static BROTLI_INLINE score_t BackwardReferenceScoreUsingLastDistance(
size_t copy_length, size_t distance_short_code) {
size_t copy_length) {
return BROTLI_LITERAL_BYTE_SCORE * (score_t)copy_length +
kDistanceShortCodeCost[distance_short_code];
BROTLI_SCORE_BASE + 15;
}
static BROTLI_INLINE void DictionarySearchStaticticsReset(
DictionarySearchStatictics* self) {
self->num_lookups = 0;
self->num_matches = 0;
static BROTLI_INLINE score_t BackwardReferencePenaltyUsingLastDistance(
size_t distance_short_code) {
return (score_t)39 + ((0x1CA10 >> (distance_short_code & 0xE)) & 0xE);
}
static BROTLI_INLINE BROTLI_BOOL TestStaticDictionaryItem(
size_t item, const uint8_t* data, size_t max_length, size_t max_backward,
const BrotliEncoderDictionary* dictionary, size_t item, const uint8_t* data,
size_t max_length, size_t max_backward, size_t max_distance,
HasherSearchResult* out) {
size_t len;
size_t dist;
size_t word_idx;
size_t offset;
size_t matchlen;
size_t backward;
score_t score;
len = item & 31;
dist = item >> 5;
offset = kBrotliDictionaryOffsetsByLength[len] + len * dist;
len = item & 0x1F;
word_idx = item >> 5;
offset = dictionary->words->offsets_by_length[len] + len * word_idx;
if (len > max_length) {
return BROTLI_FALSE;
}
matchlen = FindMatchLengthWithLimit(data, &kBrotliDictionary[offset], len);
if (matchlen + kCutoffTransformsCount <= len || matchlen == 0) {
matchlen =
FindMatchLengthWithLimit(data, &dictionary->words->data[offset], len);
if (matchlen + dictionary->cutoffTransformsCount <= len || matchlen == 0) {
return BROTLI_FALSE;
}
{
size_t transform_id = kCutoffTransforms[len - matchlen];
backward = max_backward + dist + 1 +
(transform_id << kBrotliDictionarySizeBitsByLength[len]);
size_t cut = len - matchlen;
size_t transform_id = (cut << 2) +
(size_t)((dictionary->cutoffTransforms >> (cut * 6)) & 0x3F);
backward = max_backward + 1 + word_idx +
(transform_id << dictionary->words->size_bits_by_length[len]);
}
if (backward > max_distance) {
return BROTLI_FALSE;
}
score = BackwardReferenceScore(matchlen, backward);
if (score < out->score) {
return BROTLI_FALSE;
}
out->len = matchlen;
out->len_x_code = len ^ matchlen;
out->len_code_delta = (int)len - (int)matchlen;
out->distance = backward;
out->score = score;
return BROTLI_TRUE;
}
static BROTLI_INLINE BROTLI_BOOL SearchInStaticDictionary(
DictionarySearchStatictics* self, const uint8_t* data, size_t max_length,
size_t max_backward, HasherSearchResult* out, BROTLI_BOOL shallow) {
static BROTLI_INLINE void SearchInStaticDictionary(
const BrotliEncoderDictionary* dictionary,
HasherHandle handle, const uint8_t* data, size_t max_length,
size_t max_backward, size_t max_distance,
HasherSearchResult* out, BROTLI_BOOL shallow) {
size_t key;
size_t i;
BROTLI_BOOL is_match_found = BROTLI_FALSE;
if (self->num_matches < (self->num_lookups >> 7)) {
return BROTLI_FALSE;
HasherCommon* self = GetHasherCommon(handle);
if (self->dict_num_matches < (self->dict_num_lookups >> 7)) {
return;
}
key = Hash14(data) << 1;
for (i = 0; i < (shallow ? 1 : 2); ++i, ++key) {
size_t item = kStaticDictionaryHash[key];
self->num_lookups++;
if (item != 0 &&
TestStaticDictionaryItem(item, data, max_length, max_backward, out)) {
self->num_matches++;
is_match_found = BROTLI_TRUE;
for (i = 0; i < (shallow ? 1u : 2u); ++i, ++key) {
size_t item = dictionary->hash_table[key];
self->dict_num_lookups++;
if (item != 0) {
BROTLI_BOOL item_matches = TestStaticDictionaryItem(
dictionary, item, data, max_length, max_backward, max_distance, out);
if (item_matches) {
self->dict_num_matches++;
}
}
}
return is_match_found;
}
typedef struct BackwardMatch {
@ -221,320 +247,26 @@ static BROTLI_INLINE size_t BackwardMatchLengthCode(const BackwardMatch* self) {
#define CAT(a, b) a ## b
#define FN(X) EXPAND_CAT(X, HASHER())
#define MAX_NUM_MATCHES_H10 (64 + MAX_TREE_SEARCH_DEPTH)
#define HASHER() H10
#define HashToBinaryTree HASHER()
#define BUCKET_BITS 17
#define BUCKET_SIZE (1 << BUCKET_BITS)
static size_t FN(HashTypeLength)(void) { return 4; }
static size_t FN(StoreLookahead)(void) { return MAX_TREE_COMP_LENGTH; }
static uint32_t FN(HashBytes)(const uint8_t *data) {
uint32_t h = BROTLI_UNALIGNED_LOAD32(data) * kHashMul32;
/* The higher bits contain more mixture from the multiplication,
so we take our results from there. */
return h >> (32 - BUCKET_BITS);
}
/* A (forgetful) hash table where each hash bucket contains a binary tree of
sequences whose first 4 bytes share the same hash code.
Each sequence is MAX_TREE_COMP_LENGTH long and is identified by its starting
position in the input data. The binary tree is sorted by the lexicographic
order of the sequences, and it is also a max-heap with respect to the
starting positions. */
typedef struct HashToBinaryTree {
/* The window size minus 1 */
size_t window_mask_;
/* Hash table that maps the 4-byte hashes of the sequence to the last
position where this hash was found, which is the root of the binary
tree of sequences that share this hash bucket. */
uint32_t buckets_[BUCKET_SIZE];
/* The union of the binary trees of each hash bucket. The root of the tree
corresponding to a hash is a sequence starting at buckets_[hash] and
the left and right children of a sequence starting at pos are
forest_[2 * pos] and forest_[2 * pos + 1]. */
uint32_t* forest_;
/* A position used to mark a non-existent sequence, i.e. a tree is empty if
its root is at invalid_pos_ and a node is a leaf if both its children
are at invalid_pos_. */
uint32_t invalid_pos_;
size_t forest_size_;
BROTLI_BOOL is_dirty_;
} HashToBinaryTree;
static void FN(Reset)(HashToBinaryTree* self) {
self->is_dirty_ = BROTLI_TRUE;
}
static void FN(Initialize)(HashToBinaryTree* self) {
self->forest_ = NULL;
self->forest_size_ = 0;
FN(Reset)(self);
}
static void FN(Cleanup)(MemoryManager* m, HashToBinaryTree* self) {
BROTLI_FREE(m, self->forest_);
}
static void FN(Init)(
MemoryManager* m, HashToBinaryTree* self, const uint8_t* data,
const BrotliEncoderParams* params, size_t position, size_t bytes,
BROTLI_BOOL is_last) {
if (self->is_dirty_) {
uint32_t invalid_pos;
size_t num_nodes;
uint32_t i;
BROTLI_UNUSED(data);
self->window_mask_ = (1u << params->lgwin) - 1u;
invalid_pos = (uint32_t)(0 - self->window_mask_);
self->invalid_pos_ = invalid_pos;
for (i = 0; i < BUCKET_SIZE; i++) {
self->buckets_[i] = invalid_pos;
}
num_nodes = (position == 0 && is_last) ? bytes : self->window_mask_ + 1;
if (num_nodes > self->forest_size_) {
BROTLI_FREE(m, self->forest_);
self->forest_ = BROTLI_ALLOC(m, uint32_t, 2 * num_nodes);
if (BROTLI_IS_OOM(m)) return;
self->forest_size_ = num_nodes;
}
self->is_dirty_ = BROTLI_FALSE;
}
}
static BROTLI_INLINE size_t FN(LeftChildIndex)(HashToBinaryTree* self,
const size_t pos) {
return 2 * (pos & self->window_mask_);
}
static BROTLI_INLINE size_t FN(RightChildIndex)(HashToBinaryTree* self,
const size_t pos) {
return 2 * (pos & self->window_mask_) + 1;
}
/* Stores the hash of the next 4 bytes and in a single tree-traversal, the
hash bucket's binary tree is searched for matches and is re-rooted at the
current position.
If less than MAX_TREE_COMP_LENGTH data is available, the hash bucket of the
current position is searched for matches, but the state of the hash table
is not changed, since we can not know the final sorting order of the
current (incomplete) sequence.
This function must be called with increasing cur_ix positions. */
static BROTLI_INLINE BackwardMatch* FN(StoreAndFindMatches)(
HashToBinaryTree* self, const uint8_t* const BROTLI_RESTRICT data,
const size_t cur_ix, const size_t ring_buffer_mask, const size_t max_length,
const size_t max_backward, size_t* const BROTLI_RESTRICT best_len,
BackwardMatch* BROTLI_RESTRICT matches) {
const size_t cur_ix_masked = cur_ix & ring_buffer_mask;
const size_t max_comp_len =
BROTLI_MIN(size_t, max_length, MAX_TREE_COMP_LENGTH);
const BROTLI_BOOL should_reroot_tree =
TO_BROTLI_BOOL(max_length >= MAX_TREE_COMP_LENGTH);
const uint32_t key = FN(HashBytes)(&data[cur_ix_masked]);
size_t prev_ix = self->buckets_[key];
/* The forest index of the rightmost node of the left subtree of the new
root, updated as we traverse and reroot the tree of the hash bucket. */
size_t node_left = FN(LeftChildIndex)(self, cur_ix);
/* The forest index of the leftmost node of the right subtree of the new
root, updated as we traverse and reroot the tree of the hash bucket. */
size_t node_right = FN(RightChildIndex)(self, cur_ix);
/* The match length of the rightmost node of the left subtree of the new
root, updated as we traverse and reroot the tree of the hash bucket. */
size_t best_len_left = 0;
/* The match length of the leftmost node of the right subtree of the new
root, updated as we traverse and reroot the tree of the hash bucket. */
size_t best_len_right = 0;
size_t depth_remaining;
if (should_reroot_tree) {
self->buckets_[key] = (uint32_t)cur_ix;
}
for (depth_remaining = MAX_TREE_SEARCH_DEPTH; ; --depth_remaining) {
const size_t backward = cur_ix - prev_ix;
const size_t prev_ix_masked = prev_ix & ring_buffer_mask;
if (backward == 0 || backward > max_backward || depth_remaining == 0) {
if (should_reroot_tree) {
self->forest_[node_left] = self->invalid_pos_;
self->forest_[node_right] = self->invalid_pos_;
}
break;
}
{
const size_t cur_len = BROTLI_MIN(size_t, best_len_left, best_len_right);
size_t len;
assert(cur_len <= MAX_TREE_COMP_LENGTH);
len = cur_len +
FindMatchLengthWithLimit(&data[cur_ix_masked + cur_len],
&data[prev_ix_masked + cur_len],
max_length - cur_len);
assert(0 == memcmp(&data[cur_ix_masked], &data[prev_ix_masked], len));
if (matches && len > *best_len) {
*best_len = len;
InitBackwardMatch(matches++, backward, len);
}
if (len >= max_comp_len) {
if (should_reroot_tree) {
self->forest_[node_left] =
self->forest_[FN(LeftChildIndex)(self, prev_ix)];
self->forest_[node_right] =
self->forest_[FN(RightChildIndex)(self, prev_ix)];
}
break;
}
if (data[cur_ix_masked + len] > data[prev_ix_masked + len]) {
best_len_left = len;
if (should_reroot_tree) {
self->forest_[node_left] = (uint32_t)prev_ix;
}
node_left = FN(RightChildIndex)(self, prev_ix);
prev_ix = self->forest_[node_left];
} else {
best_len_right = len;
if (should_reroot_tree) {
self->forest_[node_right] = (uint32_t)prev_ix;
}
node_right = FN(LeftChildIndex)(self, prev_ix);
prev_ix = self->forest_[node_right];
}
}
}
return matches;
}
/* Finds all backward matches of &data[cur_ix & ring_buffer_mask] up to the
length of max_length and stores the position cur_ix in the hash table.
Sets *num_matches to the number of matches found, and stores the found
matches in matches[0] to matches[*num_matches - 1]. The matches will be
sorted by strictly increasing length and (non-strictly) increasing
distance. */
static BROTLI_INLINE size_t FN(FindAllMatches)(HashToBinaryTree* self,
const uint8_t* data, const size_t ring_buffer_mask, const size_t cur_ix,
const size_t max_length, const size_t max_backward,
const BrotliEncoderParams* params, BackwardMatch* matches) {
BackwardMatch* const orig_matches = matches;
const size_t cur_ix_masked = cur_ix & ring_buffer_mask;
size_t best_len = 1;
const size_t short_match_max_backward =
params->quality != HQ_ZOPFLIFICATION_QUALITY ? 16 : 64;
size_t stop = cur_ix - short_match_max_backward;
uint32_t dict_matches[BROTLI_MAX_STATIC_DICTIONARY_MATCH_LEN + 1];
size_t i;
if (cur_ix < short_match_max_backward) { stop = 0; }
for (i = cur_ix - 1; i > stop && best_len <= 2; --i) {
size_t prev_ix = i;
const size_t backward = cur_ix - prev_ix;
if (PREDICT_FALSE(backward > max_backward)) {
break;
}
prev_ix &= ring_buffer_mask;
if (data[cur_ix_masked] != data[prev_ix] ||
data[cur_ix_masked + 1] != data[prev_ix + 1]) {
continue;
}
{
const size_t len =
FindMatchLengthWithLimit(&data[prev_ix], &data[cur_ix_masked],
max_length);
if (len > best_len) {
best_len = len;
InitBackwardMatch(matches++, backward, len);
}
}
}
if (best_len < max_length) {
matches = FN(StoreAndFindMatches)(self, data, cur_ix, ring_buffer_mask,
max_length, max_backward, &best_len, matches);
}
for (i = 0; i <= BROTLI_MAX_STATIC_DICTIONARY_MATCH_LEN; ++i) {
dict_matches[i] = kInvalidMatch;
}
{
size_t minlen = BROTLI_MAX(size_t, 4, best_len + 1);
if (BrotliFindAllStaticDictionaryMatches(&data[cur_ix_masked], minlen,
max_length, &dict_matches[0])) {
size_t maxlen = BROTLI_MIN(
size_t, BROTLI_MAX_STATIC_DICTIONARY_MATCH_LEN, max_length);
size_t l;
for (l = minlen; l <= maxlen; ++l) {
uint32_t dict_id = dict_matches[l];
if (dict_id < kInvalidMatch) {
InitDictionaryBackwardMatch(matches++,
max_backward + (dict_id >> 5) + 1, l, dict_id & 31);
}
}
}
}
return (size_t)(matches - orig_matches);
}
/* Stores the hash of the next 4 bytes and re-roots the binary tree at the
current sequence, without returning any matches.
REQUIRES: ix + MAX_TREE_COMP_LENGTH <= end-of-current-block */
static BROTLI_INLINE void FN(Store)(HashToBinaryTree* self, const uint8_t *data,
const size_t mask, const size_t ix) {
/* Maximum distance is window size - 16, see section 9.1. of the spec. */
const size_t max_backward = self->window_mask_ - 15;
FN(StoreAndFindMatches)(self, data, ix, mask, MAX_TREE_COMP_LENGTH,
max_backward, NULL, NULL);
}
static BROTLI_INLINE void FN(StoreRange)(HashToBinaryTree* self,
const uint8_t *data, const size_t mask, const size_t ix_start,
const size_t ix_end) {
size_t i = ix_start + 63 <= ix_end ? ix_end - 63 : ix_start;
for (; i < ix_end; ++i) {
FN(Store)(self, data, mask, i);
}
}
static BROTLI_INLINE void FN(StitchToPreviousBlock)(HashToBinaryTree* self,
size_t num_bytes, size_t position, const uint8_t* ringbuffer,
size_t ringbuffer_mask) {
if (num_bytes >= FN(HashTypeLength)() - 1 &&
position >= MAX_TREE_COMP_LENGTH) {
/* Store the last `MAX_TREE_COMP_LENGTH - 1` positions in the hasher.
These could not be calculated before, since they require knowledge
of both the previous and the current block. */
const size_t i_start = position - MAX_TREE_COMP_LENGTH + 1;
const size_t i_end = BROTLI_MIN(size_t, position, i_start + num_bytes);
size_t i;
for (i = i_start; i < i_end; ++i) {
/* Maximum distance is window size - 16, see section 9.1. of the spec.
Furthermore, we have to make sure that we don't look further back
from the start of the next block than the window size, otherwise we
could access already overwritten areas of the ringbuffer. */
const size_t max_backward =
self->window_mask_ - BROTLI_MAX(size_t, 15, position - i);
/* We know that i + MAX_TREE_COMP_LENGTH <= position + num_bytes, i.e. the
end of the current block and that we have at least
MAX_TREE_COMP_LENGTH tail in the ringbuffer. */
FN(StoreAndFindMatches)(self, ringbuffer, i, ringbuffer_mask,
MAX_TREE_COMP_LENGTH, max_backward, NULL, NULL);
}
}
}
#undef BUCKET_SIZE
#define MAX_TREE_SEARCH_DEPTH 64
#define MAX_TREE_COMP_LENGTH 128
#include "./hash_to_binary_tree_inc.h" /* NOLINT(build/include) */
#undef MAX_TREE_SEARCH_DEPTH
#undef MAX_TREE_COMP_LENGTH
#undef BUCKET_BITS
#undef HASHER
/* MAX_NUM_MATCHES == 64 + MAX_TREE_SEARCH_DEPTH */
#define MAX_NUM_MATCHES_H10 128
/* For BUCKET_SWEEP == 1, enabling the dictionary lookup makes compression
a little faster (0.5% - 1%) and it compresses 0.15% better on small text
and html inputs. */
and HTML inputs. */
#define HASHER() H2
#define BUCKET_BITS 16
#define BUCKET_SWEEP 1
#define HASH_LEN 5
#define USE_DICTIONARY 1
#include "./hash_longest_match_quickly_inc.h" /* NOLINT(build/include) */
#undef BUCKET_SWEEP
@ -556,48 +288,17 @@ static BROTLI_INLINE void FN(StitchToPreviousBlock)(HashToBinaryTree* self,
#define USE_DICTIONARY 1
#include "./hash_longest_match_quickly_inc.h" /* NOLINT(build/include) */
#undef USE_DICTIONARY
#undef HASH_LEN
#undef BUCKET_SWEEP
#undef BUCKET_BITS
#undef HASHER
#define HASHER() H5
#define BUCKET_BITS 14
#define BLOCK_BITS 4
#define NUM_LAST_DISTANCES_TO_CHECK 4
#include "./hash_longest_match_inc.h" /* NOLINT(build/include) */
#undef BLOCK_BITS
#undef HASHER
#define HASHER() H6
#define BLOCK_BITS 5
#include "./hash_longest_match_inc.h" /* NOLINT(build/include) */
#undef NUM_LAST_DISTANCES_TO_CHECK
#undef BLOCK_BITS
#undef BUCKET_BITS
#undef HASHER
#define HASHER() H7
#define BUCKET_BITS 15
#define BLOCK_BITS 6
#define NUM_LAST_DISTANCES_TO_CHECK 10
#include "./hash_longest_match_inc.h" /* NOLINT(build/include) */
#undef BLOCK_BITS
#undef HASHER
#define HASHER() H8
#define BLOCK_BITS 7
#include "./hash_longest_match_inc.h" /* NOLINT(build/include) */
#undef NUM_LAST_DISTANCES_TO_CHECK
#undef BLOCK_BITS
#undef HASHER
#define HASHER() H9
#define BLOCK_BITS 8
#define NUM_LAST_DISTANCES_TO_CHECK 16
#include "./hash_longest_match_inc.h" /* NOLINT(build/include) */
#undef NUM_LAST_DISTANCES_TO_CHECK
#undef BLOCK_BITS
#undef BUCKET_BITS
#include "./hash_longest_match64_inc.h" /* NOLINT(build/include) */
#undef HASHER
#define BUCKET_BITS 15
@ -630,86 +331,165 @@ static BROTLI_INLINE void FN(StitchToPreviousBlock)(HashToBinaryTree* self,
#undef BUCKET_BITS
#define HASHER() H54
#define BUCKET_BITS 20
#define BUCKET_SWEEP 4
#define HASH_LEN 7
#define USE_DICTIONARY 0
#include "./hash_longest_match_quickly_inc.h" /* NOLINT(build/include) */
#undef USE_DICTIONARY
#undef HASH_LEN
#undef BUCKET_SWEEP
#undef BUCKET_BITS
#undef HASHER
/* fast large window hashers */
#define HASHER() HROLLING_FAST
#define CHUNKLEN 32
#define JUMP 4
#define NUMBUCKETS 16777216
#define MASK ((NUMBUCKETS * 64) - 1)
#include "./hash_rolling_inc.h" /* NOLINT(build/include) */
#undef JUMP
#undef HASHER
#define HASHER() HROLLING
#define JUMP 1
#include "./hash_rolling_inc.h" /* NOLINT(build/include) */
#undef MASK
#undef NUMBUCKETS
#undef JUMP
#undef CHUNKLEN
#undef HASHER
#define HASHER() H35
#define HASHER_A H3
#define HASHER_B HROLLING_FAST
#include "./hash_composite_inc.h" /* NOLINT(build/include) */
#undef HASHER_A
#undef HASHER_B
#undef HASHER
#define HASHER() H55
#define HASHER_A H54
#define HASHER_B HROLLING_FAST
#include "./hash_composite_inc.h" /* NOLINT(build/include) */
#undef HASHER_A
#undef HASHER_B
#undef HASHER
#define HASHER() H65
#define HASHER_A H6
#define HASHER_B HROLLING
#include "./hash_composite_inc.h" /* NOLINT(build/include) */
#undef HASHER_A
#undef HASHER_B
#undef HASHER
#undef FN
#undef CAT
#undef EXPAND_CAT
#define FOR_GENERIC_HASHERS(H) H(2) H(3) H(4) H(5) H(6) H(7) H(8) H(9) \
H(40) H(41) H(42)
#define FOR_GENERIC_HASHERS(H) H(2) H(3) H(4) H(5) H(6) H(40) H(41) H(42) H(54)\
H(35) H(55) H(65)
#define FOR_ALL_HASHERS(H) FOR_GENERIC_HASHERS(H) H(10)
typedef struct Hashers {
#define _MEMBER(N) H ## N* h ## N;
FOR_ALL_HASHERS(_MEMBER)
#undef _MEMBER
} Hashers;
static BROTLI_INLINE void InitHashers(Hashers* self) {
#define _INIT(N) self->h ## N = 0;
FOR_ALL_HASHERS(_INIT)
#undef _INIT
static BROTLI_INLINE void DestroyHasher(
MemoryManager* m, HasherHandle* handle) {
if (*handle == NULL) return;
BROTLI_FREE(m, *handle);
}
static BROTLI_INLINE void DestroyHashers(MemoryManager* m, Hashers* self) {
if (self->h10) CleanupH10(m, self->h10);
#define _CLEANUP(N) BROTLI_FREE(m, self->h ## N)
FOR_ALL_HASHERS(_CLEANUP)
#undef _CLEANUP
static BROTLI_INLINE void HasherReset(HasherHandle handle) {
if (handle == NULL) return;
GetHasherCommon(handle)->is_prepared_ = BROTLI_FALSE;
}
static BROTLI_INLINE void HashersReset(Hashers* self, int type) {
switch (type) {
#define _RESET(N) case N: ResetH ## N(self->h ## N); break;
FOR_ALL_HASHERS(_RESET)
#undef _RESET
default: break;
static BROTLI_INLINE size_t HasherSize(const BrotliEncoderParams* params,
BROTLI_BOOL one_shot, const size_t input_size) {
size_t result = sizeof(HasherCommon);
switch (params->hasher.type) {
#define SIZE_(N) \
case N: \
result += HashMemAllocInBytesH ## N(params, one_shot, input_size); \
break;
FOR_ALL_HASHERS(SIZE_)
#undef SIZE_
default:
break;
}
return result;
}
static BROTLI_INLINE void HasherSetup(MemoryManager* m, HasherHandle* handle,
BrotliEncoderParams* params, const uint8_t* data, size_t position,
size_t input_size, BROTLI_BOOL is_last) {
HasherHandle self = NULL;
HasherCommon* common = NULL;
BROTLI_BOOL one_shot = (position == 0 && is_last);
if (*handle == NULL) {
size_t alloc_size;
ChooseHasher(params, &params->hasher);
alloc_size = HasherSize(params, one_shot, input_size);
self = BROTLI_ALLOC(m, uint8_t, alloc_size);
if (BROTLI_IS_OOM(m)) return;
*handle = self;
common = GetHasherCommon(self);
common->params = params->hasher;
switch (common->params.type) {
#define INITIALIZE_(N) \
case N: \
InitializeH ## N(*handle, params); \
break;
FOR_ALL_HASHERS(INITIALIZE_);
#undef INITIALIZE_
default:
break;
}
HasherReset(*handle);
}
self = *handle;
common = GetHasherCommon(self);
if (!common->is_prepared_) {
switch (common->params.type) {
#define PREPARE_(N) \
case N: \
PrepareH ## N(self, one_shot, input_size, data); \
break;
FOR_ALL_HASHERS(PREPARE_)
#undef PREPARE_
default: break;
}
if (position == 0) {
common->dict_num_lookups = 0;
common->dict_num_matches = 0;
}
common->is_prepared_ = BROTLI_TRUE;
}
}
static BROTLI_INLINE void HashersSetup(
MemoryManager* m, Hashers* self, int type) {
switch (type) {
#define _SETUP(N) case N: self->h ## N = BROTLI_ALLOC(m, H ## N, 1); break;
FOR_ALL_HASHERS(_SETUP)
#undef _SETUP
default: break;
}
static BROTLI_INLINE void InitOrStitchToPreviousBlock(
MemoryManager* m, HasherHandle* handle, const uint8_t* data, size_t mask,
BrotliEncoderParams* params, size_t position, size_t input_size,
BROTLI_BOOL is_last) {
HasherHandle self;
HasherSetup(m, handle, params, data, position, input_size, is_last);
if (BROTLI_IS_OOM(m)) return;
if (type == 10) InitializeH10(self->h10);
HashersReset(self, type);
}
#define _WARMUP_HASH(N) \
static BROTLI_INLINE void WarmupHashH ## N(MemoryManager* m, \
const BrotliEncoderParams* params, const size_t size, const uint8_t* dict, \
H ## N* hasher) { \
size_t overlap = (StoreLookaheadH ## N()) - 1; \
size_t i; \
InitH ## N(m, hasher, dict, params, 0, size, BROTLI_FALSE); \
if (BROTLI_IS_OOM(m)) return; \
for (i = 0; i + overlap < size; i++) { \
StoreH ## N(hasher, dict, ~(size_t)0, i); \
} \
}
FOR_ALL_HASHERS(_WARMUP_HASH)
#undef _WARMUP_HASH
/* Custom LZ77 window. */
static BROTLI_INLINE void HashersPrependCustomDictionary(
MemoryManager* m, Hashers* self, const BrotliEncoderParams* params,
const size_t size, const uint8_t* dict) {
int hasher_type = ChooseHasher(params);
switch (hasher_type) {
#define _PREPEND(N) \
case N: WarmupHashH ## N(m, params, size, dict, self->h ## N); break;
FOR_ALL_HASHERS(_PREPEND)
#undef _PREPEND
self = *handle;
switch (GetHasherCommon(self)->params.type) {
#define INIT_(N) \
case N: \
StitchToPreviousBlockH ## N(self, input_size, position, data, mask); \
break;
FOR_ALL_HASHERS(INIT_)
#undef INIT_
default: break;
}
if (BROTLI_IS_OOM(m)) return;
}
#if defined(__cplusplus) || defined(c_plusplus)
} /* extern "C" */
#endif

View File

@ -0,0 +1,133 @@
/* NOLINT(build/header_guard) */
/* Copyright 2018 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* template parameters: FN, HASHER_A, HASHER_B */
/* Composite hasher: This hasher allows to combine two other hashers, HASHER_A
and HASHER_B. */
#define HashComposite HASHER()
#define FN_A(X) EXPAND_CAT(X, HASHER_A)
#define FN_B(X) EXPAND_CAT(X, HASHER_B)
static BROTLI_INLINE size_t FN(HashTypeLength)(void) {
size_t a = FN_A(HashTypeLength)();
size_t b = FN_B(HashTypeLength)();
return a > b ? a : b;
}
static BROTLI_INLINE size_t FN(StoreLookahead)(void) {
size_t a = FN_A(StoreLookahead)();
size_t b = FN_B(StoreLookahead)();
return a > b ? a : b;
}
typedef struct HashComposite {
HasherHandle ha;
HasherHandle hb;
const BrotliEncoderParams* params;
} HashComposite;
static BROTLI_INLINE HashComposite* FN(Self)(HasherHandle handle) {
return (HashComposite*)&(GetHasherCommon(handle)[1]);
}
static void FN(Initialize)(
HasherHandle handle, const BrotliEncoderParams* params) {
HashComposite* self = FN(Self)(handle);
self->ha = 0;
self->hb = 0;
self->params = params;
/* TODO: Initialize of the hashers is defered to Prepare (and params
remembered here) because we don't get the one_shot and input_size params
here that are needed to know the memory size of them. Instead provide
those params to all hashers FN(Initialize) */
}
static void FN(Prepare)(HasherHandle handle, BROTLI_BOOL one_shot,
size_t input_size, const uint8_t* data) {
HashComposite* self = FN(Self)(handle);
if (!self->ha) {
HasherCommon* common_a;
HasherCommon* common_b;
self->ha = handle + sizeof(HasherCommon) + sizeof(HashComposite);
common_a = (HasherCommon*)self->ha;
common_a->params = self->params->hasher;
common_a->is_prepared_ = BROTLI_FALSE;
common_a->dict_num_lookups = 0;
common_a->dict_num_matches = 0;
FN_A(Initialize)(self->ha, self->params);
self->hb = self->ha + sizeof(HasherCommon) + FN_A(HashMemAllocInBytes)(
self->params, one_shot, input_size);
common_b = (HasherCommon*)self->hb;
common_b->params = self->params->hasher;
common_b->is_prepared_ = BROTLI_FALSE;
common_b->dict_num_lookups = 0;
common_b->dict_num_matches = 0;
FN_B(Initialize)(self->hb, self->params);
}
FN_A(Prepare)(self->ha, one_shot, input_size, data);
FN_B(Prepare)(self->hb, one_shot, input_size, data);
}
static BROTLI_INLINE size_t FN(HashMemAllocInBytes)(
const BrotliEncoderParams* params, BROTLI_BOOL one_shot,
size_t input_size) {
return sizeof(HashComposite) + 2 * sizeof(HasherCommon) +
FN_A(HashMemAllocInBytes)(params, one_shot, input_size) +
FN_B(HashMemAllocInBytes)(params, one_shot, input_size);
}
static BROTLI_INLINE void FN(Store)(HasherHandle BROTLI_RESTRICT handle,
const uint8_t* BROTLI_RESTRICT data, const size_t mask, const size_t ix) {
HashComposite* self = FN(Self)(handle);
FN_A(Store)(self->ha, data, mask, ix);
FN_B(Store)(self->hb, data, mask, ix);
}
static BROTLI_INLINE void FN(StoreRange)(HasherHandle handle,
const uint8_t* data, const size_t mask, const size_t ix_start,
const size_t ix_end) {
HashComposite* self = FN(Self)(handle);
FN_A(StoreRange)(self->ha, data, mask, ix_start, ix_end);
FN_B(StoreRange)(self->hb, data, mask, ix_start, ix_end);
}
static BROTLI_INLINE void FN(StitchToPreviousBlock)(HasherHandle handle,
size_t num_bytes, size_t position, const uint8_t* ringbuffer,
size_t ring_buffer_mask) {
HashComposite* self = FN(Self)(handle);
FN_A(StitchToPreviousBlock)(self->ha, num_bytes, position, ringbuffer,
ring_buffer_mask);
FN_B(StitchToPreviousBlock)(self->hb, num_bytes, position, ringbuffer,
ring_buffer_mask);
}
static BROTLI_INLINE void FN(PrepareDistanceCache)(
HasherHandle handle, int* BROTLI_RESTRICT distance_cache) {
HashComposite* self = FN(Self)(handle);
FN_A(PrepareDistanceCache)(self->ha, distance_cache);
FN_B(PrepareDistanceCache)(self->hb, distance_cache);
}
static BROTLI_INLINE void FN(FindLongestMatch)(HasherHandle handle,
const BrotliEncoderDictionary* dictionary,
const uint8_t* BROTLI_RESTRICT data, const size_t ring_buffer_mask,
const int* BROTLI_RESTRICT distance_cache, const size_t cur_ix,
const size_t max_length, const size_t max_backward, const size_t gap,
const size_t max_distance, HasherSearchResult* BROTLI_RESTRICT out) {
HashComposite* self = FN(Self)(handle);
FN_A(FindLongestMatch)(self->ha, dictionary, data, ring_buffer_mask,
distance_cache, cur_ix, max_length, max_backward, gap, max_distance, out);
FN_B(FindLongestMatch)(self->hb, dictionary, data, ring_buffer_mask,
distance_cache, cur_ix, max_length, max_backward, gap, max_distance, out);
}
#undef HashComposite

View File

@ -28,8 +28,8 @@ static BROTLI_INLINE size_t FN(HashTypeLength)(void) { return 4; }
static BROTLI_INLINE size_t FN(StoreLookahead)(void) { return 4; }
/* HashBytes is the function that chooses the bucket to place the address in.*/
static BROTLI_INLINE size_t FN(HashBytes)(const uint8_t *data) {
const uint32_t h = BROTLI_UNALIGNED_LOAD32(data) * kHashMul32;
static BROTLI_INLINE size_t FN(HashBytes)(const uint8_t* data) {
const uint32_t h = BROTLI_UNALIGNED_LOAD32LE(data) * kHashMul32;
/* The higher bits contain more mixture from the multiplication,
so we take our results from there. */
return h >> (32 - BUCKET_BITS);
@ -51,65 +51,57 @@ typedef struct HashForgetfulChain {
uint8_t tiny_hash[65536];
FN(Bank) banks[NUM_BANKS];
uint16_t free_slot_idx[NUM_BANKS];
BROTLI_BOOL is_dirty_;
DictionarySearchStatictics dict_search_stats_;
size_t max_hops;
} HashForgetfulChain;
static void FN(Reset)(HashForgetfulChain* self) {
self->is_dirty_ = BROTLI_TRUE;
DictionarySearchStaticticsReset(&self->dict_search_stats_);
static BROTLI_INLINE HashForgetfulChain* FN(Self)(HasherHandle handle) {
return (HashForgetfulChain*)&(GetHasherCommon(handle)[1]);
}
static void FN(InitEmpty)(HashForgetfulChain* self) {
if (self->is_dirty_) {
static void FN(Initialize)(
HasherHandle handle, const BrotliEncoderParams* params) {
FN(Self)(handle)->max_hops =
(params->quality > 6 ? 7u : 8u) << (params->quality - 4);
}
static void FN(Prepare)(HasherHandle handle, BROTLI_BOOL one_shot,
size_t input_size, const uint8_t* data) {
HashForgetfulChain* self = FN(Self)(handle);
/* Partial preparation is 100 times slower (per socket). */
size_t partial_prepare_threshold = BUCKET_SIZE >> 6;
if (one_shot && input_size <= partial_prepare_threshold) {
size_t i;
for (i = 0; i < input_size; ++i) {
size_t bucket = FN(HashBytes)(&data[i]);
/* See InitEmpty comment. */
self->addr[bucket] = 0xCCCCCCCC;
self->head[bucket] = 0xCCCC;
}
} else {
/* Fill |addr| array with 0xCCCCCCCC value. Because of wrapping, position
processed by hasher never reaches 3GB + 64M; this makes all new chains
to be terminated after the first node. */
memset(self->addr, 0xCC, sizeof(self->addr));
memset(self->head, 0, sizeof(self->head));
memset(self->tiny_hash, 0, sizeof(self->tiny_hash));
memset(self->free_slot_idx, 0, sizeof(self->free_slot_idx));
self->is_dirty_ = BROTLI_FALSE;
}
}
static void FN(InitForData)(HashForgetfulChain* self, const uint8_t* data,
size_t num) {
size_t i;
for (i = 0; i < num; ++i) {
size_t bucket = FN(HashBytes)(&data[i]);
/* See InitEmpty comment. */
self->addr[bucket] = 0xCCCCCCCC;
self->head[bucket] = 0xCCCC;
}
memset(self->tiny_hash, 0, sizeof(self->tiny_hash));
memset(self->free_slot_idx, 0, sizeof(self->free_slot_idx));
if (num != 0) {
self->is_dirty_ = BROTLI_FALSE;
}
}
static void FN(Init)(
MemoryManager* m, HashForgetfulChain* self, const uint8_t* data,
const BrotliEncoderParams* params, size_t position, size_t bytes,
BROTLI_BOOL is_last) {
/* Choose which init method is faster.
Init() is about 100 times faster than InitForData(). */
const size_t kMaxBytesForPartialHashInit = BUCKET_SIZE >> 6;
BROTLI_UNUSED(m);
self->max_hops = (params->quality > 6 ? 7u : 8u) << (params->quality - 4);
if (position == 0 && is_last && bytes <= kMaxBytesForPartialHashInit) {
FN(InitForData)(self, data, bytes);
} else {
FN(InitEmpty)(self);
}
static BROTLI_INLINE size_t FN(HashMemAllocInBytes)(
const BrotliEncoderParams* params, BROTLI_BOOL one_shot,
size_t input_size) {
BROTLI_UNUSED(params);
BROTLI_UNUSED(one_shot);
BROTLI_UNUSED(input_size);
return sizeof(HashForgetfulChain);
}
/* Look at 4 bytes at &data[ix & mask]. Compute a hash from these, and prepend
node to corresponding chain; also update tiny_hash for current position. */
static BROTLI_INLINE void FN(Store)(HashForgetfulChain* BROTLI_RESTRICT self,
static BROTLI_INLINE void FN(Store)(HasherHandle BROTLI_RESTRICT handle,
const uint8_t* BROTLI_RESTRICT data, const size_t mask, const size_t ix) {
HashForgetfulChain* self = FN(Self)(handle);
const size_t key = FN(HashBytes)(&data[ix & mask]);
const size_t bank = key & (NUM_BANKS - 1);
const size_t idx = self->free_slot_idx[bank]++ & (BANK_SIZE - 1);
@ -122,56 +114,68 @@ static BROTLI_INLINE void FN(Store)(HashForgetfulChain* BROTLI_RESTRICT self,
self->head[key] = (uint16_t)idx;
}
static BROTLI_INLINE void FN(StoreRange)(HashForgetfulChain* self,
const uint8_t *data, const size_t mask, const size_t ix_start,
static BROTLI_INLINE void FN(StoreRange)(HasherHandle handle,
const uint8_t* data, const size_t mask, const size_t ix_start,
const size_t ix_end) {
size_t i;
for (i = ix_start; i < ix_end; ++i) {
FN(Store)(self, data, mask, i);
FN(Store)(handle, data, mask, i);
}
}
static BROTLI_INLINE void FN(StitchToPreviousBlock)(HashForgetfulChain* self,
static BROTLI_INLINE void FN(StitchToPreviousBlock)(HasherHandle handle,
size_t num_bytes, size_t position, const uint8_t* ringbuffer,
size_t ring_buffer_mask) {
if (num_bytes >= FN(HashTypeLength)() - 1 && position >= 3) {
/* Prepare the hashes for three last bytes of the last write.
These could not be calculated before, since they require knowledge
of both the previous and the current block. */
FN(Store)(self, ringbuffer, ring_buffer_mask, position - 3);
FN(Store)(self, ringbuffer, ring_buffer_mask, position - 2);
FN(Store)(self, ringbuffer, ring_buffer_mask, position - 1);
FN(Store)(handle, ringbuffer, ring_buffer_mask, position - 3);
FN(Store)(handle, ringbuffer, ring_buffer_mask, position - 2);
FN(Store)(handle, ringbuffer, ring_buffer_mask, position - 1);
}
}
static BROTLI_INLINE void FN(PrepareDistanceCache)(
HasherHandle handle, int* BROTLI_RESTRICT distance_cache) {
BROTLI_UNUSED(handle);
PrepareDistanceCache(distance_cache, NUM_LAST_DISTANCES_TO_CHECK);
}
/* Find a longest backward match of &data[cur_ix] up to the length of
max_length and stores the position cur_ix in the hash table.
REQUIRES: FN(PrepareDistanceCache) must be invoked for current distance cache
values; if this method is invoked repeatedly with the same distance
cache values, it is enough to invoke FN(PrepareDistanceCache) once.
Does not look for matches longer than max_length.
Does not look for matches further away than max_backward.
Writes the best match into |out|.
Returns 1 when match is found, otherwise 0. */
static BROTLI_INLINE BROTLI_BOOL FN(FindLongestMatch)(
HashForgetfulChain* self, const uint8_t* BROTLI_RESTRICT data,
const size_t ring_buffer_mask, const int* BROTLI_RESTRICT distance_cache,
|out|->score is updated only if a better match is found. */
static BROTLI_INLINE void FN(FindLongestMatch)(HasherHandle handle,
const BrotliEncoderDictionary* dictionary,
const uint8_t* BROTLI_RESTRICT data, const size_t ring_buffer_mask,
const int* BROTLI_RESTRICT distance_cache,
const size_t cur_ix, const size_t max_length, const size_t max_backward,
const size_t gap, const size_t max_distance,
HasherSearchResult* BROTLI_RESTRICT out) {
HashForgetfulChain* self = FN(Self)(handle);
const size_t cur_ix_masked = cur_ix & ring_buffer_mask;
BROTLI_BOOL is_match_found = BROTLI_FALSE;
/* Don't accept a short copy from far away. */
score_t min_score = out->score;
score_t best_score = out->score;
size_t best_len = out->len;
size_t i;
const size_t key = FN(HashBytes)(&data[cur_ix_masked]);
const uint8_t tiny_hash = (uint8_t)(key);
out->len = 0;
out->len_x_code = 0;
out->len_code_delta = 0;
/* Try last distance first. */
for (i = 0; i < NUM_LAST_DISTANCES_TO_CHECK; ++i) {
const size_t idx = kDistanceCacheIndex[i];
const size_t backward =
(size_t)(distance_cache[idx] + kDistanceCacheOffset[i]);
const size_t backward = (size_t)distance_cache[i];
size_t prev_ix = (cur_ix - backward);
/* For distance code 0 we want to consider 2-byte matches. */
if (i > 0 && self->tiny_hash[(uint16_t)prev_ix] != tiny_hash) continue;
if (prev_ix >= cur_ix || backward > max_backward) {
continue;
@ -182,14 +186,16 @@ static BROTLI_INLINE BROTLI_BOOL FN(FindLongestMatch)(
&data[cur_ix_masked],
max_length);
if (len >= 2) {
score_t score = BackwardReferenceScoreUsingLastDistance(len, i);
score_t score = BackwardReferenceScoreUsingLastDistance(len);
if (best_score < score) {
best_score = score;
best_len = len;
out->len = best_len;
out->distance = backward;
out->score = best_score;
is_match_found = BROTLI_TRUE;
if (i != 0) score -= BackwardReferencePenaltyUsingLastDistance(i);
if (best_score < score) {
best_score = score;
best_len = len;
out->len = best_len;
out->distance = backward;
out->score = best_score;
}
}
}
}
@ -228,18 +234,17 @@ static BROTLI_INLINE BROTLI_BOOL FN(FindLongestMatch)(
out->len = best_len;
out->distance = backward;
out->score = best_score;
is_match_found = BROTLI_TRUE;
}
}
}
}
FN(Store)(self, data, ring_buffer_mask, cur_ix);
FN(Store)(handle, data, ring_buffer_mask, cur_ix);
}
if (!is_match_found) {
is_match_found = SearchInStaticDictionary(&self->dict_search_stats_,
&data[cur_ix_masked], max_length, max_backward, out, BROTLI_FALSE);
if (out->score == min_score) {
SearchInStaticDictionary(dictionary,
handle, &data[cur_ix_masked], max_length, max_backward + gap,
max_distance, out, BROTLI_FALSE);
}
return is_match_found;
}
#undef BANK_SIZE

View File

@ -0,0 +1,266 @@
/* NOLINT(build/header_guard) */
/* Copyright 2010 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* template parameters: FN */
/* A (forgetful) hash table to the data seen by the compressor, to
help create backward references to previous data.
This is a hash map of fixed size (bucket_size_) to a ring buffer of
fixed size (block_size_). The ring buffer contains the last block_size_
index positions of the given hash key in the compressed data. */
#define HashLongestMatch HASHER()
static BROTLI_INLINE size_t FN(HashTypeLength)(void) { return 8; }
static BROTLI_INLINE size_t FN(StoreLookahead)(void) { return 8; }
/* HashBytes is the function that chooses the bucket to place the address in. */
static BROTLI_INLINE uint32_t FN(HashBytes)(const uint8_t* data,
const uint64_t mask,
const int shift) {
const uint64_t h = (BROTLI_UNALIGNED_LOAD64LE(data) & mask) * kHashMul64Long;
/* The higher bits contain more mixture from the multiplication,
so we take our results from there. */
return (uint32_t)(h >> shift);
}
typedef struct HashLongestMatch {
/* Number of hash buckets. */
size_t bucket_size_;
/* Only block_size_ newest backward references are kept,
and the older are forgotten. */
size_t block_size_;
/* Left-shift for computing hash bucket index from hash value. */
int hash_shift_;
/* Mask for selecting the next 4-8 bytes of input */
uint64_t hash_mask_;
/* Mask for accessing entries in a block (in a ring-buffer manner). */
uint32_t block_mask_;
/* --- Dynamic size members --- */
/* Number of entries in a particular bucket. */
/* uint16_t num[bucket_size]; */
/* Buckets containing block_size_ of backward references. */
/* uint32_t* buckets[bucket_size * block_size]; */
} HashLongestMatch;
static BROTLI_INLINE HashLongestMatch* FN(Self)(HasherHandle handle) {
return (HashLongestMatch*)&(GetHasherCommon(handle)[1]);
}
static BROTLI_INLINE uint16_t* FN(Num)(HashLongestMatch* self) {
return (uint16_t*)(&self[1]);
}
static BROTLI_INLINE uint32_t* FN(Buckets)(HashLongestMatch* self) {
return (uint32_t*)(&FN(Num)(self)[self->bucket_size_]);
}
static void FN(Initialize)(
HasherHandle handle, const BrotliEncoderParams* params) {
HasherCommon* common = GetHasherCommon(handle);
HashLongestMatch* self = FN(Self)(handle);
BROTLI_UNUSED(params);
self->hash_shift_ = 64 - common->params.bucket_bits;
self->hash_mask_ = (~((uint64_t)0U)) >> (64 - 8 * common->params.hash_len);
self->bucket_size_ = (size_t)1 << common->params.bucket_bits;
self->block_size_ = (size_t)1 << common->params.block_bits;
self->block_mask_ = (uint32_t)(self->block_size_ - 1);
}
static void FN(Prepare)(HasherHandle handle, BROTLI_BOOL one_shot,
size_t input_size, const uint8_t* data) {
HashLongestMatch* self = FN(Self)(handle);
uint16_t* num = FN(Num)(self);
/* Partial preparation is 100 times slower (per socket). */
size_t partial_prepare_threshold = self->bucket_size_ >> 6;
if (one_shot && input_size <= partial_prepare_threshold) {
size_t i;
for (i = 0; i < input_size; ++i) {
const uint32_t key = FN(HashBytes)(&data[i], self->hash_mask_,
self->hash_shift_);
num[key] = 0;
}
} else {
memset(num, 0, self->bucket_size_ * sizeof(num[0]));
}
}
static BROTLI_INLINE size_t FN(HashMemAllocInBytes)(
const BrotliEncoderParams* params, BROTLI_BOOL one_shot,
size_t input_size) {
size_t bucket_size = (size_t)1 << params->hasher.bucket_bits;
size_t block_size = (size_t)1 << params->hasher.block_bits;
BROTLI_UNUSED(one_shot);
BROTLI_UNUSED(input_size);
return sizeof(HashLongestMatch) + bucket_size * (2 + 4 * block_size);
}
/* Look at 4 bytes at &data[ix & mask].
Compute a hash from these, and store the value of ix at that position. */
static BROTLI_INLINE void FN(Store)(HasherHandle handle, const uint8_t* data,
const size_t mask, const size_t ix) {
HashLongestMatch* self = FN(Self)(handle);
uint16_t* num = FN(Num)(self);
const uint32_t key = FN(HashBytes)(&data[ix & mask], self->hash_mask_,
self->hash_shift_);
const size_t minor_ix = num[key] & self->block_mask_;
const size_t offset =
minor_ix + (key << GetHasherCommon(handle)->params.block_bits);
FN(Buckets)(self)[offset] = (uint32_t)ix;
++num[key];
}
static BROTLI_INLINE void FN(StoreRange)(HasherHandle handle,
const uint8_t* data, const size_t mask, const size_t ix_start,
const size_t ix_end) {
size_t i;
for (i = ix_start; i < ix_end; ++i) {
FN(Store)(handle, data, mask, i);
}
}
static BROTLI_INLINE void FN(StitchToPreviousBlock)(HasherHandle handle,
size_t num_bytes, size_t position, const uint8_t* ringbuffer,
size_t ringbuffer_mask) {
if (num_bytes >= FN(HashTypeLength)() - 1 && position >= 3) {
/* Prepare the hashes for three last bytes of the last write.
These could not be calculated before, since they require knowledge
of both the previous and the current block. */
FN(Store)(handle, ringbuffer, ringbuffer_mask, position - 3);
FN(Store)(handle, ringbuffer, ringbuffer_mask, position - 2);
FN(Store)(handle, ringbuffer, ringbuffer_mask, position - 1);
}
}
static BROTLI_INLINE void FN(PrepareDistanceCache)(
HasherHandle handle, int* BROTLI_RESTRICT distance_cache) {
PrepareDistanceCache(distance_cache,
GetHasherCommon(handle)->params.num_last_distances_to_check);
}
/* Find a longest backward match of &data[cur_ix] up to the length of
max_length and stores the position cur_ix in the hash table.
REQUIRES: FN(PrepareDistanceCache) must be invoked for current distance cache
values; if this method is invoked repeatedly with the same distance
cache values, it is enough to invoke FN(PrepareDistanceCache) once.
Does not look for matches longer than max_length.
Does not look for matches further away than max_backward.
Writes the best match into |out|.
|out|->score is updated only if a better match is found. */
static BROTLI_INLINE void FN(FindLongestMatch)(HasherHandle handle,
const BrotliEncoderDictionary* dictionary,
const uint8_t* BROTLI_RESTRICT data, const size_t ring_buffer_mask,
const int* BROTLI_RESTRICT distance_cache, const size_t cur_ix,
const size_t max_length, const size_t max_backward, const size_t gap,
const size_t max_distance, HasherSearchResult* BROTLI_RESTRICT out) {
HasherCommon* common = GetHasherCommon(handle);
HashLongestMatch* self = FN(Self)(handle);
uint16_t* num = FN(Num)(self);
uint32_t* buckets = FN(Buckets)(self);
const size_t cur_ix_masked = cur_ix & ring_buffer_mask;
/* Don't accept a short copy from far away. */
score_t min_score = out->score;
score_t best_score = out->score;
size_t best_len = out->len;
size_t i;
out->len = 0;
out->len_code_delta = 0;
/* Try last distance first. */
for (i = 0; i < (size_t)common->params.num_last_distances_to_check; ++i) {
const size_t backward = (size_t)distance_cache[i];
size_t prev_ix = (size_t)(cur_ix - backward);
if (prev_ix >= cur_ix) {
continue;
}
if (BROTLI_PREDICT_FALSE(backward > max_backward)) {
continue;
}
prev_ix &= ring_buffer_mask;
if (cur_ix_masked + best_len > ring_buffer_mask ||
prev_ix + best_len > ring_buffer_mask ||
data[cur_ix_masked + best_len] != data[prev_ix + best_len]) {
continue;
}
{
const size_t len = FindMatchLengthWithLimit(&data[prev_ix],
&data[cur_ix_masked],
max_length);
if (len >= 3 || (len == 2 && i < 2)) {
/* Comparing for >= 2 does not change the semantics, but just saves for
a few unnecessary binary logarithms in backward reference score,
since we are not interested in such short matches. */
score_t score = BackwardReferenceScoreUsingLastDistance(len);
if (best_score < score) {
if (i != 0) score -= BackwardReferencePenaltyUsingLastDistance(i);
if (best_score < score) {
best_score = score;
best_len = len;
out->len = best_len;
out->distance = backward;
out->score = best_score;
}
}
}
}
}
{
const uint32_t key = FN(HashBytes)(
&data[cur_ix_masked], self->hash_mask_, self->hash_shift_);
uint32_t* BROTLI_RESTRICT bucket =
&buckets[key << common->params.block_bits];
const size_t down =
(num[key] > self->block_size_) ?
(num[key] - self->block_size_) : 0u;
for (i = num[key]; i > down;) {
size_t prev_ix = bucket[--i & self->block_mask_];
const size_t backward = cur_ix - prev_ix;
if (BROTLI_PREDICT_FALSE(backward > max_backward)) {
break;
}
prev_ix &= ring_buffer_mask;
if (cur_ix_masked + best_len > ring_buffer_mask ||
prev_ix + best_len > ring_buffer_mask ||
data[cur_ix_masked + best_len] != data[prev_ix + best_len]) {
continue;
}
{
const size_t len = FindMatchLengthWithLimit(&data[prev_ix],
&data[cur_ix_masked],
max_length);
if (len >= 4) {
/* Comparing for >= 3 does not change the semantics, but just saves
for a few unnecessary binary logarithms in backward reference
score, since we are not interested in such short matches. */
score_t score = BackwardReferenceScore(len, backward);
if (best_score < score) {
best_score = score;
best_len = len;
out->len = best_len;
out->distance = backward;
out->score = best_score;
}
}
}
}
bucket[num[key] & self->block_mask_] = (uint32_t)cur_ix;
++num[key];
}
if (min_score == out->score) {
SearchInStaticDictionary(dictionary,
handle, &data[cur_ix_masked], max_length, max_backward + gap,
max_distance, out, BROTLI_FALSE);
}
}
#undef HashLongestMatch

View File

@ -5,158 +5,177 @@
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* template parameters: FN, BUCKET_BITS, BLOCK_BITS,
NUM_LAST_DISTANCES_TO_CHECK */
/* template parameters: FN */
/* A (forgetful) hash table to the data seen by the compressor, to
help create backward references to previous data.
This is a hash map of fixed size (BUCKET_SIZE) to a ring buffer of
fixed size (BLOCK_SIZE). The ring buffer contains the last BLOCK_SIZE
This is a hash map of fixed size (bucket_size_) to a ring buffer of
fixed size (block_size_). The ring buffer contains the last block_size_
index positions of the given hash key in the compressed data. */
#define HashLongestMatch HASHER()
/* Number of hash buckets. */
#define BUCKET_SIZE (1 << BUCKET_BITS)
/* Only BLOCK_SIZE newest backward references are kept,
and the older are forgotten. */
#define BLOCK_SIZE (1u << BLOCK_BITS)
/* Mask for accessing entries in a block (in a ringbuffer manner). */
#define BLOCK_MASK ((1 << BLOCK_BITS) - 1)
#define HASH_MAP_SIZE (2 << BUCKET_BITS)
static BROTLI_INLINE size_t FN(HashTypeLength)(void) { return 4; }
static BROTLI_INLINE size_t FN(StoreLookahead)(void) { return 4; }
/* HashBytes is the function that chooses the bucket to place
the address in. The HashLongestMatch and HashLongestMatchQuickly
classes have separate, different implementations of hashing. */
static uint32_t FN(HashBytes)(const uint8_t *data) {
uint32_t h = BROTLI_UNALIGNED_LOAD32(data) * kHashMul32;
/* HashBytes is the function that chooses the bucket to place the address in. */
static uint32_t FN(HashBytes)(const uint8_t* data, const int shift) {
uint32_t h = BROTLI_UNALIGNED_LOAD32LE(data) * kHashMul32;
/* The higher bits contain more mixture from the multiplication,
so we take our results from there. */
return h >> (32 - BUCKET_BITS);
return (uint32_t)(h >> shift);
}
typedef struct HashLongestMatch {
/* Number of hash buckets. */
size_t bucket_size_;
/* Only block_size_ newest backward references are kept,
and the older are forgotten. */
size_t block_size_;
/* Left-shift for computing hash bucket index from hash value. */
int hash_shift_;
/* Mask for accessing entries in a block (in a ring-buffer manner). */
uint32_t block_mask_;
/* --- Dynamic size members --- */
/* Number of entries in a particular bucket. */
uint16_t num_[BUCKET_SIZE];
/* uint16_t num[bucket_size]; */
/* Buckets containing BLOCK_SIZE of backward references. */
uint32_t buckets_[BLOCK_SIZE << BUCKET_BITS];
/* True if num_ array needs to be initialized. */
BROTLI_BOOL is_dirty_;
DictionarySearchStatictics dict_search_stats_;
/* Buckets containing block_size_ of backward references. */
/* uint32_t* buckets[bucket_size * block_size]; */
} HashLongestMatch;
static void FN(Reset)(HashLongestMatch* self) {
self->is_dirty_ = BROTLI_TRUE;
DictionarySearchStaticticsReset(&self->dict_search_stats_);
static BROTLI_INLINE HashLongestMatch* FN(Self)(HasherHandle handle) {
return (HashLongestMatch*)&(GetHasherCommon(handle)[1]);
}
static void FN(InitEmpty)(HashLongestMatch* self) {
if (self->is_dirty_) {
memset(self->num_, 0, sizeof(self->num_));
self->is_dirty_ = BROTLI_FALSE;
}
static BROTLI_INLINE uint16_t* FN(Num)(HashLongestMatch* self) {
return (uint16_t*)(&self[1]);
}
static void FN(InitForData)(HashLongestMatch* self, const uint8_t* data,
size_t num) {
size_t i;
for (i = 0; i < num; ++i) {
const uint32_t key = FN(HashBytes)(&data[i]);
self->num_[key] = 0;
}
if (num != 0) {
self->is_dirty_ = BROTLI_FALSE;
}
static BROTLI_INLINE uint32_t* FN(Buckets)(HashLongestMatch* self) {
return (uint32_t*)(&FN(Num)(self)[self->bucket_size_]);
}
static void FN(Init)(
MemoryManager* m, HashLongestMatch* self, const uint8_t* data,
const BrotliEncoderParams* params, size_t position, size_t bytes,
BROTLI_BOOL is_last) {
/* Choose which init method is faster.
Init() is about 100 times faster than InitForData(). */
const size_t kMaxBytesForPartialHashInit = HASH_MAP_SIZE >> 7;
BROTLI_UNUSED(m);
static void FN(Initialize)(
HasherHandle handle, const BrotliEncoderParams* params) {
HasherCommon* common = GetHasherCommon(handle);
HashLongestMatch* self = FN(Self)(handle);
BROTLI_UNUSED(params);
if (position == 0 && is_last && bytes <= kMaxBytesForPartialHashInit) {
FN(InitForData)(self, data, bytes);
self->hash_shift_ = 32 - common->params.bucket_bits;
self->bucket_size_ = (size_t)1 << common->params.bucket_bits;
self->block_size_ = (size_t)1 << common->params.block_bits;
self->block_mask_ = (uint32_t)(self->block_size_ - 1);
}
static void FN(Prepare)(HasherHandle handle, BROTLI_BOOL one_shot,
size_t input_size, const uint8_t* data) {
HashLongestMatch* self = FN(Self)(handle);
uint16_t* num = FN(Num)(self);
/* Partial preparation is 100 times slower (per socket). */
size_t partial_prepare_threshold = self->bucket_size_ >> 6;
if (one_shot && input_size <= partial_prepare_threshold) {
size_t i;
for (i = 0; i < input_size; ++i) {
const uint32_t key = FN(HashBytes)(&data[i], self->hash_shift_);
num[key] = 0;
}
} else {
FN(InitEmpty)(self);
memset(num, 0, self->bucket_size_ * sizeof(num[0]));
}
}
static BROTLI_INLINE size_t FN(HashMemAllocInBytes)(
const BrotliEncoderParams* params, BROTLI_BOOL one_shot,
size_t input_size) {
size_t bucket_size = (size_t)1 << params->hasher.bucket_bits;
size_t block_size = (size_t)1 << params->hasher.block_bits;
BROTLI_UNUSED(one_shot);
BROTLI_UNUSED(input_size);
return sizeof(HashLongestMatch) + bucket_size * (2 + 4 * block_size);
}
/* Look at 4 bytes at &data[ix & mask].
Compute a hash from these, and store the value of ix at that position. */
static BROTLI_INLINE void FN(Store)(HashLongestMatch* self, const uint8_t *data,
static BROTLI_INLINE void FN(Store)(HasherHandle handle, const uint8_t* data,
const size_t mask, const size_t ix) {
const uint32_t key = FN(HashBytes)(&data[ix & mask]);
const size_t minor_ix = self->num_[key] & BLOCK_MASK;
self->buckets_[minor_ix + (key << BLOCK_BITS)] = (uint32_t)ix;
++self->num_[key];
HashLongestMatch* self = FN(Self)(handle);
uint16_t* num = FN(Num)(self);
const uint32_t key = FN(HashBytes)(&data[ix & mask], self->hash_shift_);
const size_t minor_ix = num[key] & self->block_mask_;
const size_t offset =
minor_ix + (key << GetHasherCommon(handle)->params.block_bits);
FN(Buckets)(self)[offset] = (uint32_t)ix;
++num[key];
}
static BROTLI_INLINE void FN(StoreRange)(HashLongestMatch* self,
const uint8_t *data, const size_t mask, const size_t ix_start,
static BROTLI_INLINE void FN(StoreRange)(HasherHandle handle,
const uint8_t* data, const size_t mask, const size_t ix_start,
const size_t ix_end) {
size_t i;
for (i = ix_start; i < ix_end; ++i) {
FN(Store)(self, data, mask, i);
FN(Store)(handle, data, mask, i);
}
}
static BROTLI_INLINE void FN(StitchToPreviousBlock)(HashLongestMatch* self,
static BROTLI_INLINE void FN(StitchToPreviousBlock)(HasherHandle handle,
size_t num_bytes, size_t position, const uint8_t* ringbuffer,
size_t ringbuffer_mask) {
if (num_bytes >= FN(HashTypeLength)() - 1 && position >= 3) {
/* Prepare the hashes for three last bytes of the last write.
These could not be calculated before, since they require knowledge
of both the previous and the current block. */
FN(Store)(self, ringbuffer, ringbuffer_mask, position - 3);
FN(Store)(self, ringbuffer, ringbuffer_mask, position - 2);
FN(Store)(self, ringbuffer, ringbuffer_mask, position - 1);
FN(Store)(handle, ringbuffer, ringbuffer_mask, position - 3);
FN(Store)(handle, ringbuffer, ringbuffer_mask, position - 2);
FN(Store)(handle, ringbuffer, ringbuffer_mask, position - 1);
}
}
static BROTLI_INLINE void FN(PrepareDistanceCache)(
HasherHandle handle, int* BROTLI_RESTRICT distance_cache) {
PrepareDistanceCache(distance_cache,
GetHasherCommon(handle)->params.num_last_distances_to_check);
}
/* Find a longest backward match of &data[cur_ix] up to the length of
max_length and stores the position cur_ix in the hash table.
REQUIRES: FN(PrepareDistanceCache) must be invoked for current distance cache
values; if this method is invoked repeatedly with the same distance
cache values, it is enough to invoke FN(PrepareDistanceCache) once.
Does not look for matches longer than max_length.
Does not look for matches further away than max_backward.
Writes the best match into |out|.
Returns true when match is found, otherwise false. */
static BROTLI_INLINE BROTLI_BOOL FN(FindLongestMatch)(HashLongestMatch* self,
|out|->score is updated only if a better match is found. */
static BROTLI_INLINE void FN(FindLongestMatch)(HasherHandle handle,
const BrotliEncoderDictionary* dictionary,
const uint8_t* BROTLI_RESTRICT data, const size_t ring_buffer_mask,
const int* BROTLI_RESTRICT distance_cache, const size_t cur_ix,
const size_t max_length, const size_t max_backward,
HasherSearchResult* BROTLI_RESTRICT out) {
const size_t max_length, const size_t max_backward, const size_t gap,
const size_t max_distance, HasherSearchResult* BROTLI_RESTRICT out) {
HasherCommon* common = GetHasherCommon(handle);
HashLongestMatch* self = FN(Self)(handle);
uint16_t* num = FN(Num)(self);
uint32_t* buckets = FN(Buckets)(self);
const size_t cur_ix_masked = cur_ix & ring_buffer_mask;
BROTLI_BOOL is_match_found = BROTLI_FALSE;
/* Don't accept a short copy from far away. */
score_t min_score = out->score;
score_t best_score = out->score;
size_t best_len = out->len;
size_t i;
out->len = 0;
out->len_x_code = 0;
out->len_code_delta = 0;
/* Try last distance first. */
for (i = 0; i < NUM_LAST_DISTANCES_TO_CHECK; ++i) {
const size_t idx = kDistanceCacheIndex[i];
const size_t backward =
(size_t)(distance_cache[idx] + kDistanceCacheOffset[i]);
for (i = 0; i < (size_t)common->params.num_last_distances_to_check; ++i) {
const size_t backward = (size_t)distance_cache[i];
size_t prev_ix = (size_t)(cur_ix - backward);
if (prev_ix >= cur_ix) {
continue;
}
if (PREDICT_FALSE(backward > max_backward)) {
if (BROTLI_PREDICT_FALSE(backward > max_backward)) {
continue;
}
prev_ix &= ring_buffer_mask;
@ -174,27 +193,31 @@ static BROTLI_INLINE BROTLI_BOOL FN(FindLongestMatch)(HashLongestMatch* self,
/* Comparing for >= 2 does not change the semantics, but just saves for
a few unnecessary binary logarithms in backward reference score,
since we are not interested in such short matches. */
score_t score = BackwardReferenceScoreUsingLastDistance(len, i);
score_t score = BackwardReferenceScoreUsingLastDistance(len);
if (best_score < score) {
best_score = score;
best_len = len;
out->len = best_len;
out->distance = backward;
out->score = best_score;
is_match_found = BROTLI_TRUE;
if (i != 0) score -= BackwardReferencePenaltyUsingLastDistance(i);
if (best_score < score) {
best_score = score;
best_len = len;
out->len = best_len;
out->distance = backward;
out->score = best_score;
}
}
}
}
}
{
const uint32_t key = FN(HashBytes)(&data[cur_ix_masked]);
uint32_t* BROTLI_RESTRICT bucket = &self->buckets_[key << BLOCK_BITS];
const uint32_t key =
FN(HashBytes)(&data[cur_ix_masked], self->hash_shift_);
uint32_t* BROTLI_RESTRICT bucket =
&buckets[key << common->params.block_bits];
const size_t down =
(self->num_[key] > BLOCK_SIZE) ? (self->num_[key] - BLOCK_SIZE) : 0u;
for (i = self->num_[key]; i > down;) {
size_t prev_ix = bucket[--i & BLOCK_MASK];
(num[key] > self->block_size_) ? (num[key] - self->block_size_) : 0u;
for (i = num[key]; i > down;) {
size_t prev_ix = bucket[--i & self->block_mask_];
const size_t backward = cur_ix - prev_ix;
if (PREDICT_FALSE(backward > max_backward)) {
if (BROTLI_PREDICT_FALSE(backward > max_backward)) {
break;
}
prev_ix &= ring_buffer_mask;
@ -218,24 +241,18 @@ static BROTLI_INLINE BROTLI_BOOL FN(FindLongestMatch)(HashLongestMatch* self,
out->len = best_len;
out->distance = backward;
out->score = best_score;
is_match_found = BROTLI_TRUE;
}
}
}
}
bucket[self->num_[key] & BLOCK_MASK] = (uint32_t)cur_ix;
++self->num_[key];
bucket[num[key] & self->block_mask_] = (uint32_t)cur_ix;
++num[key];
}
if (!is_match_found) {
is_match_found = SearchInStaticDictionary(&self->dict_search_stats_,
&data[cur_ix_masked], max_length, max_backward, out, BROTLI_FALSE);
if (min_score == out->score) {
SearchInStaticDictionary(dictionary,
handle, &data[cur_ix_masked], max_length, max_backward + gap,
max_distance, out, BROTLI_FALSE);
}
return is_match_found;
}
#undef HASH_MAP_SIZE
#undef BLOCK_MASK
#undef BLOCK_SIZE
#undef BUCKET_SIZE
#undef HashLongestMatch

View File

@ -5,7 +5,9 @@
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* template parameters: FN, BUCKET_BITS, BUCKET_SWEEP, USE_DICTIONARY */
/* template parameters: FN, BUCKET_BITS, BUCKET_SWEEP, HASH_LEN,
USE_DICTIONARY
*/
#define HashLongestMatchQuickly HASHER()
@ -19,10 +21,9 @@ static BROTLI_INLINE size_t FN(StoreLookahead)(void) { return 8; }
/* HashBytes is the function that chooses the bucket to place
the address in. The HashLongestMatch and HashLongestMatchQuickly
classes have separate, different implementations of hashing. */
static uint32_t FN(HashBytes)(const uint8_t *data) {
/* Computing a hash based on 5 bytes works much better for
qualities 1 and 3, where the next hash value is likely to replace */
uint64_t h = (BROTLI_UNALIGNED_LOAD64(data) << 24) * kHashMul32;
static uint32_t FN(HashBytes)(const uint8_t* data) {
const uint64_t h = ((BROTLI_UNALIGNED_LOAD64LE(data) << (64 - 8 * HASH_LEN)) *
kHashMul64);
/* The higher bits contain more mixture from the multiplication,
so we take our results from there. */
return (uint32_t)(h >> (64 - BUCKET_BITS));
@ -35,88 +36,86 @@ static uint32_t FN(HashBytes)(const uint8_t *data) {
given index, BUCKET_SWEEP buckets are used to store values of a key. */
typedef struct HashLongestMatchQuickly {
uint32_t buckets_[BUCKET_SIZE + BUCKET_SWEEP];
/* True if buckets_ array needs to be initialized. */
BROTLI_BOOL is_dirty_;
DictionarySearchStatictics dict_search_stats_;
} HashLongestMatchQuickly;
static void FN(Reset)(HashLongestMatchQuickly* self) {
self->is_dirty_ = BROTLI_TRUE;
DictionarySearchStaticticsReset(&self->dict_search_stats_);
static BROTLI_INLINE HashLongestMatchQuickly* FN(Self)(HasherHandle handle) {
return (HashLongestMatchQuickly*)&(GetHasherCommon(handle)[1]);
}
static void FN(InitEmpty)(HashLongestMatchQuickly* self) {
if (self->is_dirty_) {
static void FN(Initialize)(
HasherHandle handle, const BrotliEncoderParams* params) {
BROTLI_UNUSED(handle);
BROTLI_UNUSED(params);
}
static void FN(Prepare)(HasherHandle handle, BROTLI_BOOL one_shot,
size_t input_size, const uint8_t* data) {
HashLongestMatchQuickly* self = FN(Self)(handle);
/* Partial preparation is 100 times slower (per socket). */
size_t partial_prepare_threshold = HASH_MAP_SIZE >> 7;
if (one_shot && input_size <= partial_prepare_threshold) {
size_t i;
for (i = 0; i < input_size; ++i) {
const uint32_t key = FN(HashBytes)(&data[i]);
memset(&self->buckets_[key], 0, BUCKET_SWEEP * sizeof(self->buckets_[0]));
}
} else {
/* It is not strictly necessary to fill this buffer here, but
not filling will make the results of the compression stochastic
(but correct). This is because random data would cause the
system to find accidentally good backward references here and there. */
memset(&self->buckets_[0], 0, sizeof(self->buckets_));
self->is_dirty_ = BROTLI_FALSE;
}
}
static void FN(InitForData)(HashLongestMatchQuickly* self, const uint8_t* data,
size_t num) {
size_t i;
for (i = 0; i < num; ++i) {
const uint32_t key = FN(HashBytes)(&data[i]);
memset(&self->buckets_[key], 0, BUCKET_SWEEP * sizeof(self->buckets_[0]));
}
if (num != 0) {
self->is_dirty_ = BROTLI_FALSE;
}
}
static void FN(Init)(
MemoryManager* m, HashLongestMatchQuickly* self, const uint8_t* data,
const BrotliEncoderParams* params, size_t position, size_t bytes,
BROTLI_BOOL is_last) {
/* Choose which init method is faster.
Init() is about 100 times faster than InitForData(). */
const size_t kMaxBytesForPartialHashInit = HASH_MAP_SIZE >> 7;
BROTLI_UNUSED(m);
static BROTLI_INLINE size_t FN(HashMemAllocInBytes)(
const BrotliEncoderParams* params, BROTLI_BOOL one_shot,
size_t input_size) {
BROTLI_UNUSED(params);
if (position == 0 && is_last && bytes <= kMaxBytesForPartialHashInit) {
FN(InitForData)(self, data, bytes);
} else {
FN(InitEmpty)(self);
}
BROTLI_UNUSED(one_shot);
BROTLI_UNUSED(input_size);
return sizeof(HashLongestMatchQuickly);
}
/* Look at 5 bytes at &data[ix & mask].
Compute a hash from these, and store the value somewhere within
[ix .. ix+3]. */
static BROTLI_INLINE void FN(Store)(HashLongestMatchQuickly* self,
const uint8_t *data, const size_t mask, const size_t ix) {
static BROTLI_INLINE void FN(Store)(HasherHandle handle,
const uint8_t* data, const size_t mask, const size_t ix) {
const uint32_t key = FN(HashBytes)(&data[ix & mask]);
/* Wiggle the value with the bucket sweep range. */
const uint32_t off = (ix >> 3) % BUCKET_SWEEP;
self->buckets_[key + off] = (uint32_t)ix;
FN(Self)(handle)->buckets_[key + off] = (uint32_t)ix;
}
static BROTLI_INLINE void FN(StoreRange)(HashLongestMatchQuickly* self,
const uint8_t *data, const size_t mask, const size_t ix_start,
static BROTLI_INLINE void FN(StoreRange)(HasherHandle handle,
const uint8_t* data, const size_t mask, const size_t ix_start,
const size_t ix_end) {
size_t i;
for (i = ix_start; i < ix_end; ++i) {
FN(Store)(self, data, mask, i);
FN(Store)(handle, data, mask, i);
}
}
static BROTLI_INLINE void FN(StitchToPreviousBlock)(
HashLongestMatchQuickly* self, size_t num_bytes, size_t position,
HasherHandle handle, size_t num_bytes, size_t position,
const uint8_t* ringbuffer, size_t ringbuffer_mask) {
if (num_bytes >= FN(HashTypeLength)() - 1 && position >= 3) {
/* Prepare the hashes for three last bytes of the last write.
These could not be calculated before, since they require knowledge
of both the previous and the current block. */
FN(Store)(self, ringbuffer, ringbuffer_mask, position - 3);
FN(Store)(self, ringbuffer, ringbuffer_mask, position - 2);
FN(Store)(self, ringbuffer, ringbuffer_mask, position - 1);
FN(Store)(handle, ringbuffer, ringbuffer_mask, position - 3);
FN(Store)(handle, ringbuffer, ringbuffer_mask, position - 2);
FN(Store)(handle, ringbuffer, ringbuffer_mask, position - 1);
}
}
static BROTLI_INLINE void FN(PrepareDistanceCache)(
HasherHandle handle, int* BROTLI_RESTRICT distance_cache) {
BROTLI_UNUSED(handle);
BROTLI_UNUSED(distance_cache);
}
/* Find a longest backward match of &data[cur_ix & ring_buffer_mask]
up to the length of max_length and stores the position cur_ix in the
hash table.
@ -124,22 +123,25 @@ static BROTLI_INLINE void FN(StitchToPreviousBlock)(
Does not look for matches longer than max_length.
Does not look for matches further away than max_backward.
Writes the best match into |out|.
Returns true if match is found, otherwise false. */
static BROTLI_INLINE BROTLI_BOOL FN(FindLongestMatch)(
HashLongestMatchQuickly* self, const uint8_t* BROTLI_RESTRICT data,
|out|->score is updated only if a better match is found. */
static BROTLI_INLINE void FN(FindLongestMatch)(
HasherHandle handle, const BrotliEncoderDictionary* dictionary,
const uint8_t* BROTLI_RESTRICT data,
const size_t ring_buffer_mask, const int* BROTLI_RESTRICT distance_cache,
const size_t cur_ix, const size_t max_length, const size_t max_backward,
const size_t gap, const size_t max_distance,
HasherSearchResult* BROTLI_RESTRICT out) {
HashLongestMatchQuickly* self = FN(Self)(handle);
const size_t best_len_in = out->len;
const size_t cur_ix_masked = cur_ix & ring_buffer_mask;
const uint32_t key = FN(HashBytes)(&data[cur_ix_masked]);
int compare_char = data[cur_ix_masked + best_len_in];
score_t min_score = out->score;
score_t best_score = out->score;
size_t best_len = best_len_in;
size_t cached_backward = (size_t)distance_cache[0];
size_t prev_ix = cur_ix - cached_backward;
BROTLI_BOOL is_match_found = BROTLI_FALSE;
out->len_x_code = 0;
out->len_code_delta = 0;
if (prev_ix < cur_ix) {
prev_ix &= (uint32_t)ring_buffer_mask;
if (compare_char == data[prev_ix + best_len]) {
@ -147,17 +149,18 @@ static BROTLI_INLINE BROTLI_BOOL FN(FindLongestMatch)(
&data[cur_ix_masked],
max_length);
if (len >= 4) {
best_score = BackwardReferenceScoreUsingLastDistance(len, 0);
best_len = len;
out->len = len;
out->distance = cached_backward;
out->score = best_score;
compare_char = data[cur_ix_masked + best_len];
if (BUCKET_SWEEP == 1) {
self->buckets_[key] = (uint32_t)cur_ix;
return BROTLI_TRUE;
} else {
is_match_found = BROTLI_TRUE;
const score_t score = BackwardReferenceScoreUsingLastDistance(len);
if (best_score < score) {
best_score = score;
best_len = len;
out->len = len;
out->distance = cached_backward;
out->score = best_score;
compare_char = data[cur_ix_masked + best_len];
if (BUCKET_SWEEP == 1) {
self->buckets_[key] = (uint32_t)cur_ix;
return;
}
}
}
}
@ -171,22 +174,25 @@ static BROTLI_INLINE BROTLI_BOOL FN(FindLongestMatch)(
backward = cur_ix - prev_ix;
prev_ix &= (uint32_t)ring_buffer_mask;
if (compare_char != data[prev_ix + best_len_in]) {
return BROTLI_FALSE;
return;
}
if (PREDICT_FALSE(backward == 0 || backward > max_backward)) {
return BROTLI_FALSE;
if (BROTLI_PREDICT_FALSE(backward == 0 || backward > max_backward)) {
return;
}
len = FindMatchLengthWithLimit(&data[prev_ix],
&data[cur_ix_masked],
max_length);
if (len >= 4) {
out->len = len;
out->distance = backward;
out->score = BackwardReferenceScore(len, backward);
return BROTLI_TRUE;
const score_t score = BackwardReferenceScore(len, backward);
if (best_score < score) {
out->len = len;
out->distance = backward;
out->score = score;
return;
}
}
} else {
uint32_t *bucket = self->buckets_ + key;
uint32_t* bucket = self->buckets_ + key;
int i;
prev_ix = *bucket++;
for (i = 0; i < BUCKET_SWEEP; ++i, prev_ix = *bucket++) {
@ -196,7 +202,7 @@ static BROTLI_INLINE BROTLI_BOOL FN(FindLongestMatch)(
if (compare_char != data[prev_ix + best_len]) {
continue;
}
if (PREDICT_FALSE(backward == 0 || backward > max_backward)) {
if (BROTLI_PREDICT_FALSE(backward == 0 || backward > max_backward)) {
continue;
}
len = FindMatchLengthWithLimit(&data[prev_ix],
@ -211,17 +217,16 @@ static BROTLI_INLINE BROTLI_BOOL FN(FindLongestMatch)(
out->distance = backward;
out->score = score;
compare_char = data[cur_ix_masked + best_len];
is_match_found = BROTLI_TRUE;
}
}
}
}
if (USE_DICTIONARY && !is_match_found) {
is_match_found = SearchInStaticDictionary(&self->dict_search_stats_,
&data[cur_ix_masked], max_length, max_backward, out, BROTLI_TRUE);
if (USE_DICTIONARY && min_score == out->score) {
SearchInStaticDictionary(dictionary,
handle, &data[cur_ix_masked], max_length, max_backward + gap,
max_distance, out, BROTLI_TRUE);
}
self->buckets_[key + ((cur_ix >> 3) % BUCKET_SWEEP)] = (uint32_t)cur_ix;
return is_match_found;
}
#undef HASH_MAP_SIZE

View File

@ -0,0 +1,215 @@
/* NOLINT(build/header_guard) */
/* Copyright 2018 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* template parameters: FN, JUMP, NUMBUCKETS, MASK, CHUNKLEN */
/* NUMBUCKETS / (MASK + 1) = probability of storing and using hash code. */
/* JUMP = skip bytes for speedup */
/* Rolling hash for long distance long string matches. Stores one position
per bucket, bucket key is computed over a long region. */
#define HashRolling HASHER()
static const uint32_t FN(kRollingHashMul32) = 69069;
static const uint32_t FN(kInvalidPos) = 0xffffffff;
/* This hasher uses a longer forward length, but returning a higher value here
will hurt compression by the main hasher when combined with a composite
hasher. The hasher tests for forward itself instead. */
static BROTLI_INLINE size_t FN(HashTypeLength)(void) { return 4; }
static BROTLI_INLINE size_t FN(StoreLookahead)(void) { return 4; }
/* Computes a code from a single byte. A lookup table of 256 values could be
used, but simply adding 1 works about as good. */
static uint32_t FN(HashByte)(uint8_t byte) {
return (uint32_t)byte + 1u;
}
static uint32_t FN(HashRollingFunctionInitial)(uint32_t state, uint8_t add,
uint32_t factor) {
return (uint32_t)(factor * state + FN(HashByte)(add));
}
static uint32_t FN(HashRollingFunction)(uint32_t state, uint8_t add,
uint8_t rem, uint32_t factor,
uint32_t factor_remove) {
return (uint32_t)(factor * state +
FN(HashByte)(add) - factor_remove * FN(HashByte)(rem));
}
typedef struct HashRolling {
uint32_t state;
uint32_t* table;
size_t next_ix;
uint32_t chunk_len;
uint32_t factor;
uint32_t factor_remove;
} HashRolling;
static BROTLI_INLINE HashRolling* FN(Self)(HasherHandle handle) {
return (HashRolling*)&(GetHasherCommon(handle)[1]);
}
static void FN(Initialize)(
HasherHandle handle, const BrotliEncoderParams* params) {
HashRolling* self = FN(Self)(handle);
size_t i;
self->state = 0;
self->next_ix = 0;
self->factor = FN(kRollingHashMul32);
/* Compute the factor of the oldest byte to remove: factor**steps modulo
0xffffffff (the multiplications rely on 32-bit overflow) */
self->factor_remove = 1;
for (i = 0; i < CHUNKLEN; i += JUMP) {
self->factor_remove *= self->factor;
}
self->table = (uint32_t*)((HasherHandle)self + sizeof(HashRolling));
for (i = 0; i < NUMBUCKETS; i++) {
self->table[i] = FN(kInvalidPos);
}
BROTLI_UNUSED(params);
}
static void FN(Prepare)(HasherHandle handle, BROTLI_BOOL one_shot,
size_t input_size, const uint8_t* data) {
HashRolling* self = FN(Self)(handle);
size_t i;
/* Too small size, cannot use this hasher. */
if (input_size < CHUNKLEN) return;
self->state = 0;
for (i = 0; i < CHUNKLEN; i += JUMP) {
self->state = FN(HashRollingFunctionInitial)(
self->state, data[i], self->factor);
}
BROTLI_UNUSED(one_shot);
}
static BROTLI_INLINE size_t FN(HashMemAllocInBytes)(
const BrotliEncoderParams* params, BROTLI_BOOL one_shot,
size_t input_size) {
return sizeof(HashRolling) + NUMBUCKETS * sizeof(uint32_t);
BROTLI_UNUSED(params);
BROTLI_UNUSED(one_shot);
BROTLI_UNUSED(input_size);
}
static BROTLI_INLINE void FN(Store)(HasherHandle BROTLI_RESTRICT handle,
const uint8_t* BROTLI_RESTRICT data, const size_t mask, const size_t ix) {
BROTLI_UNUSED(handle);
BROTLI_UNUSED(data);
BROTLI_UNUSED(mask);
BROTLI_UNUSED(ix);
}
static BROTLI_INLINE void FN(StoreRange)(HasherHandle handle,
const uint8_t* data, const size_t mask, const size_t ix_start,
const size_t ix_end) {
BROTLI_UNUSED(handle);
BROTLI_UNUSED(data);
BROTLI_UNUSED(mask);
BROTLI_UNUSED(ix_start);
BROTLI_UNUSED(ix_end);
}
static BROTLI_INLINE void FN(StitchToPreviousBlock)(HasherHandle handle,
size_t num_bytes, size_t position, const uint8_t* ringbuffer,
size_t ring_buffer_mask) {
/* In this case we must re-initialize the hasher from scratch from the
current position. */
HashRolling* self = FN(Self)(handle);
size_t position_masked;
size_t available = num_bytes;
if ((position & (JUMP - 1)) != 0) {
size_t diff = JUMP - (position & (JUMP - 1));
available = (diff > available) ? 0 : (available - diff);
position += diff;
}
position_masked = position & ring_buffer_mask;
/* wrapping around ringbuffer not handled. */
if (available > ring_buffer_mask - position_masked) {
available = ring_buffer_mask - position_masked;
}
FN(Prepare)(handle, BROTLI_FALSE, available,
ringbuffer + (position & ring_buffer_mask));
self->next_ix = position;
BROTLI_UNUSED(num_bytes);
}
static BROTLI_INLINE void FN(PrepareDistanceCache)(
HasherHandle handle, int* BROTLI_RESTRICT distance_cache) {
BROTLI_UNUSED(handle);
BROTLI_UNUSED(distance_cache);
}
static BROTLI_INLINE void FN(FindLongestMatch)(HasherHandle handle,
const BrotliEncoderDictionary* dictionary,
const uint8_t* BROTLI_RESTRICT data, const size_t ring_buffer_mask,
const int* BROTLI_RESTRICT distance_cache, const size_t cur_ix,
const size_t max_length, const size_t max_backward, const size_t gap,
const size_t max_distance, HasherSearchResult* BROTLI_RESTRICT out) {
HashRolling* self = FN(Self)(handle);
const size_t cur_ix_masked = cur_ix & ring_buffer_mask;
size_t pos = self->next_ix;
if ((cur_ix & (JUMP - 1)) != 0) return;
/* Not enough lookahead */
if (max_length < CHUNKLEN) return;
for (pos = self->next_ix; pos <= cur_ix; pos += JUMP) {
uint32_t code = self->state & MASK;
uint8_t rem = data[pos & ring_buffer_mask];
uint8_t add = data[(pos + CHUNKLEN) & ring_buffer_mask];
size_t found_ix = FN(kInvalidPos);
self->state = FN(HashRollingFunction)(
self->state, add, rem, self->factor, self->factor_remove);
if (code < NUMBUCKETS) {
found_ix = self->table[code];
self->table[code] = (uint32_t)pos;
if (pos == cur_ix && found_ix != FN(kInvalidPos)) {
/* The cast to 32-bit makes backward distances up to 4GB work even
if cur_ix is above 4GB, despite using 32-bit values in the table. */
size_t backward = (uint32_t)(cur_ix - found_ix);
if (backward <= max_backward) {
const size_t found_ix_masked = found_ix & ring_buffer_mask;
const size_t len = FindMatchLengthWithLimit(&data[found_ix_masked],
&data[cur_ix_masked],
max_length);
if (len >= 4 && len > out->len) {
score_t score = BackwardReferenceScore(len, backward);
if (score > out->score) {
out->len = len;
out->distance = backward;
out->score = score;
out->len_code_delta = 0;
}
}
}
}
}
}
self->next_ix = cur_ix + JUMP;
/* NOTE: this hasher does not search in the dictionary. It is used as
backup-hasher, the main hasher already searches in it. */
BROTLI_UNUSED(dictionary);
BROTLI_UNUSED(distance_cache);
BROTLI_UNUSED(gap);
BROTLI_UNUSED(max_distance);
}
#undef HashRolling

View File

@ -0,0 +1,327 @@
/* NOLINT(build/header_guard) */
/* Copyright 2016 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* template parameters: FN, BUCKET_BITS, MAX_TREE_COMP_LENGTH,
MAX_TREE_SEARCH_DEPTH */
/* A (forgetful) hash table where each hash bucket contains a binary tree of
sequences whose first 4 bytes share the same hash code.
Each sequence is MAX_TREE_COMP_LENGTH long and is identified by its starting
position in the input data. The binary tree is sorted by the lexicographic
order of the sequences, and it is also a max-heap with respect to the
starting positions. */
#define HashToBinaryTree HASHER()
#define BUCKET_SIZE (1 << BUCKET_BITS)
static BROTLI_INLINE size_t FN(HashTypeLength)(void) { return 4; }
static BROTLI_INLINE size_t FN(StoreLookahead)(void) {
return MAX_TREE_COMP_LENGTH;
}
static uint32_t FN(HashBytes)(const uint8_t* data) {
uint32_t h = BROTLI_UNALIGNED_LOAD32LE(data) * kHashMul32;
/* The higher bits contain more mixture from the multiplication,
so we take our results from there. */
return h >> (32 - BUCKET_BITS);
}
typedef struct HashToBinaryTree {
/* The window size minus 1 */
size_t window_mask_;
/* Hash table that maps the 4-byte hashes of the sequence to the last
position where this hash was found, which is the root of the binary
tree of sequences that share this hash bucket. */
uint32_t buckets_[BUCKET_SIZE];
/* A position used to mark a non-existent sequence, i.e. a tree is empty if
its root is at invalid_pos_ and a node is a leaf if both its children
are at invalid_pos_. */
uint32_t invalid_pos_;
/* --- Dynamic size members --- */
/* The union of the binary trees of each hash bucket. The root of the tree
corresponding to a hash is a sequence starting at buckets_[hash] and
the left and right children of a sequence starting at pos are
forest_[2 * pos] and forest_[2 * pos + 1]. */
/* uint32_t forest[2 * num_nodes] */
} HashToBinaryTree;
static BROTLI_INLINE HashToBinaryTree* FN(Self)(HasherHandle handle) {
return (HashToBinaryTree*)&(GetHasherCommon(handle)[1]);
}
static BROTLI_INLINE uint32_t* FN(Forest)(HashToBinaryTree* self) {
return (uint32_t*)(&self[1]);
}
static void FN(Initialize)(
HasherHandle handle, const BrotliEncoderParams* params) {
HashToBinaryTree* self = FN(Self)(handle);
self->window_mask_ = (1u << params->lgwin) - 1u;
self->invalid_pos_ = (uint32_t)(0 - self->window_mask_);
}
static void FN(Prepare)(HasherHandle handle, BROTLI_BOOL one_shot,
size_t input_size, const uint8_t* data) {
HashToBinaryTree* self = FN(Self)(handle);
uint32_t invalid_pos = self->invalid_pos_;
uint32_t i;
BROTLI_UNUSED(data);
BROTLI_UNUSED(one_shot);
BROTLI_UNUSED(input_size);
for (i = 0; i < BUCKET_SIZE; i++) {
self->buckets_[i] = invalid_pos;
}
}
static BROTLI_INLINE size_t FN(HashMemAllocInBytes)(
const BrotliEncoderParams* params, BROTLI_BOOL one_shot,
size_t input_size) {
size_t num_nodes = (size_t)1 << params->lgwin;
if (one_shot && input_size < num_nodes) {
num_nodes = input_size;
}
return sizeof(HashToBinaryTree) + 2 * sizeof(uint32_t) * num_nodes;
}
static BROTLI_INLINE size_t FN(LeftChildIndex)(HashToBinaryTree* self,
const size_t pos) {
return 2 * (pos & self->window_mask_);
}
static BROTLI_INLINE size_t FN(RightChildIndex)(HashToBinaryTree* self,
const size_t pos) {
return 2 * (pos & self->window_mask_) + 1;
}
/* Stores the hash of the next 4 bytes and in a single tree-traversal, the
hash bucket's binary tree is searched for matches and is re-rooted at the
current position.
If less than MAX_TREE_COMP_LENGTH data is available, the hash bucket of the
current position is searched for matches, but the state of the hash table
is not changed, since we can not know the final sorting order of the
current (incomplete) sequence.
This function must be called with increasing cur_ix positions. */
static BROTLI_INLINE BackwardMatch* FN(StoreAndFindMatches)(
HashToBinaryTree* self, const uint8_t* const BROTLI_RESTRICT data,
const size_t cur_ix, const size_t ring_buffer_mask, const size_t max_length,
const size_t max_backward, size_t* const BROTLI_RESTRICT best_len,
BackwardMatch* BROTLI_RESTRICT matches) {
const size_t cur_ix_masked = cur_ix & ring_buffer_mask;
const size_t max_comp_len =
BROTLI_MIN(size_t, max_length, MAX_TREE_COMP_LENGTH);
const BROTLI_BOOL should_reroot_tree =
TO_BROTLI_BOOL(max_length >= MAX_TREE_COMP_LENGTH);
const uint32_t key = FN(HashBytes)(&data[cur_ix_masked]);
uint32_t* forest = FN(Forest)(self);
size_t prev_ix = self->buckets_[key];
/* The forest index of the rightmost node of the left subtree of the new
root, updated as we traverse and re-root the tree of the hash bucket. */
size_t node_left = FN(LeftChildIndex)(self, cur_ix);
/* The forest index of the leftmost node of the right subtree of the new
root, updated as we traverse and re-root the tree of the hash bucket. */
size_t node_right = FN(RightChildIndex)(self, cur_ix);
/* The match length of the rightmost node of the left subtree of the new
root, updated as we traverse and re-root the tree of the hash bucket. */
size_t best_len_left = 0;
/* The match length of the leftmost node of the right subtree of the new
root, updated as we traverse and re-root the tree of the hash bucket. */
size_t best_len_right = 0;
size_t depth_remaining;
if (should_reroot_tree) {
self->buckets_[key] = (uint32_t)cur_ix;
}
for (depth_remaining = MAX_TREE_SEARCH_DEPTH; ; --depth_remaining) {
const size_t backward = cur_ix - prev_ix;
const size_t prev_ix_masked = prev_ix & ring_buffer_mask;
if (backward == 0 || backward > max_backward || depth_remaining == 0) {
if (should_reroot_tree) {
forest[node_left] = self->invalid_pos_;
forest[node_right] = self->invalid_pos_;
}
break;
}
{
const size_t cur_len = BROTLI_MIN(size_t, best_len_left, best_len_right);
size_t len;
BROTLI_DCHECK(cur_len <= MAX_TREE_COMP_LENGTH);
len = cur_len +
FindMatchLengthWithLimit(&data[cur_ix_masked + cur_len],
&data[prev_ix_masked + cur_len],
max_length - cur_len);
BROTLI_DCHECK(
0 == memcmp(&data[cur_ix_masked], &data[prev_ix_masked], len));
if (matches && len > *best_len) {
*best_len = len;
InitBackwardMatch(matches++, backward, len);
}
if (len >= max_comp_len) {
if (should_reroot_tree) {
forest[node_left] = forest[FN(LeftChildIndex)(self, prev_ix)];
forest[node_right] = forest[FN(RightChildIndex)(self, prev_ix)];
}
break;
}
if (data[cur_ix_masked + len] > data[prev_ix_masked + len]) {
best_len_left = len;
if (should_reroot_tree) {
forest[node_left] = (uint32_t)prev_ix;
}
node_left = FN(RightChildIndex)(self, prev_ix);
prev_ix = forest[node_left];
} else {
best_len_right = len;
if (should_reroot_tree) {
forest[node_right] = (uint32_t)prev_ix;
}
node_right = FN(LeftChildIndex)(self, prev_ix);
prev_ix = forest[node_right];
}
}
}
return matches;
}
/* Finds all backward matches of &data[cur_ix & ring_buffer_mask] up to the
length of max_length and stores the position cur_ix in the hash table.
Sets *num_matches to the number of matches found, and stores the found
matches in matches[0] to matches[*num_matches - 1]. The matches will be
sorted by strictly increasing length and (non-strictly) increasing
distance. */
static BROTLI_INLINE size_t FN(FindAllMatches)(HasherHandle handle,
const BrotliEncoderDictionary* dictionary, const uint8_t* data,
const size_t ring_buffer_mask, const size_t cur_ix,
const size_t max_length, const size_t max_backward, const size_t gap,
const BrotliEncoderParams* params, BackwardMatch* matches) {
BackwardMatch* const orig_matches = matches;
const size_t cur_ix_masked = cur_ix & ring_buffer_mask;
size_t best_len = 1;
const size_t short_match_max_backward =
params->quality != HQ_ZOPFLIFICATION_QUALITY ? 16 : 64;
size_t stop = cur_ix - short_match_max_backward;
uint32_t dict_matches[BROTLI_MAX_STATIC_DICTIONARY_MATCH_LEN + 1];
size_t i;
if (cur_ix < short_match_max_backward) { stop = 0; }
for (i = cur_ix - 1; i > stop && best_len <= 2; --i) {
size_t prev_ix = i;
const size_t backward = cur_ix - prev_ix;
if (BROTLI_PREDICT_FALSE(backward > max_backward)) {
break;
}
prev_ix &= ring_buffer_mask;
if (data[cur_ix_masked] != data[prev_ix] ||
data[cur_ix_masked + 1] != data[prev_ix + 1]) {
continue;
}
{
const size_t len =
FindMatchLengthWithLimit(&data[prev_ix], &data[cur_ix_masked],
max_length);
if (len > best_len) {
best_len = len;
InitBackwardMatch(matches++, backward, len);
}
}
}
if (best_len < max_length) {
matches = FN(StoreAndFindMatches)(FN(Self)(handle), data, cur_ix,
ring_buffer_mask, max_length, max_backward, &best_len, matches);
}
for (i = 0; i <= BROTLI_MAX_STATIC_DICTIONARY_MATCH_LEN; ++i) {
dict_matches[i] = kInvalidMatch;
}
{
size_t minlen = BROTLI_MAX(size_t, 4, best_len + 1);
if (BrotliFindAllStaticDictionaryMatches(dictionary,
&data[cur_ix_masked], minlen, max_length, &dict_matches[0])) {
size_t maxlen = BROTLI_MIN(
size_t, BROTLI_MAX_STATIC_DICTIONARY_MATCH_LEN, max_length);
size_t l;
for (l = minlen; l <= maxlen; ++l) {
uint32_t dict_id = dict_matches[l];
if (dict_id < kInvalidMatch) {
size_t distance = max_backward + gap + (dict_id >> 5) + 1;
if (distance <= params->dist.max_distance) {
InitDictionaryBackwardMatch(matches++, distance, l, dict_id & 31);
}
}
}
}
}
return (size_t)(matches - orig_matches);
}
/* Stores the hash of the next 4 bytes and re-roots the binary tree at the
current sequence, without returning any matches.
REQUIRES: ix + MAX_TREE_COMP_LENGTH <= end-of-current-block */
static BROTLI_INLINE void FN(Store)(HasherHandle handle, const uint8_t* data,
const size_t mask, const size_t ix) {
HashToBinaryTree* self = FN(Self)(handle);
/* Maximum distance is window size - 16, see section 9.1. of the spec. */
const size_t max_backward = self->window_mask_ - BROTLI_WINDOW_GAP + 1;
FN(StoreAndFindMatches)(self, data, ix, mask, MAX_TREE_COMP_LENGTH,
max_backward, NULL, NULL);
}
static BROTLI_INLINE void FN(StoreRange)(HasherHandle handle,
const uint8_t* data, const size_t mask, const size_t ix_start,
const size_t ix_end) {
size_t i = ix_start;
size_t j = ix_start;
if (ix_start + 63 <= ix_end) {
i = ix_end - 63;
}
if (ix_start + 512 <= i) {
for (; j < i; j += 8) {
FN(Store)(handle, data, mask, j);
}
}
for (; i < ix_end; ++i) {
FN(Store)(handle, data, mask, i);
}
}
static BROTLI_INLINE void FN(StitchToPreviousBlock)(HasherHandle handle,
size_t num_bytes, size_t position, const uint8_t* ringbuffer,
size_t ringbuffer_mask) {
HashToBinaryTree* self = FN(Self)(handle);
if (num_bytes >= FN(HashTypeLength)() - 1 &&
position >= MAX_TREE_COMP_LENGTH) {
/* Store the last `MAX_TREE_COMP_LENGTH - 1` positions in the hasher.
These could not be calculated before, since they require knowledge
of both the previous and the current block. */
const size_t i_start = position - MAX_TREE_COMP_LENGTH + 1;
const size_t i_end = BROTLI_MIN(size_t, position, i_start + num_bytes);
size_t i;
for (i = i_start; i < i_end; ++i) {
/* Maximum distance is window size - 16, see section 9.1. of the spec.
Furthermore, we have to make sure that we don't look further back
from the start of the next block than the window size, otherwise we
could access already overwritten areas of the ring-buffer. */
const size_t max_backward =
self->window_mask_ - BROTLI_MAX(size_t,
BROTLI_WINDOW_GAP - 1,
position - i);
/* We know that i + MAX_TREE_COMP_LENGTH <= position + num_bytes, i.e. the
end of the current block and that we have at least
MAX_TREE_COMP_LENGTH tail in the ring-buffer. */
FN(StoreAndFindMatches)(self, ringbuffer, i, ringbuffer_mask,
MAX_TREE_COMP_LENGTH, max_backward, NULL, NULL);
}
}
}
#undef BUCKET_SIZE
#undef HashToBinaryTree

View File

@ -8,9 +8,9 @@
#include "./histogram.h"
#include "../common/context.h"
#include "./block_splitter.h"
#include "./command.h"
#include "./context.h"
#if defined(__cplusplus) || defined(c_plusplus)
extern "C" {
@ -63,11 +63,16 @@ void BrotliBuildHistogramsWithContext(
BlockSplitIteratorNext(&insert_and_copy_it);
HistogramAddCommand(&insert_and_copy_histograms[insert_and_copy_it.type_],
cmd->cmd_prefix_);
/* TODO: unwrap iterator blocks. */
for (j = cmd->insert_len_; j != 0; --j) {
size_t context;
BlockSplitIteratorNext(&literal_it);
context = (literal_it.type_ << BROTLI_LITERAL_CONTEXT_BITS) +
Context(prev_byte, prev_byte2, context_modes[literal_it.type_]);
context = literal_it.type_;
if (context_modes) {
ContextLut lut = BROTLI_CONTEXT_LUT(context_modes[context]);
context = (context << BROTLI_LITERAL_CONTEXT_BITS) +
BROTLI_CONTEXT(prev_byte, prev_byte2, lut);
}
HistogramAddLiteral(&literal_histograms[context],
ringbuffer[pos & mask]);
prev_byte2 = prev_byte;
@ -84,7 +89,7 @@ void BrotliBuildHistogramsWithContext(
context = (dist_it.type_ << BROTLI_DISTANCE_CONTEXT_BITS) +
CommandDistanceContext(cmd);
HistogramAddDistance(&copy_dist_histograms[context],
cmd->dist_prefix_);
cmd->dist_prefix_ & 0x3FF);
}
}
}

View File

@ -12,16 +12,19 @@
#include <string.h> /* memset */
#include "../common/constants.h"
#include "../common/types.h"
#include "../common/context.h"
#include "../common/platform.h"
#include <brotli/types.h>
#include "./block_splitter.h"
#include "./command.h"
#include "./context.h"
#include "./port.h"
#if defined(__cplusplus) || defined(c_plusplus)
extern "C" {
#endif
/* The distance symbols effectively used by "Large Window Brotli" (32-bit). */
#define BROTLI_NUM_HISTOGRAM_DISTANCE_SYMBOLS 544
#define FN(X) X ## Literal
#define DATA_SIZE BROTLI_NUM_LITERAL_SYMBOLS
#define DataType uint8_t
@ -38,7 +41,7 @@ extern "C" {
#undef FN
#define FN(X) X ## Distance
#define DATA_SIZE BROTLI_NUM_DISTANCE_SYMBOLS
#define DATA_SIZE BROTLI_NUM_HISTOGRAM_DISTANCE_SYMBOLS
#include "./histogram_inc.h" /* NOLINT(build/include) */
#undef DataType
#undef DATA_SIZE

View File

@ -33,7 +33,7 @@ static BROTLI_INLINE void FN(HistogramAdd)(FN(Histogram)* self, size_t val) {
}
static BROTLI_INLINE void FN(HistogramAddVector)(FN(Histogram)* self,
const DataType *p, size_t n) {
const DataType* p, size_t n) {
self->total_count_ += n;
n += 1;
while (--n) ++self->data_[*p++];

View File

@ -9,9 +9,9 @@
#include "./literal_cost.h"
#include "../common/types.h"
#include "../common/platform.h"
#include <brotli/types.h>
#include "./fast_log.h"
#include "./port.h"
#include "./utf8_util.h"
#if defined(__cplusplus) || defined(c_plusplus)
@ -25,7 +25,7 @@ static size_t UTF8Position(size_t last, size_t c, size_t clamp) {
return BROTLI_MIN(size_t, 1, clamp);
} else {
/* Let's decide over the last byte if this ends the sequence. */
if (last < 0xe0) {
if (last < 0xE0) {
return 0; /* Completed two or three byte coding. */
} else { /* Next one is the 'Byte 3' of utf-8 encoding. */
return BROTLI_MIN(size_t, 2, clamp);
@ -34,16 +34,14 @@ static size_t UTF8Position(size_t last, size_t c, size_t clamp) {
}
static size_t DecideMultiByteStatsLevel(size_t pos, size_t len, size_t mask,
const uint8_t *data) {
const uint8_t* data) {
size_t counts[3] = { 0 };
size_t max_utf8 = 1; /* should be 2, but 1 compresses better. */
size_t last_c = 0;
size_t utf8_pos = 0;
size_t i;
for (i = 0; i < len; ++i) {
size_t c = data[(pos + i) & mask];
utf8_pos = UTF8Position(last_c, c, 2);
++counts[utf8_pos];
++counts[UTF8Position(last_c, c, 2)];
last_c = c;
}
if (counts[2] < 500) {
@ -56,16 +54,15 @@ static size_t DecideMultiByteStatsLevel(size_t pos, size_t len, size_t mask,
}
static void EstimateBitCostsForLiteralsUTF8(size_t pos, size_t len, size_t mask,
const uint8_t *data, float *cost) {
/* max_utf8 is 0 (normal ascii single byte modeling),
1 (for 2-byte utf-8 modeling), or 2 (for 3-byte utf-8 modeling). */
const uint8_t* data, float* cost) {
/* max_utf8 is 0 (normal ASCII single byte modeling),
1 (for 2-byte UTF-8 modeling), or 2 (for 3-byte UTF-8 modeling). */
const size_t max_utf8 = DecideMultiByteStatsLevel(pos, len, mask, data);
size_t histogram[3][256] = { { 0 } };
size_t window_half = 495;
size_t in_window = BROTLI_MIN(size_t, window_half, len);
size_t in_window_utf8[3] = { 0 };
size_t i;
{ /* Bootstrap histograms. */
size_t last_c = 0;
@ -128,7 +125,7 @@ static void EstimateBitCostsForLiteralsUTF8(size_t pos, size_t len, size_t mask,
}
void BrotliEstimateBitCostsForLiterals(size_t pos, size_t len, size_t mask,
const uint8_t *data, float *cost) {
const uint8_t* data, float* cost) {
if (BrotliIsMostlyUTF8(data, pos, mask, len, kMinUTF8Ratio)) {
EstimateBitCostsForLiteralsUTF8(pos, len, mask, data, cost);
return;

View File

@ -10,18 +10,18 @@
#ifndef BROTLI_ENC_LITERAL_COST_H_
#define BROTLI_ENC_LITERAL_COST_H_
#include "../common/types.h"
#include "./port.h"
#include "../common/platform.h"
#include <brotli/types.h>
#if defined(__cplusplus) || defined(c_plusplus)
extern "C" {
#endif
/* Estimates how many bits the literals in the interval [pos, pos + len) in the
ringbuffer (data, mask) will take entropy coded and writes these estimates
ring-buffer (data, mask) will take entropy coded and writes these estimates
to the cost[0..len) array. */
BROTLI_INTERNAL void BrotliEstimateBitCostsForLiterals(
size_t pos, size_t len, size_t mask, const uint8_t *data, float *cost);
size_t pos, size_t len, size_t mask, const uint8_t* data, float* cost);
#if defined(__cplusplus) || defined(c_plusplus)
} /* extern "C" */

View File

@ -9,12 +9,11 @@
#include "./memory.h"
#include <assert.h>
#include <stdlib.h> /* exit, free, malloc */
#include <string.h> /* memcpy */
#include "../common/types.h"
#include "./port.h"
#include "../common/platform.h"
#include <brotli/types.h>
#if defined(__cplusplus) || defined(c_plusplus)
extern "C" {
@ -28,22 +27,12 @@ extern "C" {
#define NEW_ALLOCATED_OFFSET MAX_PERM_ALLOCATED
#define NEW_FREED_OFFSET (MAX_PERM_ALLOCATED + MAX_NEW_ALLOCATED)
static void* DefaultAllocFunc(void* opaque, size_t size) {
BROTLI_UNUSED(opaque);
return malloc(size);
}
static void DefaultFreeFunc(void* opaque, void* address) {
BROTLI_UNUSED(opaque);
free(address);
}
void BrotliInitMemoryManager(
MemoryManager* m, brotli_alloc_func alloc_func, brotli_free_func free_func,
void* opaque) {
if (!alloc_func) {
m->alloc_func = DefaultAllocFunc;
m->free_func = DefaultFreeFunc;
m->alloc_func = BrotliDefaultAllocFunc;
m->free_func = BrotliDefaultFreeFunc;
m->opaque = 0;
} else {
m->alloc_func = alloc_func;
@ -132,11 +121,11 @@ static void CollectGarbagePointers(MemoryManager* m) {
m->pointers + NEW_FREED_OFFSET, m->new_freed);
m->perm_allocated -= annihilated;
m->new_freed -= annihilated;
assert(m->new_freed == 0);
BROTLI_DCHECK(m->new_freed == 0);
}
if (m->new_allocated != 0) {
assert(m->perm_allocated + m->new_allocated <= MAX_PERM_ALLOCATED);
BROTLI_DCHECK(m->perm_allocated + m->new_allocated <= MAX_PERM_ALLOCATED);
memcpy(m->pointers + PERM_ALLOCATED_OFFSET + m->perm_allocated,
m->pointers + NEW_ALLOCATED_OFFSET,
sizeof(void*) * m->new_allocated);

View File

@ -9,8 +9,10 @@
#ifndef BROTLI_ENC_MEMORY_H_
#define BROTLI_ENC_MEMORY_H_
#include "../common/types.h"
#include "./port.h"
#include <string.h> /* memcpy */
#include "../common/platform.h"
#include <brotli/types.h>
#if defined(__cplusplus) || defined(c_plusplus)
extern "C" {
@ -39,7 +41,8 @@ BROTLI_INTERNAL void BrotliInitMemoryManager(
void* opaque);
BROTLI_INTERNAL void* BrotliAllocate(MemoryManager* m, size_t n);
#define BROTLI_ALLOC(M, T, N) ((T*)BrotliAllocate((M), (N) * sizeof(T)))
#define BROTLI_ALLOC(M, T, N) \
((N) > 0 ? ((T*)BrotliAllocate((M), (N) * sizeof(T))) : NULL)
BROTLI_INTERNAL void BrotliFree(MemoryManager* m, void* p);
#define BROTLI_FREE(M, P) { \
@ -55,6 +58,43 @@ BROTLI_INTERNAL void BrotliFree(MemoryManager* m, void* p);
BROTLI_INTERNAL void BrotliWipeOutMemoryManager(MemoryManager* m);
/*
Dynamically grows array capacity to at least the requested size
M: MemoryManager
T: data type
A: array
C: capacity
R: requested size
*/
#define BROTLI_ENSURE_CAPACITY(M, T, A, C, R) { \
if (C < (R)) { \
size_t _new_size = (C == 0) ? (R) : C; \
T* new_array; \
while (_new_size < (R)) _new_size *= 2; \
new_array = BROTLI_ALLOC((M), T, _new_size); \
if (!BROTLI_IS_OOM(M) && C != 0) \
memcpy(new_array, A, C * sizeof(T)); \
BROTLI_FREE((M), A); \
A = new_array; \
C = _new_size; \
} \
}
/*
Appends value and dynamically grows array capacity when needed
M: MemoryManager
T: data type
A: array
C: array capacity
S: array size
V: value to append
*/
#define BROTLI_ENSURE_CAPACITY_APPEND(M, T, A, C, S, V) { \
(S)++; \
BROTLI_ENSURE_CAPACITY(M, T, A, C, S); \
A[(S) - 1] = (V); \
}
#if defined(__cplusplus) || defined(c_plusplus)
} /* extern "C" */
#endif

View File

@ -10,29 +10,131 @@
#include "./metablock.h"
#include "../common/constants.h"
#include "../common/types.h"
#include "../common/context.h"
#include "../common/platform.h"
#include <brotli/types.h>
#include "./bit_cost.h"
#include "./block_splitter.h"
#include "./cluster.h"
#include "./context.h"
#include "./entropy_encode.h"
#include "./histogram.h"
#include "./memory.h"
#include "./port.h"
#include "./quality.h"
#if defined(__cplusplus) || defined(c_plusplus)
extern "C" {
#endif
void BrotliInitDistanceParams(BrotliEncoderParams* params,
uint32_t npostfix, uint32_t ndirect) {
BrotliDistanceParams* dist_params = &params->dist;
uint32_t alphabet_size, max_distance;
dist_params->distance_postfix_bits = npostfix;
dist_params->num_direct_distance_codes = ndirect;
alphabet_size = BROTLI_DISTANCE_ALPHABET_SIZE(
npostfix, ndirect, BROTLI_MAX_DISTANCE_BITS);
max_distance = ndirect + (1U << (BROTLI_MAX_DISTANCE_BITS + npostfix + 2)) -
(1U << (npostfix + 2));
if (params->large_window) {
static const uint32_t bound[BROTLI_MAX_NPOSTFIX + 1] = {0, 4, 12, 28};
uint32_t postfix = 1U << npostfix;
alphabet_size = BROTLI_DISTANCE_ALPHABET_SIZE(
npostfix, ndirect, BROTLI_LARGE_MAX_DISTANCE_BITS);
/* The maximum distance is set so that no distance symbol used can encode
a distance larger than BROTLI_MAX_ALLOWED_DISTANCE with all
its extra bits set. */
if (ndirect < bound[npostfix]) {
max_distance = BROTLI_MAX_ALLOWED_DISTANCE - (bound[npostfix] - ndirect);
} else if (ndirect >= bound[npostfix] + postfix) {
max_distance = (3U << 29) - 4 + (ndirect - bound[npostfix]);
} else {
max_distance = BROTLI_MAX_ALLOWED_DISTANCE;
}
}
dist_params->alphabet_size = alphabet_size;
dist_params->max_distance = max_distance;
}
static void RecomputeDistancePrefixes(Command* cmds,
size_t num_commands,
const BrotliDistanceParams* orig_params,
const BrotliDistanceParams* new_params) {
size_t i;
if (orig_params->distance_postfix_bits == new_params->distance_postfix_bits &&
orig_params->num_direct_distance_codes ==
new_params->num_direct_distance_codes) {
return;
}
for (i = 0; i < num_commands; ++i) {
Command* cmd = &cmds[i];
if (CommandCopyLen(cmd) && cmd->cmd_prefix_ >= 128) {
PrefixEncodeCopyDistance(CommandRestoreDistanceCode(cmd, orig_params),
new_params->num_direct_distance_codes,
new_params->distance_postfix_bits,
&cmd->dist_prefix_,
&cmd->dist_extra_);
}
}
}
static BROTLI_BOOL ComputeDistanceCost(const Command* cmds,
size_t num_commands,
const BrotliDistanceParams* orig_params,
const BrotliDistanceParams* new_params,
double* cost) {
size_t i;
BROTLI_BOOL equal_params = BROTLI_FALSE;
uint16_t dist_prefix;
uint32_t dist_extra;
double extra_bits = 0.0;
HistogramDistance histo;
HistogramClearDistance(&histo);
if (orig_params->distance_postfix_bits == new_params->distance_postfix_bits &&
orig_params->num_direct_distance_codes ==
new_params->num_direct_distance_codes) {
equal_params = BROTLI_TRUE;
}
for (i = 0; i < num_commands; i++) {
const Command* cmd = &cmds[i];
if (CommandCopyLen(cmd) && cmd->cmd_prefix_ >= 128) {
if (equal_params) {
dist_prefix = cmd->dist_prefix_;
} else {
uint32_t distance = CommandRestoreDistanceCode(cmd, orig_params);
if (distance > new_params->max_distance) {
return BROTLI_FALSE;
}
PrefixEncodeCopyDistance(distance,
new_params->num_direct_distance_codes,
new_params->distance_postfix_bits,
&dist_prefix,
&dist_extra);
}
HistogramAddDistance(&histo, dist_prefix & 0x3FF);
extra_bits += dist_prefix >> 10;
}
}
*cost = BrotliPopulationCostDistance(&histo) + extra_bits;
return BROTLI_TRUE;
}
void BrotliBuildMetaBlock(MemoryManager* m,
const uint8_t* ringbuffer,
const size_t pos,
const size_t mask,
const BrotliEncoderParams* params,
BrotliEncoderParams* params,
uint8_t prev_byte,
uint8_t prev_byte2,
const Command* cmds,
Command* cmds,
size_t num_commands,
ContextType literal_context_mode,
MetaBlockSplit* mb) {
@ -40,10 +142,52 @@ void BrotliBuildMetaBlock(MemoryManager* m,
static const size_t kMaxNumberOfHistograms = 256;
HistogramDistance* distance_histograms;
HistogramLiteral* literal_histograms;
ContextType* literal_context_modes;
size_t num_literal_contexts;
size_t num_distance_contexts;
ContextType* literal_context_modes = NULL;
size_t literal_histograms_size;
size_t distance_histograms_size;
size_t i;
size_t literal_context_multiplier = 1;
uint32_t npostfix;
uint32_t ndirect_msb = 0;
BROTLI_BOOL check_orig = BROTLI_TRUE;
double best_dist_cost = 1e99;
BrotliEncoderParams orig_params = *params;
BrotliEncoderParams new_params = *params;
for (npostfix = 0; npostfix <= BROTLI_MAX_NPOSTFIX; npostfix++) {
for (; ndirect_msb < 16; ndirect_msb++) {
uint32_t ndirect = ndirect_msb << npostfix;
BROTLI_BOOL skip;
double dist_cost;
BrotliInitDistanceParams(&new_params, npostfix, ndirect);
if (npostfix == orig_params.dist.distance_postfix_bits &&
ndirect == orig_params.dist.num_direct_distance_codes) {
check_orig = BROTLI_FALSE;
}
skip = !ComputeDistanceCost(
cmds, num_commands,
&orig_params.dist, &new_params.dist, &dist_cost);
if (skip || (dist_cost > best_dist_cost)) {
break;
}
best_dist_cost = dist_cost;
params->dist = new_params.dist;
}
if (ndirect_msb > 0) ndirect_msb--;
ndirect_msb /= 2;
}
if (check_orig) {
double dist_cost;
ComputeDistanceCost(cmds, num_commands,
&orig_params.dist, &orig_params.dist, &dist_cost);
if (dist_cost < best_dist_cost) {
/* NB: currently unused; uncomment when more param tuning is added. */
/* best_dist_cost = dist_cost; */
params->dist = orig_params.dist;
}
}
RecomputeDistancePrefixes(cmds, num_commands,
&orig_params.dist, &params->dist);
BrotliSplitBlock(m, cmds, num_commands,
ringbuffer, pos, mask, params,
@ -52,68 +196,87 @@ void BrotliBuildMetaBlock(MemoryManager* m,
&mb->distance_split);
if (BROTLI_IS_OOM(m)) return;
literal_context_modes =
BROTLI_ALLOC(m, ContextType, mb->literal_split.num_types);
if (BROTLI_IS_OOM(m)) return;
for (i = 0; i < mb->literal_split.num_types; ++i) {
literal_context_modes[i] = literal_context_mode;
if (!params->disable_literal_context_modeling) {
literal_context_multiplier = 1 << BROTLI_LITERAL_CONTEXT_BITS;
literal_context_modes =
BROTLI_ALLOC(m, ContextType, mb->literal_split.num_types);
if (BROTLI_IS_OOM(m)) return;
for (i = 0; i < mb->literal_split.num_types; ++i) {
literal_context_modes[i] = literal_context_mode;
}
}
num_literal_contexts =
mb->literal_split.num_types << BROTLI_LITERAL_CONTEXT_BITS;
num_distance_contexts =
mb->distance_split.num_types << BROTLI_DISTANCE_CONTEXT_BITS;
literal_histograms = BROTLI_ALLOC(m, HistogramLiteral, num_literal_contexts);
literal_histograms_size =
mb->literal_split.num_types * literal_context_multiplier;
literal_histograms =
BROTLI_ALLOC(m, HistogramLiteral, literal_histograms_size);
if (BROTLI_IS_OOM(m)) return;
ClearHistogramsLiteral(literal_histograms, num_literal_contexts);
ClearHistogramsLiteral(literal_histograms, literal_histograms_size);
assert(mb->command_histograms == 0);
distance_histograms_size =
mb->distance_split.num_types << BROTLI_DISTANCE_CONTEXT_BITS;
distance_histograms =
BROTLI_ALLOC(m, HistogramDistance, distance_histograms_size);
if (BROTLI_IS_OOM(m)) return;
ClearHistogramsDistance(distance_histograms, distance_histograms_size);
BROTLI_DCHECK(mb->command_histograms == 0);
mb->command_histograms_size = mb->command_split.num_types;
mb->command_histograms =
BROTLI_ALLOC(m, HistogramCommand, mb->command_histograms_size);
if (BROTLI_IS_OOM(m)) return;
ClearHistogramsCommand(mb->command_histograms, mb->command_histograms_size);
distance_histograms =
BROTLI_ALLOC(m, HistogramDistance, num_distance_contexts);
if (BROTLI_IS_OOM(m)) return;
ClearHistogramsDistance(distance_histograms, num_distance_contexts);
BrotliBuildHistogramsWithContext(cmds, num_commands,
&mb->literal_split, &mb->command_split, &mb->distance_split,
ringbuffer, pos, mask, prev_byte, prev_byte2, literal_context_modes,
literal_histograms, mb->command_histograms, distance_histograms);
BROTLI_FREE(m, literal_context_modes);
assert(mb->literal_context_map == 0);
BROTLI_DCHECK(mb->literal_context_map == 0);
mb->literal_context_map_size =
mb->literal_split.num_types << BROTLI_LITERAL_CONTEXT_BITS;
mb->literal_context_map =
BROTLI_ALLOC(m, uint32_t, mb->literal_context_map_size);
if (BROTLI_IS_OOM(m)) return;
assert(mb->literal_histograms == 0);
BROTLI_DCHECK(mb->literal_histograms == 0);
mb->literal_histograms_size = mb->literal_context_map_size;
mb->literal_histograms =
BROTLI_ALLOC(m, HistogramLiteral, mb->literal_histograms_size);
if (BROTLI_IS_OOM(m)) return;
BrotliClusterHistogramsLiteral(m, literal_histograms,
mb->literal_context_map_size,
kMaxNumberOfHistograms,
mb->literal_histograms,
&mb->literal_histograms_size,
mb->literal_context_map);
BrotliClusterHistogramsLiteral(m, literal_histograms, literal_histograms_size,
kMaxNumberOfHistograms, mb->literal_histograms,
&mb->literal_histograms_size, mb->literal_context_map);
if (BROTLI_IS_OOM(m)) return;
BROTLI_FREE(m, literal_histograms);
assert(mb->distance_context_map == 0);
if (params->disable_literal_context_modeling) {
/* Distribute assignment to all contexts. */
for (i = mb->literal_split.num_types; i != 0;) {
size_t j = 0;
i--;
for (; j < (1 << BROTLI_LITERAL_CONTEXT_BITS); j++) {
mb->literal_context_map[(i << BROTLI_LITERAL_CONTEXT_BITS) + j] =
mb->literal_context_map[i];
}
}
}
BROTLI_DCHECK(mb->distance_context_map == 0);
mb->distance_context_map_size =
mb->distance_split.num_types << BROTLI_DISTANCE_CONTEXT_BITS;
mb->distance_context_map =
BROTLI_ALLOC(m, uint32_t, mb->distance_context_map_size);
if (BROTLI_IS_OOM(m)) return;
assert(mb->distance_histograms == 0);
BROTLI_DCHECK(mb->distance_histograms == 0);
mb->distance_histograms_size = mb->distance_context_map_size;
mb->distance_histograms =
BROTLI_ALLOC(m, HistogramDistance, mb->distance_histograms_size);
if (BROTLI_IS_OOM(m)) return;
BrotliClusterHistogramsDistance(m, distance_histograms,
mb->distance_context_map_size,
kMaxNumberOfHistograms,
@ -136,53 +299,7 @@ void BrotliBuildMetaBlock(MemoryManager* m,
#include "./metablock_inc.h" /* NOLINT(build/include) */
#undef FN
void BrotliBuildMetaBlockGreedy(MemoryManager* m,
const uint8_t* ringbuffer,
size_t pos,
size_t mask,
const Command *commands,
size_t n_commands,
MetaBlockSplit* mb) {
BlockSplitterLiteral lit_blocks;
BlockSplitterCommand cmd_blocks;
BlockSplitterDistance dist_blocks;
size_t num_literals = 0;
size_t i;
for (i = 0; i < n_commands; ++i) {
num_literals += commands[i].insert_len_;
}
InitBlockSplitterLiteral(m, &lit_blocks, 256, 512, 400.0, num_literals,
&mb->literal_split, &mb->literal_histograms,
&mb->literal_histograms_size);
if (BROTLI_IS_OOM(m)) return;
InitBlockSplitterCommand(m, &cmd_blocks, BROTLI_NUM_COMMAND_SYMBOLS, 1024,
500.0, n_commands, &mb->command_split, &mb->command_histograms,
&mb->command_histograms_size);
if (BROTLI_IS_OOM(m)) return;
InitBlockSplitterDistance(m, &dist_blocks, 64, 512, 100.0, n_commands,
&mb->distance_split, &mb->distance_histograms,
&mb->distance_histograms_size);
if (BROTLI_IS_OOM(m)) return;
for (i = 0; i < n_commands; ++i) {
const Command cmd = commands[i];
size_t j;
BlockSplitterAddSymbolCommand(&cmd_blocks, cmd.cmd_prefix_);
for (j = cmd.insert_len_; j != 0; --j) {
BlockSplitterAddSymbolLiteral(&lit_blocks, ringbuffer[pos & mask]);
++pos;
}
pos += CommandCopyLen(&cmd);
if (CommandCopyLen(&cmd) && cmd.cmd_prefix_ >= 128) {
BlockSplitterAddSymbolDistance(&dist_blocks, cmd.dist_prefix_);
}
}
BlockSplitterFinishBlockLiteral(&lit_blocks, /* is_final = */ BROTLI_TRUE);
BlockSplitterFinishBlockCommand(&cmd_blocks, /* is_final = */ BROTLI_TRUE);
BlockSplitterFinishBlockDistance(&dist_blocks, /* is_final = */ BROTLI_TRUE);
}
#define BROTLI_MAX_STATIC_CONTEXTS 13
/* Greedy block splitter for one block category (literal, command or distance).
Gathers histograms for all context buckets. */
@ -214,7 +331,7 @@ typedef struct ContextBlockSplitter {
/* Offset of the histograms of the previous two block types. */
size_t last_histogram_ix_[2];
/* Entropy of the previous two block types. */
double* last_entropy_;
double last_entropy_[2 * BROTLI_MAX_STATIC_CONTEXTS];
/* The number of times we merged the current block with the last one. */
size_t merge_last_count_;
} ContextBlockSplitter;
@ -226,6 +343,7 @@ static void InitContextBlockSplitter(
size_t* histograms_size) {
size_t max_num_blocks = num_symbols / min_block_size + 1;
size_t max_num_types;
BROTLI_DCHECK(num_contexts <= BROTLI_MAX_STATIC_CONTEXTS);
self->alphabet_size_ = alphabet_size;
self->num_contexts_ = num_contexts;
@ -250,29 +368,23 @@ static void InitContextBlockSplitter(
split->lengths, split->lengths_alloc_size, max_num_blocks);
if (BROTLI_IS_OOM(m)) return;
split->num_blocks = max_num_blocks;
self->last_entropy_ = BROTLI_ALLOC(m, double, 2 * num_contexts);
if (BROTLI_IS_OOM(m)) return;
assert(*histograms == 0);
BROTLI_DCHECK(*histograms == 0);
*histograms_size = max_num_types * num_contexts;
*histograms = BROTLI_ALLOC(m, HistogramLiteral, *histograms_size);
self->histograms_ = *histograms;
if (BROTLI_IS_OOM(m)) return;
/* Clear only current historgram. */
/* Clear only current histogram. */
ClearHistogramsLiteral(&self->histograms_[0], num_contexts);
self->last_histogram_ix_[0] = self->last_histogram_ix_[1] = 0;
}
static void CleanupContextBlockSplitter(
MemoryManager* m, ContextBlockSplitter* self) {
BROTLI_FREE(m, self->last_entropy_);
}
/* Does either of three things:
(1) emits the current block with a new block type;
(2) emits the current block with the type of the second last block;
(3) merges the current block with the last block. */
static void ContextBlockSplitterFinishBlock(
MemoryManager* m, ContextBlockSplitter* self, BROTLI_BOOL is_final) {
ContextBlockSplitter* self, MemoryManager* m, BROTLI_BOOL is_final) {
BlockSplit* split = self->split_;
const size_t num_contexts = self->num_contexts_;
double* last_entropy = self->last_entropy_;
@ -305,10 +417,10 @@ static void ContextBlockSplitterFinishBlock(
respective set of histograms for the last and second last block types.
Decide over the split based on the total reduction of entropy across
all contexts. */
double* entropy = BROTLI_ALLOC(m, double, num_contexts);
double entropy[BROTLI_MAX_STATIC_CONTEXTS];
HistogramLiteral* combined_histo =
BROTLI_ALLOC(m, HistogramLiteral, 2 * num_contexts);
double* combined_entropy = BROTLI_ALLOC(m, double, 2 * num_contexts);
double combined_entropy[2 * BROTLI_MAX_STATIC_CONTEXTS];
double diff[2] = { 0.0 };
size_t i;
if (BROTLI_IS_OOM(m)) return;
@ -383,9 +495,7 @@ static void ContextBlockSplitterFinishBlock(
self->target_block_size_ += self->min_block_size_;
}
}
BROTLI_FREE(m, combined_entropy);
BROTLI_FREE(m, combined_histo);
BROTLI_FREE(m, entropy);
}
if (is_final) {
*self->histograms_size_ = split->num_types * num_contexts;
@ -395,30 +505,49 @@ static void ContextBlockSplitterFinishBlock(
/* Adds the next symbol to the current block type and context. When the
current block reaches the target size, decides on merging the block. */
static void ContextBlockSplitterAddSymbol(MemoryManager* m,
ContextBlockSplitter* self, size_t symbol, size_t context) {
static void ContextBlockSplitterAddSymbol(
ContextBlockSplitter* self, MemoryManager* m,
size_t symbol, size_t context) {
HistogramAddLiteral(&self->histograms_[self->curr_histogram_ix_ + context],
symbol);
++self->block_size_;
if (self->block_size_ == self->target_block_size_) {
ContextBlockSplitterFinishBlock(m, self, /* is_final = */ BROTLI_FALSE);
ContextBlockSplitterFinishBlock(self, m, /* is_final = */ BROTLI_FALSE);
if (BROTLI_IS_OOM(m)) return;
}
}
void BrotliBuildMetaBlockGreedyWithContexts(MemoryManager* m,
const uint8_t* ringbuffer,
size_t pos,
size_t mask,
uint8_t prev_byte,
uint8_t prev_byte2,
ContextType literal_context_mode,
size_t num_contexts,
const uint32_t* static_context_map,
const Command *commands,
size_t n_commands,
MetaBlockSplit* mb) {
ContextBlockSplitter lit_blocks;
static void MapStaticContexts(MemoryManager* m,
size_t num_contexts,
const uint32_t* static_context_map,
MetaBlockSplit* mb) {
size_t i;
BROTLI_DCHECK(mb->literal_context_map == 0);
mb->literal_context_map_size =
mb->literal_split.num_types << BROTLI_LITERAL_CONTEXT_BITS;
mb->literal_context_map =
BROTLI_ALLOC(m, uint32_t, mb->literal_context_map_size);
if (BROTLI_IS_OOM(m)) return;
for (i = 0; i < mb->literal_split.num_types; ++i) {
uint32_t offset = (uint32_t)(i * num_contexts);
size_t j;
for (j = 0; j < (1u << BROTLI_LITERAL_CONTEXT_BITS); ++j) {
mb->literal_context_map[(i << BROTLI_LITERAL_CONTEXT_BITS) + j] =
offset + static_context_map[j];
}
}
}
static BROTLI_INLINE void BrotliBuildMetaBlockGreedyInternal(
MemoryManager* m, const uint8_t* ringbuffer, size_t pos, size_t mask,
uint8_t prev_byte, uint8_t prev_byte2, ContextLut literal_context_lut,
const size_t num_contexts, const uint32_t* static_context_map,
const Command* commands, size_t n_commands, MetaBlockSplit* mb) {
union {
BlockSplitterLiteral plain;
ContextBlockSplitter ctx;
} lit_blocks;
BlockSplitterCommand cmd_blocks;
BlockSplitterDistance dist_blocks;
size_t num_literals = 0;
@ -427,9 +556,15 @@ void BrotliBuildMetaBlockGreedyWithContexts(MemoryManager* m,
num_literals += commands[i].insert_len_;
}
InitContextBlockSplitter(m, &lit_blocks, 256, num_contexts, 512, 400.0,
num_literals, &mb->literal_split, &mb->literal_histograms,
&mb->literal_histograms_size);
if (num_contexts == 1) {
InitBlockSplitterLiteral(m, &lit_blocks.plain, 256, 512, 400.0,
num_literals, &mb->literal_split, &mb->literal_histograms,
&mb->literal_histograms_size);
} else {
InitContextBlockSplitter(m, &lit_blocks.ctx, 256, num_contexts, 512, 400.0,
num_literals, &mb->literal_split, &mb->literal_histograms,
&mb->literal_histograms_size);
}
if (BROTLI_IS_OOM(m)) return;
InitBlockSplitterCommand(m, &cmd_blocks, BROTLI_NUM_COMMAND_SYMBOLS, 1024,
500.0, n_commands, &mb->command_split, &mb->command_histograms,
@ -445,12 +580,17 @@ void BrotliBuildMetaBlockGreedyWithContexts(MemoryManager* m,
size_t j;
BlockSplitterAddSymbolCommand(&cmd_blocks, cmd.cmd_prefix_);
for (j = cmd.insert_len_; j != 0; --j) {
size_t context = Context(prev_byte, prev_byte2, literal_context_mode);
uint8_t literal = ringbuffer[pos & mask];
ContextBlockSplitterAddSymbol(
m, &lit_blocks, literal, static_context_map[context]);
if (num_contexts == 1) {
BlockSplitterAddSymbolLiteral(&lit_blocks.plain, literal);
} else {
size_t context =
BROTLI_CONTEXT(prev_byte, prev_byte2, literal_context_lut);
ContextBlockSplitterAddSymbol(&lit_blocks.ctx, m, literal,
static_context_map[context]);
if (BROTLI_IS_OOM(m)) return;
}
prev_byte2 = prev_byte;
if (BROTLI_IS_OOM(m)) return;
prev_byte = literal;
++pos;
}
@ -459,38 +599,52 @@ void BrotliBuildMetaBlockGreedyWithContexts(MemoryManager* m,
prev_byte2 = ringbuffer[(pos - 2) & mask];
prev_byte = ringbuffer[(pos - 1) & mask];
if (cmd.cmd_prefix_ >= 128) {
BlockSplitterAddSymbolDistance(&dist_blocks, cmd.dist_prefix_);
BlockSplitterAddSymbolDistance(&dist_blocks, cmd.dist_prefix_ & 0x3FF);
}
}
}
ContextBlockSplitterFinishBlock(m, &lit_blocks, /* is_final = */ BROTLI_TRUE);
if (BROTLI_IS_OOM(m)) return;
CleanupContextBlockSplitter(m, &lit_blocks);
if (num_contexts == 1) {
BlockSplitterFinishBlockLiteral(
&lit_blocks.plain, /* is_final = */ BROTLI_TRUE);
} else {
ContextBlockSplitterFinishBlock(
&lit_blocks.ctx, m, /* is_final = */ BROTLI_TRUE);
if (BROTLI_IS_OOM(m)) return;
}
BlockSplitterFinishBlockCommand(&cmd_blocks, /* is_final = */ BROTLI_TRUE);
BlockSplitterFinishBlockDistance(&dist_blocks, /* is_final = */ BROTLI_TRUE);
assert(mb->literal_context_map == 0);
mb->literal_context_map_size =
mb->literal_split.num_types << BROTLI_LITERAL_CONTEXT_BITS;
mb->literal_context_map =
BROTLI_ALLOC(m, uint32_t, mb->literal_context_map_size);
if (BROTLI_IS_OOM(m)) return;
for (i = 0; i < mb->literal_split.num_types; ++i) {
size_t j;
for (j = 0; j < (1u << BROTLI_LITERAL_CONTEXT_BITS); ++j) {
mb->literal_context_map[(i << BROTLI_LITERAL_CONTEXT_BITS) + j] =
(uint32_t)(i * num_contexts) + static_context_map[j];
}
if (num_contexts > 1) {
MapStaticContexts(m, num_contexts, static_context_map, mb);
}
}
void BrotliOptimizeHistograms(size_t num_direct_distance_codes,
size_t distance_postfix_bits,
void BrotliBuildMetaBlockGreedy(MemoryManager* m,
const uint8_t* ringbuffer,
size_t pos,
size_t mask,
uint8_t prev_byte,
uint8_t prev_byte2,
ContextLut literal_context_lut,
size_t num_contexts,
const uint32_t* static_context_map,
const Command* commands,
size_t n_commands,
MetaBlockSplit* mb) {
if (num_contexts == 1) {
BrotliBuildMetaBlockGreedyInternal(m, ringbuffer, pos, mask, prev_byte,
prev_byte2, literal_context_lut, 1, NULL, commands, n_commands, mb);
} else {
BrotliBuildMetaBlockGreedyInternal(m, ringbuffer, pos, mask, prev_byte,
prev_byte2, literal_context_lut, num_contexts, static_context_map,
commands, n_commands, mb);
}
}
void BrotliOptimizeHistograms(uint32_t num_distance_codes,
MetaBlockSplit* mb) {
uint8_t good_for_rle[BROTLI_NUM_COMMAND_SYMBOLS];
size_t num_distance_codes;
size_t i;
for (i = 0; i < mb->literal_histograms_size; ++i) {
BrotliOptimizeHuffmanCountsForRle(256, mb->literal_histograms[i].data_,
@ -501,8 +655,6 @@ void BrotliOptimizeHistograms(size_t num_direct_distance_codes,
mb->command_histograms[i].data_,
good_for_rle);
}
num_distance_codes = BROTLI_NUM_DISTANCE_SHORT_CODES +
num_direct_distance_codes + (48u << distance_postfix_bits);
for (i = 0; i < mb->distance_histograms_size; ++i) {
BrotliOptimizeHuffmanCountsForRle(num_distance_codes,
mb->distance_histograms[i].data_,

View File

@ -10,13 +10,13 @@
#ifndef BROTLI_ENC_METABLOCK_H_
#define BROTLI_ENC_METABLOCK_H_
#include "../common/types.h"
#include "../common/context.h"
#include "../common/platform.h"
#include <brotli/types.h>
#include "./block_splitter.h"
#include "./command.h"
#include "./context.h"
#include "./histogram.h"
#include "./memory.h"
#include "./port.h"
#include "./quality.h"
#if defined(__cplusplus) || defined(c_plusplus)
@ -67,42 +67,37 @@ static BROTLI_INLINE void DestroyMetaBlockSplit(
BROTLI_FREE(m, mb->distance_histograms);
}
/* Uses the slow shortest-path block splitter and does context clustering. */
/* Uses the slow shortest-path block splitter and does context clustering.
The distance parameters are dynamically selected based on the commands
which get recomputed under the new distance parameters. The new distance
parameters are stored into *params. */
BROTLI_INTERNAL void BrotliBuildMetaBlock(MemoryManager* m,
const uint8_t* ringbuffer,
const size_t pos,
const size_t mask,
const BrotliEncoderParams* params,
BrotliEncoderParams* params,
uint8_t prev_byte,
uint8_t prev_byte2,
const Command* cmds,
Command* cmds,
size_t num_commands,
ContextType literal_context_mode,
MetaBlockSplit* mb);
/* Uses a fast greedy block splitter that tries to merge current block with the
last or the second last block and does not do any context modeling. */
BROTLI_INTERNAL void BrotliBuildMetaBlockGreedy(MemoryManager* m,
const uint8_t* ringbuffer,
size_t pos,
size_t mask,
const Command* commands,
size_t n_commands,
MetaBlockSplit* mb);
/* Uses a fast greedy block splitter that tries to merge current block with the
last or the second last block and uses a static context clustering which
is the same for all block types. */
BROTLI_INTERNAL void BrotliBuildMetaBlockGreedyWithContexts(
BROTLI_INTERNAL void BrotliBuildMetaBlockGreedy(
MemoryManager* m, const uint8_t* ringbuffer, size_t pos, size_t mask,
uint8_t prev_byte, uint8_t prev_byte2, ContextType literal_context_mode,
uint8_t prev_byte, uint8_t prev_byte2, ContextLut literal_context_lut,
size_t num_contexts, const uint32_t* static_context_map,
const Command* commands, size_t n_commands, MetaBlockSplit* mb);
BROTLI_INTERNAL void BrotliOptimizeHistograms(size_t num_direct_distance_codes,
size_t distance_postfix_bits,
BROTLI_INTERNAL void BrotliOptimizeHistograms(uint32_t num_distance_codes,
MetaBlockSplit* mb);
BROTLI_INTERNAL void BrotliInitDistanceParams(BrotliEncoderParams* params,
uint32_t npostfix, uint32_t ndirect);
#if defined(__cplusplus) || defined(c_plusplus)
} /* extern "C" */
#endif

View File

@ -67,7 +67,7 @@ static void FN(InitBlockSplitter)(
split->lengths, split->lengths_alloc_size, max_num_blocks);
if (BROTLI_IS_OOM(m)) return;
self->split_->num_blocks = max_num_blocks;
assert(*histograms == 0);
BROTLI_DCHECK(*histograms == 0);
*histograms_size = max_num_types;
*histograms = BROTLI_ALLOC(m, HistogramType, *histograms_size);
self->histograms_ = *histograms;

Some files were not shown because too many files have changed in this diff Show More