5061 freebsd boot loader integration (loader project)

Review Request #173 — Created March 8, 2016 and submitted

5061 freebsd boot loader integration (loader project)

  • 0
  • 0
  • 74
  • 12
  • 86
Description From Last Updated
  1. My biggest concern with this is the one big wad dropped into usr/src/boot. I don't think it's the way to go - I understand that you want it that way to make synching with upstream easier, but would there really be any merges except for bug fixes? It's going to diverge from FreeBSD more and more. Can you split it into smaller chunks? eg, put libstand to usr/src/lib (I guess that's were it lives in FreeBSD), move man pages to usr/src/man, etc.

    1. actually if you sort of "remove" the boot from usr/src/boot path, thats the root you can see in bsd; I used this layout for initial tree setup to simplify tracking of original sources in "first phase". I do agree, it is a bit artificial and not the ideal from point of view of overall source tree we got. I have been thinking about relocating bits to more proper place - including moving the string functions in boot/lib/libc to libtsand etc, but after having little chat with Jeff, I figured I would rather leave this for "second phase", otherwise it is quite hard to see what was brought in and why those rearrangements are done. in bsd the boot loader itself is actually in "kernel" tree - usr/src/sys/boot, where sys is their "uts".

    2. I guess I'm more interested in that "second phase" result then, having the final version of what you propose to put back in to the gate.

    3. I forgot to add, I actually already have moved one build tool used by loader - btxld to our usr/src/tools, its originally located in boot/i386/btx tree - but its really small piece.

    4. There is also the concern and confusion in our current tree; grub itself is in usr/src/grub, while its actually standalone and should not be there - ok, from my point of view the grub is dead and does not matter.

      Then we have usr/src/stand and usr/src/psm, with sparc related code (forth bootblock etc) is mostly in psm. In a sense, the loader and its libstand would really belong where standalone bits are.

      Also for time being I have kept around the acpica and compression libs - eventually we should have single instance for both, with compression it would be easy to do even now, but acpica has to wait till devs working on its update will sort it out, which is also why I hope to see usr/src/common/acpica, but thats another topic.

      And finally, I think the more "cleanup" would be in order in sense that the bsd code is mostly trying to use libstand specific headers, but some are bsd public headers, and I think we should get this code changed to more "native". But then again, its more work for future so more people could be involved, but also, it means we need somehow get the baseline source to gate.

      Surely, I can go on with polishing those bits forever (or till I get bored, whichever will happen first), but really, this is not work for one person and I have already spent 2 years on this whole topic. For conclusion - I really dont need to get another bike-shedding about which bit should belong where even before this project has chance to get to gate. For reasons given above, I would rather go for one single baseline import now, and then work out the proper final locations on followups. As I wrote in list - this is not final "product", its just about the beginning of the road and if we want to get anywhere on that path, we should start walking.

  2. usr/src/boot/lib/libstand/dosfs.c (Diff revision 22)

    The variable "i" is no longer used now that dos_checksum does the loop.

  3. usr/src/boot/lib/libstand/dosfs.c (Diff revision 22)

    The variable "i" is no longer used now that dos_checksum does the loop.

  4. usr/src/boot/lib/libstand/dosfs.c (Diff revision 22)

    This function is a mixture of tabs and spaces, but it looks like the rest of this file is just spaces. This function's changes should probably be updated to be all spaces, too.

    1. Actually it is a bit bad plan I think - the current fbsd cstyle is also using 8 char tabs, and the style conversion in general is happening on the go. It does create a bit of mess in between, true.

  5. usr/src/boot/lib/libstand/dosfs.c (Diff revision 22)


  6. usr/src/boot/lib/libstand/dosfs.c (Diff revision 22)

    The rest of the returns have their values parenthesized, but this one doesn't.

  7. usr/src/boot/lib/libstand/gzipfs.c (Diff revision 22)

    Need space after "if".

  8. usr/src/boot/lib/libstand/read.c (Diff revision 22)

    Not an issue, but I guess I don't fully understand the twiddle() stuff. Can you point me towards something that explains twiddle() and its impact?

    1. Essentially its setting the rate of -|/ printout sequence to produce spinner in between operations. While reading larger files we want to have indicator the loader is still alive and working but what I have found is, while reading large files, too frequent spinner printout will add noticeable delay:) So the actual value is balance between having reasonable feedback and not delaying file read too much. For example this change did save ~5 seconds in case of smartos boot archive loading;)

    2. Thanks for the explanation!

  9. usr/src/boot/lib/libstand/tftp.c (Diff revision 22)

    Can you make this "tsize != 0"?

  10. usr/src/boot/sys/boot/common/bcache.c (Diff revision 22)

    These are changes that have already been reviewed upstream, right?

    1. yes, and integrated.

  11. usr/src/boot/sys/boot/common/boot.c (Diff revision 22)

    I don't understand this change. Is this to gag some compiler warning?

    1. oh, yes, its leftover from experiment, sneaky one has managed to slip in...

  12. usr/src/boot/sys/boot/common/bootstrap.h (Diff revision 22)
    While you're touching these lines, maybe also remove the space following the *'s?
  13. usr/src/boot/sys/boot/common/console.c (Diff revision 22)
    Can you wrap the body of this for loop with { }?
  14. usr/src/boot/sys/boot/common/console.c (Diff revision 22)

    The indentation of this curly bracket is misleading: it's actually closing the body of the "if". Please place it on the same indentation level as the "if" keyword.

  15. usr/src/boot/sys/boot/common/help.common (Diff revision 22)
    How about:
    beadm activate unloads the currently loaded configuration and modules,
    sets currdev to <device> and loads the configuration from the new device.
  16. usr/src/boot/sys/boot/common/interp.c (Diff revision 22)

    How about:

    The PXE TFTP service allows opening exactly one connection at a time,
    so we need to read the included file into memory, and then process it
    line by line as it may contain embedded include commands.

  17. usr/src/boot/sys/boot/common/interp.c (Diff revision 22)

    Add a space after the comma and before the opening double quote.

    1. Ah, you have missed the update sprintf -> snprintf, which also fixed the spaces;)

  18. usr/src/boot/sys/boot/common/interp.c (Diff revision 22)

    While you're touching lines near here, can you remove the trailing on lines 279, 284, and 295?

  19. Trailing space.

  20. Trailing space.

  21. Trailing space.

  22. Is there a reason that this extern isn't part of one of the included headers? Could we add it to one of them?

  23. Add { } around the body of the "else".

  24. Add { } around the body.

  25. Move the "else" to right after the "}".

  26. usr/src/boot/sys/boot/common/load_elf.c (Diff revision 22)
    I'm assuming that here and lines 1058, 1089, and 1090 are to gag compiler warnings about unused variables. Could we use __attribute((unused)) like you did elsewhere instead?
  27. Same thing as load_elf.c: can we use __attribute((unused)) instead?
  28. usr/src/boot/sys/boot/common/merge_help.awk (Diff revision 22)

    This is unfortunate, I wasn't aware of this. It turns out that /usr/xpg4/bin/awk uses regcomp(3C) so it supports POSIX regex classes. You can just change the shebang and any other ways that this script is invoked to use that instead.

    (Aside: I've started taking a look into how hard it would be to update our nawk, and it seems like it wouldn't be too bad, just require extensive study and testing to make sure we don't lose any features.)

    1. yep, however, I think, even if this workaround is ugly, I would rather try to keep the build tools as they are and wait till the tools will get more sane. This way at least we have predictible build environment, without surprises... I'm more than happy to revert the change once we have sane nawk.. ;)

    2. I just checked, and /usr/bin/awk refers to oawk on OmniOS, which won't support some of the things used in this awk script. I think changing it to /usr/xpg4/bin/awk is the best solution here.

    3. I think you are right here, did revert merge_help.awk and set AWK=/usr/xpg4/bin/awk in i386/loader/Makefile.

  29. Once you use /usr/xpg4/bin/awk, you can revert these back to the character classes.

  30. usr/src/boot/sys/boot/common/part.h (Diff revision 22)

    I'm not too familiar with disk layouts. What's the difference between VTOC and VTOC8? Can you point me towards some place to read up on this?

    1. the header is in usr/src/boot/sys/sys/vtoc.h, as it appears, our vtoc (or now dk_label) and vtoc8 in bsd do have same roots (and also relations to AT&T SVr4, as it appears from comments in our /usr/include/sys/dklabel.h), but with some differences... So I did decide to leave vtoc8 as is, and did add dk_label instead. However, the current modern fbsd is rather using either mbr+bsd label or gpt.

  31. usr/src/boot/sys/boot/common/part.c (Diff revision 22)


  32. usr/src/boot/sys/boot/common/part.c (Diff revision 22)

    This line is indented by 3 levels, but should be indented by 2.

  33. usr/src/boot/sys/boot/common/part.c (Diff revision 22)

    Should there be a PREF_ILLUMOS, and use that here, on line 901, and elsewhere?

    1. yea, well, actually this whole autoselect code is useless for us, but as long as its here, you are right. The fbsd boot code is built on reading the partition table and about deciding what to use based on partition type/flags. Our current grub and also loader are using pre-recorded LBA based approach (loader actually also does have partition browsing as fallback). So in sense it is sort of historical trash, but I would really leave it for future cleanups otherwise;)

  34. usr/src/boot/sys/boot/common/ufsread.c (Diff revision 22)

    The final return is parenthesized, but not this one.

  35. usr/src/boot/sys/boot/common/ufsread.c (Diff revision 22)

    The final return is parenthesized, but not this one.

  36. usr/src/boot/sys/boot/efi/Makefile.inc (Diff revision 22)

    Why are these commented out? Should these lines just be deleted?

    1. The whole efi branch right now as of this update is reference code only, and not targeted for actual build. Most of the issues also pointed here are already addressed in followup development and therefore I would rather leave those files as is for now.

      The reason for not including efi branch now is simple - without updates to kernel (frame buffer console support, avoiding accessing bios calls and data), the efi code is still unusable. However, it will provide the base for future commits.

  37. usr/src/boot/sys/boot/efi/include/efiapi.h (Diff revision 22)
    I assume all of these additions here and elsewhere under usr/src/boot/sys/boot/efi/include/efi*.h are directly from updating headers from Intel, and not stuff that you've added?
    If this is stuff that you've added, has it also been updated upstream?
    1. yes, those are from Intel headers, I did add to help the debugging. I haven't pushed them because in current form they don't provide too much value for upstream, and some of listed UUID's are already received updates from upstream, with implementation of additional features.

  38. Should there be a comment on this line similar to the others before it?

  39. usr/src/boot/sys/boot/efi/loader/main.c (Diff revision 22)

    Why the uintptr_t cast?

    1. another sneaky one..

  40. usr/src/boot/sys/boot/efi/loader/main.c (Diff revision 22)

    s/track on cursor/track of cursor/

    1. well, this whole nvram command will get replaced later (as part of rework for variable support in freebsd), so, I just will leave it as is for now..

  41. usr/src/boot/sys/boot/efi/loader/main.c (Diff revision 22)

    Is this #ifdef'd because it doesn't currently work? This seems like something that it would be nice for us to eventually have.

    1. We actually have it, just the approach is a bit different. FreeBSD is reading BE list directly from zfs (dataset list), but since our libbe is using file based approach (<boot pool>/boot/menu.lst), I did implement BE support by reading the menu.lst file instead - the reader code is in beadm.4th. The idea is to make sure loader and OS does have exactly the same view on be list and its order. While reading dataset list directly from zfs would allow to drop menu.lst file, there are two "but's" - first I'm not entirely sure about dataset ordering in zfs zap, it may be the timestamps would be needed to be checked (which would add extra code into loader), also using menu.lst based approach will give us exact match with sparc, as it is in fact the same file and method used in case of sparc.

    2. Ah, that makes sense. Thanks for sorting all of that out!

  42. usr/src/boot/sys/boot/efi/loader/main.c (Diff revision 22)

    efi_serial_init() writes into command_errbuf, but it doesn't return any type of status code. It looks like this means that if this function fails in some way after being called on line 420, it'll drive on and never display the error message. Should this be returning a status code so that we can at least display the message in some way, and maybe abort? (Or is there a check for a non-empty buffer that I'm missing?)

    1. this one also is having its changes waiting already...

  43. usr/src/boot/sys/boot/ficl/Makefile.inc (Diff revision 22)

    It looks like you made this file. Can you add a CDDL header and copyright with your name at the top?

  44. usr/src/boot/sys/boot/forth/beadm.4th (Diff revision 22)

    I believe placing the copyright after the CDDL notice is preferred. Can you move this down?

  45. usr/src/boot/sys/boot/forth/beadm.4th (Diff revision 22)

    Can you insert a comment here summarizing what this code does?

  46. usr/src/boot/sys/boot/forth/loader.4th (Diff revision 22)

    s/Path is starting/Paths start/

  47. usr/src/boot/sys/boot/forth/loader.4th (Diff revision 22)

    How about:

    This is needed so that the menu can manage these options. Unfortunately, this
    also means that boot-args will override previously set options, but we have no
    way to control the processing order here. boot-args will be rebuilt at boot.

    NOTE: The best way to address the order is to not set any of the above
    options in boot-args.

  48. usr/src/boot/sys/boot/forth/support.4th (Diff revision 22)

    How about:

    built-in prefix directory name; it must end with /, so we don't
    need to check and insert it.

  49. usr/src/boot/sys/boot/forth/support.4th (Diff revision 22)

    s/xen kernel loaded/loaded the xen kernel/

  50. This file looks like it came straight from FreeBSD, so I just skimmed it.

    1. quite so. To support different coonfigurations, the FreeBSD implementation does provide multiple alternate stage files - this is due to various size limits, and historical background. Since our primary rootfs is zfs, I see no reason to keep the multitudes of modules and have consolidated to provide single mbr, stage2 (gptzfsboot) and loader binaries (+ cdboot and pxeboot as those are for specific cases). Also to provide full zfs support, I do build stage2 against libstand, which does increase the size of the binary, but also did remove the need for duplicated code. There is still some cleanup to do, but I leave the cleanup for later phases of the project.

  51. Are the changes in this file going to be part of the loader changes going in? It looks like you reverted these in your "loader" branch on GitHub.

    1. actually yes, and I just figured I need to update it. This version was left in from efi builds, which was sharing the biosacpi.c with bios loader. Later I did create separate (simpler) implementation in libefi, as for efi we dont really scan memory to find tables, the pointers are provided in efi system table. So I have updated this code to drop unneeded efi bits. Of course in illumos case the extracted information is just for humans, fakebop.c does its own magic to extract acpi.

  52. Why caddr_t and not void *?
    1. originally it was to keep in mind there are different address mappings for efi and V86 bios loader, removed this argument now as its not needed any more.

  53. Is the intptr_t cast to do sign extension?

    1. for this one: warning: assignment makes pointer from integer without a cast

  54. Since this is a void *, can you write NULL here?
  55. Space after sizeof keyword.

  56. s/for case/in case/

  57. Newline after start of block comment.

  58. s/Illumos/illumos/

  59. It looks like this file uses 4-space continuations elsewhere, so this should match.

  60. usr/src/boot/sys/boot/i386/libi386/multiboot.c (Diff revision 22)

    How about:

    Since for now we have no way to pass the environment to the kernel other than
    through arguments, we need to take care of console setup.

    • If the console is in mirror mode, set the kernel console from $os_console.
    • If it's unset, use the first item from $console.
    • If $console is "ttyX", also pass $ttyX-mode, since it may have been set by
      the user.

    In case of memory allocation errors, just return the original command line, so
    that we have a chance of booting.

    On success, cl will be freed and a new, allocated command line string is

  61. Space after return keyword.

  62. strstr() will look for the first occurrence of "tty" throughout the whole string. strncmp() would be better here so that we just check the start of the string.

  63. s/10/sizeof (mode)/

  64. Space after return keyword.

  65. Space after return keyword.

  66. Space after return keyword.

  67. Space after return keyword.

  68. You want "strlen(mbl_name) + 1" here, not "strlen(mbl_name + 1)".

  69. Place "else" on previous line. { } around the body would be good, too.

  70. usr/src/boot/sys/boot/i386/libi386/pxe.c (Diff revision 22)

    s/255/sizeof (line)/

  71. Put a copyright with your name here.

  72. s/placed to/placed in an/
    s/this code is moved to safe/the code is moved to a safe/

  73. s/set pointer/set the pointer/
    s/use call/call/

  74. s/so on entry, we have new/so that on entry, we have the new/

  75. s/32bit/32-bit/

  76. Replace with:
    if ((v86.eax & 0xff) != 0) {
  77. This now says "Uncomment", but the below line is not commented. Should the text remain what it was previously, or should the below line actually be commented out?

  78. usr/src/boot/sys/boot/i386/pmbr/pmbr.s (Diff revision 22)

    s/read boot program/read a boot program/

  79. usr/src/boot/sys/boot/zfs/libzfs.h (Diff revision 22)

    Is this #ifdef because boot environment stuff doesn't work on us yet?

    1. As I did comment before, we dont read BE list from dataset list:)

  80. usr/src/boot/sys/boot/zfs/zfsimpl.c (Diff revision 22)

    s/80/sizeof (line)/

  1. Ship It!
Review request changed

Status: Closed (submitted)