Sep 10 18:31:12.591892 ip-10-0-134-166 systemd[1]: Starting Kubernetes Kubelet... Sep 10 18:31:12.969535 ip-10-0-134-166 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 18:31:12.969535 ip-10-0-134-166 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 10 18:31:12.969535 ip-10-0-134-166 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 18:31:12.969535 ip-10-0-134-166 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 10 18:31:12.969535 ip-10-0-134-166 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 18:31:12.970525 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.970430 2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 10 18:31:12.977490 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977475 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 18:31:12.977490 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977490 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 18:31:12.977557 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977494 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 18:31:12.977557 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977498 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 18:31:12.977557 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977501 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 18:31:12.977557 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977504 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 18:31:12.977557 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977506 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 18:31:12.977557 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977509 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 18:31:12.977557 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977512 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 18:31:12.977557 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977515 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 18:31:12.977557 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977517 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 18:31:12.977557 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977520 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 18:31:12.977557 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977528 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 18:31:12.977557 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977531 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 18:31:12.977557 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977533 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 18:31:12.977557 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977537 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 18:31:12.977557 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977541 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 18:31:12.977557 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977544 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 18:31:12.977557 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977547 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 18:31:12.977557 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977550 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 18:31:12.977557 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977553 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 18:31:12.978021 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977555 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 18:31:12.978021 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977558 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 18:31:12.978021 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977561 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 18:31:12.978021 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977564 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 18:31:12.978021 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977566 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 18:31:12.978021 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977569 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 18:31:12.978021 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977572 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 18:31:12.978021 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977574 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 18:31:12.978021 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977577 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 18:31:12.978021 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977579 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 18:31:12.978021 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977582 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 18:31:12.978021 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977584 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 18:31:12.978021 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977587 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 18:31:12.978021 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977589 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 18:31:12.978021 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977592 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 18:31:12.978021 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977594 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 18:31:12.978021 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977596 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 18:31:12.978021 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977599 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 18:31:12.978021 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977601 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 18:31:12.978021 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977603 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 18:31:12.978508 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977606 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 18:31:12.978508 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977623 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 18:31:12.978508 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977625 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 18:31:12.978508 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977628 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 18:31:12.978508 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977630 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 18:31:12.978508 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977633 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 18:31:12.978508 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977635 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 18:31:12.978508 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977638 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 18:31:12.978508 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977640 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 18:31:12.978508 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977643 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 18:31:12.978508 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977645 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 18:31:12.978508 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977647 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 18:31:12.978508 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977650 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 18:31:12.978508 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977653 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 18:31:12.978508 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977656 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 18:31:12.978508 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977659 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 18:31:12.978508 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977661 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 18:31:12.978508 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977664 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 18:31:12.978508 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977667 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 10 18:31:12.978508 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977669 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 18:31:12.979010 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977672 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 18:31:12.979010 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977674 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 18:31:12.979010 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977677 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 18:31:12.979010 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977679 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 18:31:12.979010 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977682 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 18:31:12.979010 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977684 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 18:31:12.979010 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977687 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 18:31:12.979010 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977690 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 18:31:12.979010 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977693 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 18:31:12.979010 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977695 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 18:31:12.979010 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977698 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 18:31:12.979010 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977701 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 18:31:12.979010 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977703 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 18:31:12.979010 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977705 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 18:31:12.979010 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977708 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 18:31:12.979010 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977711 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 18:31:12.979010 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977713 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 18:31:12.979010 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977715 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 18:31:12.979010 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977718 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 18:31:12.979533 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977722 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 18:31:12.979533 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977725 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 18:31:12.979533 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977728 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 18:31:12.979533 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977731 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 18:31:12.979533 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977733 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 18:31:12.979533 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977736 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 18:31:12.979533 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977740 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 18:31:12.979533 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977743 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 18:31:12.979533 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977746 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 18:31:12.979533 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977750 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 18:31:12.979533 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977753 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 18:31:12.979533 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977756 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 18:31:12.979533 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977759 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 18:31:12.979533 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977761 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 18:31:12.979533 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977764 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 18:31:12.979533 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977766 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 18:31:12.979533 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977769 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 18:31:12.979533 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977771 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 18:31:12.979533 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.977774 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 18:31:12.980022 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978459 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 18:31:12.980022 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978466 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 18:31:12.980022 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978469 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 18:31:12.980022 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978472 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 18:31:12.980022 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978474 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 18:31:12.980022 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978478 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 10 18:31:12.980022 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978480 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 18:31:12.980022 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978483 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 18:31:12.980022 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978486 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 18:31:12.980022 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978488 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 18:31:12.980022 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978491 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 18:31:12.980022 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978493 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 18:31:12.980022 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978496 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 18:31:12.980022 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978498 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 18:31:12.980022 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978503 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 18:31:12.980022 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978507 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 18:31:12.980022 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978510 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 18:31:12.980022 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978513 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 18:31:12.980022 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978516 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 18:31:12.980022 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978520 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 18:31:12.980498 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978523 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 18:31:12.980498 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978526 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 18:31:12.980498 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978528 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 18:31:12.980498 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978531 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 18:31:12.980498 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978534 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 18:31:12.980498 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978536 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 18:31:12.980498 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978538 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 18:31:12.980498 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978541 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 18:31:12.980498 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978543 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 18:31:12.980498 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978545 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 18:31:12.980498 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978548 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 18:31:12.980498 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978550 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 18:31:12.980498 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978553 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 18:31:12.980498 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978555 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 18:31:12.980498 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978558 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 18:31:12.980498 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978561 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 18:31:12.980498 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978564 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 18:31:12.980498 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978567 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 18:31:12.980498 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978570 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 18:31:12.980498 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978573 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 18:31:12.980993 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978575 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 18:31:12.980993 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978577 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 18:31:12.980993 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978580 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 18:31:12.980993 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978582 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 18:31:12.980993 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978585 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 18:31:12.980993 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978587 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 18:31:12.980993 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978592 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 18:31:12.980993 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978595 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 18:31:12.980993 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978598 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 18:31:12.980993 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978601 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 18:31:12.980993 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978604 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 18:31:12.980993 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978624 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 18:31:12.980993 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978628 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 18:31:12.980993 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978630 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 18:31:12.980993 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978633 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 18:31:12.980993 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978636 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 18:31:12.980993 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978638 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 18:31:12.980993 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978641 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 18:31:12.981436 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978643 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 18:31:12.981436 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978646 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 18:31:12.981436 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978649 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 18:31:12.981436 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978651 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 18:31:12.981436 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978654 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 18:31:12.981436 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978656 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 18:31:12.981436 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978659 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 18:31:12.981436 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978661 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 18:31:12.981436 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978663 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 18:31:12.981436 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978666 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 18:31:12.981436 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978668 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 18:31:12.981436 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978671 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 18:31:12.981436 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978673 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 18:31:12.981436 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978677 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 18:31:12.981436 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978679 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 18:31:12.981436 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978682 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 18:31:12.981436 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978684 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 18:31:12.981436 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978687 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 18:31:12.981436 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978689 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 18:31:12.981436 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978691 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 18:31:12.981937 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978694 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 18:31:12.981937 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978697 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 18:31:12.981937 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978700 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 18:31:12.981937 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978702 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 18:31:12.981937 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978705 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 18:31:12.981937 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978708 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 18:31:12.981937 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978711 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 18:31:12.981937 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978714 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 18:31:12.981937 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978716 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 18:31:12.981937 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978719 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 18:31:12.981937 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978722 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 18:31:12.981937 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978724 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 18:31:12.981937 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978726 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 18:31:12.981937 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978729 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 18:31:12.981937 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978732 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 18:31:12.981937 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978734 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 18:31:12.981937 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978737 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 18:31:12.981937 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978739 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 18:31:12.981937 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978741 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 18:31:12.981937 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978744 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 18:31:12.982426 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.978746 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 18:31:12.982426 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979090 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 18:31:12.982426 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979095 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 18:31:12.982426 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979098 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 18:31:12.982426 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979100 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 18:31:12.982426 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979104 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 18:31:12.982426 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979106 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 18:31:12.982426 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979109 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 18:31:12.982426 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979111 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 18:31:12.982426 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979114 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 18:31:12.982426 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979117 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 18:31:12.982426 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979119 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 18:31:12.982426 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979122 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 18:31:12.982426 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979124 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 18:31:12.982426 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979127 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 18:31:12.982426 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979129 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 18:31:12.982426 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979133 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 18:31:12.982426 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979136 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 18:31:12.982426 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979139 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 18:31:12.982426 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979141 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 18:31:12.982917 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979145 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 18:31:12.982917 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979149 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 18:31:12.982917 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979151 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 18:31:12.982917 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979154 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 18:31:12.982917 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979156 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 18:31:12.982917 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979159 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 18:31:12.982917 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979161 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 18:31:12.982917 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979164 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 18:31:12.982917 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979166 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 18:31:12.982917 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979168 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 18:31:12.982917 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979171 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 18:31:12.982917 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979174 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 18:31:12.982917 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979176 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 18:31:12.982917 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979178 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 18:31:12.982917 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979182 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 18:31:12.982917 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979185 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 18:31:12.982917 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979187 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 18:31:12.982917 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979189 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 18:31:12.982917 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979193 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 18:31:12.983372 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979195 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 18:31:12.983372 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979198 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 18:31:12.983372 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979200 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 18:31:12.983372 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979203 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 18:31:12.983372 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979205 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 18:31:12.983372 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979208 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 18:31:12.983372 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979210 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 18:31:12.983372 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979212 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 18:31:12.983372 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979215 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 18:31:12.983372 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979217 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 18:31:12.983372 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979221 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 18:31:12.983372 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979223 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 18:31:12.983372 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979226 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 18:31:12.983372 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979228 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 10 18:31:12.983372 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979231 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 18:31:12.983372 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979233 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 18:31:12.983372 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979235 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 18:31:12.983372 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979238 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 18:31:12.983372 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979240 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 18:31:12.983372 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979243 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 18:31:12.983886 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979245 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 18:31:12.983886 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979248 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 18:31:12.983886 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979250 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 18:31:12.983886 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979253 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 18:31:12.983886 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979255 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 18:31:12.983886 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979257 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 18:31:12.983886 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979260 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 18:31:12.983886 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979262 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 18:31:12.983886 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979265 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 18:31:12.983886 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979267 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 18:31:12.983886 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979270 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 18:31:12.983886 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979272 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 18:31:12.983886 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979276 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 18:31:12.983886 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979279 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 18:31:12.983886 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979281 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 18:31:12.983886 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979283 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 18:31:12.983886 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979286 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 18:31:12.983886 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979288 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 18:31:12.983886 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979291 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 18:31:12.983886 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979293 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 18:31:12.984400 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979295 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 18:31:12.984400 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979298 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 18:31:12.984400 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979301 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 18:31:12.984400 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979304 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 18:31:12.984400 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979306 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 18:31:12.984400 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979309 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 18:31:12.984400 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979311 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 18:31:12.984400 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979313 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 18:31:12.984400 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979316 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 18:31:12.984400 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979318 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 18:31:12.984400 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979321 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 18:31:12.984400 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979323 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 18:31:12.984400 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979327 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 18:31:12.984400 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979331 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 18:31:12.984400 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979335 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 18:31:12.984400 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979337 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 18:31:12.984400 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979340 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 18:31:12.984400 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979343 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 18:31:12.984400 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979345 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 18:31:12.984877 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979348 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 18:31:12.984877 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.979351 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 18:31:12.984877 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.979428 2581 flags.go:64] FLAG: --address="0.0.0.0" Sep 10 18:31:12.984877 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.979463 2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 10 18:31:12.984877 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.979470 2581 flags.go:64] FLAG: --anonymous-auth="true" Sep 10 18:31:12.984877 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.979475 2581 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 10 18:31:12.984877 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.979479 2581 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 10 18:31:12.984877 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.979482 2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 10 18:31:12.984877 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.979486 2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 10 18:31:12.984877 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.979490 2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 10 18:31:12.984877 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.979493 2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 10 18:31:12.984877 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.979496 2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 10 18:31:12.984877 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.979499 2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 10 18:31:12.984877 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.979502 2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 10 18:31:12.984877 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.979505 2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 10 18:31:12.984877 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.979510 2581 flags.go:64] FLAG: --cgroup-root="" Sep 10 18:31:12.984877 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.979513 2581 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 10 18:31:12.984877 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.979515 2581 flags.go:64] FLAG: --client-ca-file="" Sep 10 18:31:12.984877 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.979518 2581 flags.go:64] FLAG: --cloud-provider="external" Sep 10 18:31:12.984877 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.979521 2581 flags.go:64] FLAG: --cluster-dns="[]" Sep 10 18:31:12.984877 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.979524 2581 flags.go:64] FLAG: --cluster-domain="" Sep 10 18:31:12.984877 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.979527 2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 10 18:31:12.984877 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.979530 2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 10 18:31:12.985432 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.979533 2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 10 18:31:12.985432 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.979536 2581 flags.go:64] FLAG: --container-log-max-files="5" Sep 10 18:31:12.985432 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.979540 2581 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 10 18:31:12.985432 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.979543 2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 10 18:31:12.985432 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.979546 2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 10 18:31:12.985432 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.979549 2581 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 10 18:31:12.985432 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.979552 2581 flags.go:64] FLAG: --contention-profiling="false" Sep 10 18:31:12.985432 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.979555 2581 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 10 18:31:12.985432 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.979557 2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 10 18:31:12.985432 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.979560 2581 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 10 18:31:12.985432 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.979563 2581 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 10 18:31:12.985432 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.979567 2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 10 18:31:12.985432 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.979569 2581 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 10 18:31:12.985432 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.979572 2581 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 10 18:31:12.985432 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.979575 2581 flags.go:64] FLAG: --enable-load-reader="false" Sep 10 18:31:12.985432 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.979577 2581 flags.go:64] FLAG: --enable-server="true" Sep 10 18:31:12.985432 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.979581 2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 10 18:31:12.985432 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980318 2581 flags.go:64] FLAG: --event-burst="100" Sep 10 18:31:12.985432 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980323 2581 flags.go:64] FLAG: --event-qps="50" Sep 10 18:31:12.985432 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980326 2581 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 10 18:31:12.985432 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980329 2581 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 10 18:31:12.985432 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980332 2581 flags.go:64] FLAG: --eviction-hard="" Sep 10 18:31:12.985432 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980336 2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 10 18:31:12.985432 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980339 2581 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 10 18:31:12.986038 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980348 2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 10 18:31:12.986038 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980350 2581 flags.go:64] FLAG: --eviction-soft="" Sep 10 18:31:12.986038 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980353 2581 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 10 18:31:12.986038 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980356 2581 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 10 18:31:12.986038 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980359 2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 10 18:31:12.986038 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980362 2581 flags.go:64] FLAG: --experimental-mounter-path="" Sep 10 18:31:12.986038 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980365 2581 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 10 18:31:12.986038 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980368 2581 flags.go:64] FLAG: --fail-swap-on="true" Sep 10 18:31:12.986038 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980370 2581 flags.go:64] FLAG: --feature-gates="" Sep 10 18:31:12.986038 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980374 2581 flags.go:64] FLAG: --file-check-frequency="20s" Sep 10 18:31:12.986038 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980377 2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 10 18:31:12.986038 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980379 2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 10 18:31:12.986038 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980382 2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 10 18:31:12.986038 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980385 2581 flags.go:64] FLAG: --healthz-port="10248" Sep 10 18:31:12.986038 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980388 2581 flags.go:64] FLAG: --help="false" Sep 10 18:31:12.986038 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980391 2581 flags.go:64] FLAG: --hostname-override="ip-10-0-134-166.ec2.internal" Sep 10 18:31:12.986038 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980394 2581 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 10 18:31:12.986038 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980397 2581 flags.go:64] FLAG: --http-check-frequency="20s" Sep 10 18:31:12.986038 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980400 2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 10 18:31:12.986038 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980403 2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 10 18:31:12.986038 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980406 2581 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 10 18:31:12.986038 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980409 2581 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 10 18:31:12.986038 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980411 2581 flags.go:64] FLAG: --image-service-endpoint="" Sep 10 18:31:12.986625 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980414 2581 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 10 18:31:12.986625 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980417 2581 flags.go:64] FLAG: --kube-api-burst="100" Sep 10 18:31:12.986625 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980423 2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 10 18:31:12.986625 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980427 2581 flags.go:64] FLAG: --kube-api-qps="50" Sep 10 18:31:12.986625 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980429 2581 flags.go:64] FLAG: --kube-reserved="" Sep 10 18:31:12.986625 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980432 2581 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 10 18:31:12.986625 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980435 2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 10 18:31:12.986625 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980438 2581 flags.go:64] FLAG: --kubelet-cgroups="" Sep 10 18:31:12.986625 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980442 2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 10 18:31:12.986625 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980446 2581 flags.go:64] FLAG: --lock-file="" Sep 10 18:31:12.986625 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980449 2581 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 10 18:31:12.986625 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980451 2581 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 10 18:31:12.986625 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980454 2581 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 10 18:31:12.986625 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980460 2581 flags.go:64] FLAG: --log-json-split-stream="false" Sep 10 18:31:12.986625 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980463 2581 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 10 18:31:12.986625 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980466 2581 flags.go:64] FLAG: --log-text-split-stream="false" Sep 10 18:31:12.986625 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980468 2581 flags.go:64] FLAG: --logging-format="text" Sep 10 18:31:12.986625 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980471 2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 10 18:31:12.986625 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980474 2581 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 10 18:31:12.986625 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980477 2581 flags.go:64] FLAG: --manifest-url="" Sep 10 18:31:12.986625 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980480 2581 flags.go:64] FLAG: --manifest-url-header="" Sep 10 18:31:12.986625 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980484 2581 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 10 18:31:12.986625 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980487 2581 flags.go:64] FLAG: --max-open-files="1000000" Sep 10 18:31:12.986625 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980491 2581 flags.go:64] FLAG: --max-pods="110" Sep 10 18:31:12.986625 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980493 2581 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 10 18:31:12.987239 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980496 2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 10 18:31:12.987239 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980499 2581 flags.go:64] FLAG: --memory-manager-policy="None" Sep 10 18:31:12.987239 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980502 2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 10 18:31:12.987239 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980505 2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 10 18:31:12.987239 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980508 2581 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 10 18:31:12.987239 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980511 2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 10 18:31:12.987239 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980517 2581 flags.go:64] FLAG: --node-status-max-images="50" Sep 10 18:31:12.987239 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980520 2581 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 10 18:31:12.987239 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980523 2581 flags.go:64] FLAG: --oom-score-adj="-999" Sep 10 18:31:12.987239 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980526 2581 flags.go:64] FLAG: --pod-cidr="" Sep 10 18:31:12.987239 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980535 2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 10 18:31:12.987239 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980539 2581 flags.go:64] FLAG: --pod-manifest-path="" Sep 10 18:31:12.987239 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980542 2581 flags.go:64] FLAG: --pod-max-pids="-1" Sep 10 18:31:12.987239 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980545 2581 flags.go:64] FLAG: --pods-per-core="0" Sep 10 18:31:12.987239 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980548 2581 flags.go:64] FLAG: --port="10250" Sep 10 18:31:12.987239 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980550 2581 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 10 18:31:12.987239 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980556 2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0cf0ec017630b9b39" Sep 10 18:31:12.987239 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980559 2581 flags.go:64] FLAG: --qos-reserved="" Sep 10 18:31:12.987239 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980562 2581 flags.go:64] FLAG: --read-only-port="10255" Sep 10 18:31:12.987239 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980565 2581 flags.go:64] FLAG: --register-node="true" Sep 10 18:31:12.987239 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980568 2581 flags.go:64] FLAG: --register-with-taints="" Sep 10 18:31:12.987239 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980571 2581 flags.go:64] FLAG: --registry-burst="10" Sep 10 18:31:12.987239 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980574 2581 flags.go:64] FLAG: --registry-qps="5" Sep 10 18:31:12.987239 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980577 2581 flags.go:64] FLAG: --reserved-cpus="" Sep 10 18:31:12.987912 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980580 2581 flags.go:64] FLAG: --reserved-memory="" Sep 10 18:31:12.987912 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980583 2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 10 18:31:12.987912 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980586 2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 10 18:31:12.987912 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980589 2581 flags.go:64] FLAG: --rotate-certificates="false" Sep 10 18:31:12.987912 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980592 2581 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 10 18:31:12.987912 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980595 2581 flags.go:64] FLAG: --runonce="false" Sep 10 18:31:12.987912 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980598 2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 10 18:31:12.987912 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980600 2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 10 18:31:12.987912 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980603 2581 flags.go:64] FLAG: --seccomp-default="false" Sep 10 18:31:12.987912 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980606 2581 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 10 18:31:12.987912 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980625 2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 10 18:31:12.987912 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980630 2581 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 10 18:31:12.987912 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980635 2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 10 18:31:12.987912 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980640 2581 flags.go:64] FLAG: --storage-driver-password="root" Sep 10 18:31:12.987912 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980643 2581 flags.go:64] FLAG: --storage-driver-secure="false" Sep 10 18:31:12.987912 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980645 2581 flags.go:64] FLAG: --storage-driver-table="stats" Sep 10 18:31:12.987912 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980648 2581 flags.go:64] FLAG: --storage-driver-user="root" Sep 10 18:31:12.987912 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980651 2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 10 18:31:12.987912 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980654 2581 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 10 18:31:12.987912 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980663 2581 flags.go:64] FLAG: --system-cgroups="" Sep 10 18:31:12.987912 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980666 2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 10 18:31:12.987912 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980671 2581 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 10 18:31:12.987912 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980674 2581 flags.go:64] FLAG: --tls-cert-file="" Sep 10 18:31:12.987912 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980677 2581 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 10 18:31:12.987912 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980683 2581 flags.go:64] FLAG: --tls-min-version="" Sep 10 18:31:12.988520 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980686 2581 flags.go:64] FLAG: --tls-private-key-file="" Sep 10 18:31:12.988520 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980688 2581 flags.go:64] FLAG: --topology-manager-policy="none" Sep 10 18:31:12.988520 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980691 2581 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 10 18:31:12.988520 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980694 2581 flags.go:64] FLAG: --topology-manager-scope="container" Sep 10 18:31:12.988520 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980697 2581 flags.go:64] FLAG: --v="2" Sep 10 18:31:12.988520 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980705 2581 flags.go:64] FLAG: --version="false" Sep 10 18:31:12.988520 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980708 2581 flags.go:64] FLAG: --vmodule="" Sep 10 18:31:12.988520 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980712 2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 10 18:31:12.988520 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.980715 2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 10 18:31:12.988520 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980850 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 18:31:12.988520 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980854 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 18:31:12.988520 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980858 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 18:31:12.988520 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980860 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 18:31:12.988520 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980863 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 18:31:12.988520 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980866 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 18:31:12.988520 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980869 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 18:31:12.988520 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980872 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 10 18:31:12.988520 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980875 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 18:31:12.988520 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980878 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 18:31:12.988520 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980881 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 18:31:12.988520 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980883 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 18:31:12.988520 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980886 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 18:31:12.989064 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980889 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 18:31:12.989064 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980892 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 18:31:12.989064 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980894 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 18:31:12.989064 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980897 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 18:31:12.989064 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980899 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 18:31:12.989064 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980902 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 18:31:12.989064 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980905 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 18:31:12.989064 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980907 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 18:31:12.989064 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980910 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 18:31:12.989064 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980915 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 18:31:12.989064 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980918 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 18:31:12.989064 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980921 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 18:31:12.989064 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980924 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 18:31:12.989064 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980926 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 18:31:12.989064 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980929 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 18:31:12.989064 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980931 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 18:31:12.989064 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980934 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 18:31:12.989064 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980936 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 18:31:12.989064 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980938 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 18:31:12.989064 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980942 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 18:31:12.989562 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980946 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 18:31:12.989562 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980948 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 18:31:12.989562 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980951 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 18:31:12.989562 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980954 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 18:31:12.989562 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980956 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 18:31:12.989562 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980959 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 18:31:12.989562 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980961 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 18:31:12.989562 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980964 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 18:31:12.989562 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980966 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 18:31:12.989562 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980968 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 18:31:12.989562 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980971 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 18:31:12.989562 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980973 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 18:31:12.989562 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980976 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 18:31:12.989562 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980978 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 18:31:12.989562 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980981 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 18:31:12.989562 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980983 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 18:31:12.989562 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980987 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 18:31:12.989562 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980990 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 18:31:12.989562 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980993 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 18:31:12.990040 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980997 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 18:31:12.990040 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.980999 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 18:31:12.990040 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981005 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 18:31:12.990040 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981007 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 18:31:12.990040 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981010 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 18:31:12.990040 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981012 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 18:31:12.990040 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981015 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 18:31:12.990040 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981017 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 18:31:12.990040 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981020 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 18:31:12.990040 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981022 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 18:31:12.990040 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981025 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 18:31:12.990040 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981027 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 18:31:12.990040 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981030 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 18:31:12.990040 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981032 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 18:31:12.990040 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981034 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 18:31:12.990040 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981048 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 18:31:12.990040 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981051 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 18:31:12.990040 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981054 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 18:31:12.990040 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981057 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 18:31:12.990040 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981060 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 18:31:12.990532 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981062 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 18:31:12.990532 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981065 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 18:31:12.990532 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981067 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 18:31:12.990532 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981070 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 18:31:12.990532 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981073 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 18:31:12.990532 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981075 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 18:31:12.990532 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981078 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 18:31:12.990532 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981080 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 18:31:12.990532 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981082 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 18:31:12.990532 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981085 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 18:31:12.990532 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981087 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 18:31:12.990532 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981089 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 18:31:12.990532 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981094 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 18:31:12.990532 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981097 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 18:31:12.990532 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981101 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 18:31:12.990532 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981104 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 18:31:12.990532 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981106 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 18:31:12.990532 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981110 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 18:31:12.990532 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981113 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 18:31:12.991088 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981116 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 18:31:12.991088 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981118 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 18:31:12.991088 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981121 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 18:31:12.991088 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981123 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 18:31:12.991088 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981126 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 18:31:12.991088 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981128 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 18:31:12.991088 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981131 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 18:31:12.991088 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.981134 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 18:31:12.991088 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.981978 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 18:31:12.991088 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.989086 2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 10 18:31:12.991088 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.989100 2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 10 18:31:12.991088 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.989116 2581 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 10 18:31:12.991088 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.989122 2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 10 18:31:12.991088 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989175 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 18:31:12.991088 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989180 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 18:31:12.991796 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989183 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 18:31:12.991796 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989186 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 18:31:12.991796 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989190 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 18:31:12.991796 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989192 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 18:31:12.991796 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989196 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 18:31:12.991796 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989200 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 18:31:12.991796 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989203 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 18:31:12.991796 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989206 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 18:31:12.991796 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989208 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 18:31:12.991796 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989211 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 18:31:12.991796 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989213 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 18:31:12.991796 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989216 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 18:31:12.991796 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989219 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 18:31:12.991796 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989221 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 18:31:12.991796 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989224 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 18:31:12.991796 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989226 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 18:31:12.991796 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989229 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 18:31:12.991796 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989232 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 18:31:12.991796 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989234 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 18:31:12.991796 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989237 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 18:31:12.992488 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989240 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 18:31:12.992488 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989242 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 18:31:12.992488 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989245 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 18:31:12.992488 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989247 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 18:31:12.992488 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989250 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 10 18:31:12.992488 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989253 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 18:31:12.992488 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989255 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 18:31:12.992488 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989259 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 18:31:12.992488 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989262 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 18:31:12.992488 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989265 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 18:31:12.992488 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989267 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 18:31:12.992488 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989270 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 18:31:12.992488 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989272 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 18:31:12.992488 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989275 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 18:31:12.992488 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989277 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 18:31:12.992488 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989280 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 18:31:12.992488 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989282 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 18:31:12.992488 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989285 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 18:31:12.992488 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989288 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 18:31:12.992488 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989290 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 18:31:12.993054 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989294 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 18:31:12.993054 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989298 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 18:31:12.993054 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989301 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 18:31:12.993054 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989304 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 18:31:12.993054 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989306 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 18:31:12.993054 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989309 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 18:31:12.993054 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989311 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 18:31:12.993054 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989314 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 18:31:12.993054 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989316 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 18:31:12.993054 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989319 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 18:31:12.993054 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989322 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 18:31:12.993054 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989324 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 18:31:12.993054 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989326 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 18:31:12.993054 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989329 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 18:31:12.993054 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989331 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 18:31:12.993054 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989334 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 18:31:12.993054 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989336 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 18:31:12.993054 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989339 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 18:31:12.993054 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989342 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 18:31:12.993509 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989345 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 18:31:12.993509 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989348 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 18:31:12.993509 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989351 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 18:31:12.993509 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989354 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 18:31:12.993509 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989357 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 18:31:12.993509 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989359 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 18:31:12.993509 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989362 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 18:31:12.993509 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989364 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 18:31:12.993509 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989367 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 18:31:12.993509 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989369 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 18:31:12.993509 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989371 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 18:31:12.993509 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989374 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 18:31:12.993509 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989376 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 18:31:12.993509 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989379 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 18:31:12.993509 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989381 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 18:31:12.993509 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989384 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 18:31:12.993509 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989386 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 18:31:12.993509 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989389 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 18:31:12.993509 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989391 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 18:31:12.993509 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989393 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 18:31:12.994059 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989396 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 18:31:12.994059 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989398 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 18:31:12.994059 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989400 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 18:31:12.994059 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989403 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 18:31:12.994059 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989405 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 18:31:12.994059 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989408 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 18:31:12.994059 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989410 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 18:31:12.994059 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989413 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 18:31:12.994059 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989417 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 18:31:12.994059 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989420 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 18:31:12.994059 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989423 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 18:31:12.994059 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989426 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 18:31:12.994059 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989429 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 18:31:12.994059 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989432 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 18:31:12.994059 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989435 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 18:31:12.994059 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989438 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 18:31:12.994059 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989441 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 18:31:12.994059 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989443 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 18:31:12.994500 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.989448 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 18:31:12.994500 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989557 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 18:31:12.994500 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989562 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 10 18:31:12.994500 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989565 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 18:31:12.994500 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989569 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 18:31:12.994500 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989572 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 18:31:12.994500 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989575 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 18:31:12.994500 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989577 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 18:31:12.994500 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989580 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 18:31:12.994500 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989582 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 18:31:12.994500 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989585 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 18:31:12.994500 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989587 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 18:31:12.994500 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989590 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 18:31:12.994500 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989592 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 18:31:12.994500 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989595 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 18:31:12.994946 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989598 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 18:31:12.994946 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989600 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 18:31:12.994946 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989602 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 18:31:12.994946 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989605 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 18:31:12.994946 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989607 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 18:31:12.994946 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989632 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 18:31:12.994946 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989635 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 18:31:12.994946 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989637 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 18:31:12.994946 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989640 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 18:31:12.994946 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989642 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 18:31:12.994946 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989645 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 18:31:12.994946 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989648 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 18:31:12.994946 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989650 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 18:31:12.994946 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989653 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 18:31:12.994946 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989656 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 18:31:12.994946 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989659 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 18:31:12.994946 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989661 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 18:31:12.994946 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989664 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 18:31:12.994946 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989666 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 18:31:12.994946 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989669 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 18:31:12.995441 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989671 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 18:31:12.995441 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989674 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 18:31:12.995441 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989676 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 18:31:12.995441 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989679 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 18:31:12.995441 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989681 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 18:31:12.995441 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989684 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 18:31:12.995441 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989686 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 18:31:12.995441 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989689 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 18:31:12.995441 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989691 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 18:31:12.995441 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989694 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 18:31:12.995441 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989696 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 18:31:12.995441 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989698 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 18:31:12.995441 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989701 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 18:31:12.995441 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989703 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 18:31:12.995441 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989706 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 18:31:12.995441 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989708 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 18:31:12.995441 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989710 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 18:31:12.995441 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989713 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 18:31:12.995441 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989715 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 18:31:12.995441 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989718 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 18:31:12.995957 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989720 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 18:31:12.995957 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989723 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 18:31:12.995957 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989726 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 18:31:12.995957 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989728 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 18:31:12.995957 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989731 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 18:31:12.995957 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989733 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 18:31:12.995957 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989736 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 18:31:12.995957 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989739 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 18:31:12.995957 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989741 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 18:31:12.995957 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989744 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 18:31:12.995957 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989746 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 18:31:12.995957 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989749 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 18:31:12.995957 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989751 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 18:31:12.995957 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989753 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 18:31:12.995957 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989756 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 18:31:12.995957 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989758 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 18:31:12.995957 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989760 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 18:31:12.995957 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989763 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 18:31:12.995957 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989765 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 18:31:12.995957 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989768 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 18:31:12.996431 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989770 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 18:31:12.996431 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989773 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 18:31:12.996431 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989775 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 18:31:12.996431 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989777 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 18:31:12.996431 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989780 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 18:31:12.996431 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989782 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 18:31:12.996431 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989785 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 18:31:12.996431 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989787 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 18:31:12.996431 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989790 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 18:31:12.996431 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989792 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 18:31:12.996431 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989795 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 18:31:12.996431 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989798 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 18:31:12.996431 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989800 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 18:31:12.996431 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989803 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 18:31:12.996431 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989806 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 18:31:12.996431 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989808 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 18:31:12.996431 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989811 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 18:31:12.996431 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989813 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 18:31:12.996431 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989816 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 18:31:12.996895 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989819 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 18:31:12.996895 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989824 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 18:31:12.996895 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989826 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 18:31:12.996895 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989830 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 18:31:12.996895 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989834 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 18:31:12.996895 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:12.989838 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 18:31:12.996895 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.989843 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 18:31:12.996895 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.989938 2581 server.go:962] "Client rotation is on, will bootstrap in background" Sep 10 18:31:12.996895 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.994700 2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 10 18:31:12.996895 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.995723 2581 server.go:1019] "Starting client certificate rotation" Sep 10 18:31:12.996895 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.995820 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 18:31:12.996895 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.995853 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 18:31:12.998045 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.998032 2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 18:31:12.999131 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:12.999113 2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 18:31:13.010859 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.010840 2581 log.go:25] "Validated CRI v1 runtime API" Sep 10 18:31:13.014426 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.014412 2581 log.go:25] "Validated CRI v1 image API" Sep 10 18:31:13.015361 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.015345 2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 10 18:31:13.020239 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.020217 2581 fs.go:135] Filesystem UUIDs: map[026d667c-d2fa-4575-875e-6949333a93d7:/dev/nvme0n1p3 49c76738-b6aa-41af-899f-3d76ce3c255d:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Sep 10 18:31:13.020302 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.020239 2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 10 18:31:13.022381 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.022354 2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 18:31:13.025487 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.025381 2581 manager.go:216] Machine: {Timestamp:2026-09-10 18:31:13.024180462 +0000 UTC m=+0.321229595 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3122528 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d01e7db8ced4212cfdd0cc87173e6 SystemUUID:ec2d01e7-db8c-ed42-12cf-dd0cc87173e6 BootID:f0b7293e-3512-4a50-8b84-528005e19d13 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:fc:fb:52:46:77 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:fc:fb:52:46:77 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:ea:e6:76:e4:a7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 10 18:31:13.025487 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.025482 2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 10 18:31:13.025628 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.025578 2581 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 10 18:31:13.027147 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.027125 2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 10 18:31:13.027300 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.027150 2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-166.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 10 18:31:13.027906 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.027892 2581 topology_manager.go:138] "Creating topology manager with none policy" Sep 10 18:31:13.027970 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.027912 2581 container_manager_linux.go:309] "Creating device plugin manager" Sep 10 18:31:13.027970 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.027922 2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 18:31:13.028653 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.028641 2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 18:31:13.030019 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.030008 2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 10 18:31:13.032828 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.032815 2581 state_mem.go:36] "Initialized new in-memory state store" Sep 10 18:31:13.032958 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.032947 2581 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 10 18:31:13.033024 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.032991 2581 kubelet.go:486] "Attempting to sync node with API server" Sep 10 18:31:13.033024 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.033013 2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 10 18:31:13.033115 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.033027 2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 10 18:31:13.033115 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.033041 2581 kubelet.go:398] "Adding apiserver pod source" Sep 10 18:31:13.033115 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.033053 2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 10 18:31:13.033809 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.033793 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 18:31:13.033884 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.033817 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 18:31:13.036944 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.036919 2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 10 18:31:13.039121 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.039107 2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 10 18:31:13.039197 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.039136 2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 10 18:31:13.039298 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.039281 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 10 18:31:13.039387 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.039305 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 10 18:31:13.039387 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.039313 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 10 18:31:13.039387 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.039319 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 10 18:31:13.039387 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.039326 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 10 18:31:13.039387 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.039331 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 10 18:31:13.039849 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.039839 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 10 18:31:13.039849 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.039849 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 10 18:31:13.039911 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.039855 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 10 18:31:13.039911 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.039860 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 10 18:31:13.039911 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.039868 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 10 18:31:13.039911 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.039876 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 10 18:31:13.040650 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.040640 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 10 18:31:13.040689 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.040652 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 10 18:31:13.043561 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.043547 2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-166.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 18:31:13.043634 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:13.043599 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 10 18:31:13.043769 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:13.043753 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-166.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 10 18:31:13.044263 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.044252 2581 server.go:1268] "Started kubelet" Sep 10 18:31:13.044389 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.044335 2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 10 18:31:13.044449 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.044335 2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 10 18:31:13.044449 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.044433 2581 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 10 18:31:13.045090 ip-10-0-134-166 systemd[1]: Started Kubernetes Kubelet. Sep 10 18:31:13.046383 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.046352 2581 server.go:310] "Adding debug handlers to kubelet server" Sep 10 18:31:13.048061 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.048043 2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 10 18:31:13.048134 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:13.048075 2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 10 18:31:13.051299 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.051284 2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 10 18:31:13.051341 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.051299 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 10 18:31:13.051726 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:13.051584 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-166.ec2.internal\" not found" Sep 10 18:31:13.051899 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.051881 2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 10 18:31:13.051989 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.051974 2581 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 10 18:31:13.052047 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.051993 2581 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 10 18:31:13.052094 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.052072 2581 reconstruct.go:97] "Volume reconstruction finished" Sep 10 18:31:13.052148 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.052100 2581 reconciler.go:29] "Reconciler: start to sync state" Sep 10 18:31:13.052214 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.052200 2581 factory.go:55] Registering systemd factory Sep 10 18:31:13.052262 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.052218 2581 factory.go:223] Registration of the systemd container factory successfully Sep 10 18:31:13.052441 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.052425 2581 factory.go:153] Registering CRI-O factory Sep 10 18:31:13.052441 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.052442 2581 factory.go:223] Registration of the crio container factory successfully Sep 10 18:31:13.052588 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.052484 2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 10 18:31:13.052588 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.052511 2581 factory.go:103] Registering Raw factory Sep 10 18:31:13.052588 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.052525 2581 manager.go:1196] Started watching for new ooms in manager Sep 10 18:31:13.052919 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.052905 2581 manager.go:349] Starting recovery of all containers Sep 10 18:31:13.055766 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.055745 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kgrkk" Sep 10 18:31:13.056580 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:13.056553 2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-166.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 10 18:31:13.057066 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:13.056940 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 10 18:31:13.058009 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:13.057087 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-166.ec2.internal.18d409347edc0578 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-166.ec2.internal,UID:ip-10-0-134-166.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-166.ec2.internal,},FirstTimestamp:2026-09-10 18:31:13.044227448 +0000 UTC m=+0.341276577,LastTimestamp:2026-09-10 18:31:13.044227448 +0000 UTC m=+0.341276577,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-166.ec2.internal,}" Sep 10 18:31:13.063485 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.063460 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 10 18:31:13.063582 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.063542 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kgrkk" Sep 10 18:31:13.065050 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.065033 2581 manager.go:354] Recovery completed Sep 10 18:31:13.069836 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.069822 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 18:31:13.071587 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.071570 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-166.ec2.internal" event="NodeHasSufficientMemory" Sep 10 18:31:13.071671 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.071607 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-166.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 18:31:13.071671 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.071662 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-166.ec2.internal" event="NodeHasSufficientPID" Sep 10 18:31:13.072177 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.072162 2581 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 10 18:31:13.072242 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.072178 2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 10 18:31:13.072242 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.072194 2581 state_mem.go:36] "Initialized new in-memory state store" Sep 10 18:31:13.072242 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.072207 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 18:31:13.072939 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.072921 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-166.ec2.internal" event="NodeHasSufficientMemory" Sep 10 18:31:13.073006 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.072951 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-166.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 18:31:13.073006 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.072960 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-166.ec2.internal" event="NodeHasSufficientPID" Sep 10 18:31:13.073234 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.073222 2581 policy_none.go:49] "None policy: Start" Sep 10 18:31:13.073269 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.073238 2581 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 10 18:31:13.073269 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.073248 2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 10 18:31:13.073818 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:13.073756 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-166.ec2.internal.18d40934807d862a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-166.ec2.internal,UID:ip-10-0-134-166.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-166.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-166.ec2.internal,},FirstTimestamp:2026-09-10 18:31:13.071588906 +0000 UTC m=+0.368638041,LastTimestamp:2026-09-10 18:31:13.071588906 +0000 UTC m=+0.368638041,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-166.ec2.internal,}" Sep 10 18:31:13.074305 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.074295 2581 policy_none.go:47] "Start" Sep 10 18:31:13.109253 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.109238 2581 manager.go:337] "Starting Device Plugin manager" Sep 10 18:31:13.109360 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:13.109282 2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 10 18:31:13.109360 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.109294 2581 server.go:87] "Starting device plugin registration server" Sep 10 18:31:13.109537 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.109523 2581 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 10 18:31:13.109585 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.109540 2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 10 18:31:13.109756 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.109653 2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 10 18:31:13.109756 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.109718 2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 10 18:31:13.109756 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.109726 2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 10 18:31:13.111721 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:13.111580 2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 10 18:31:13.111721 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:13.111671 2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-166.ec2.internal\" not found" Sep 10 18:31:13.163135 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.163113 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 10 18:31:13.163272 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.163167 2581 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 10 18:31:13.163272 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.163187 2581 kubelet.go:2448] "Starting kubelet main sync loop" Sep 10 18:31:13.163272 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:13.163221 2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 10 18:31:13.165487 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.165466 2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 18:31:13.210454 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.210435 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 18:31:13.211200 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.211184 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-166.ec2.internal" event="NodeHasSufficientMemory" Sep 10 18:31:13.211269 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.211217 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-166.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 18:31:13.211269 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.211230 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-166.ec2.internal" event="NodeHasSufficientPID" Sep 10 18:31:13.211269 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.211251 2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-166.ec2.internal" Sep 10 18:31:13.218708 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.218694 2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-166.ec2.internal" Sep 10 18:31:13.218754 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:13.218713 2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-166.ec2.internal\": node \"ip-10-0-134-166.ec2.internal\" not found" Sep 10 18:31:13.234801 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:13.234733 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-166.ec2.internal\" not found" Sep 10 18:31:13.264173 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.264155 2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-166.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-166.ec2.internal"] Sep 10 18:31:13.264238 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.264209 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 18:31:13.265319 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.265304 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-166.ec2.internal" event="NodeHasSufficientMemory" Sep 10 18:31:13.265389 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.265331 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-166.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 18:31:13.265389 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.265340 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-166.ec2.internal" event="NodeHasSufficientPID" Sep 10 18:31:13.266662 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.266650 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 18:31:13.266761 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.266746 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-166.ec2.internal" Sep 10 18:31:13.266805 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.266772 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 18:31:13.267408 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.267394 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-166.ec2.internal" event="NodeHasSufficientMemory" Sep 10 18:31:13.267469 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.267406 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-166.ec2.internal" event="NodeHasSufficientMemory" Sep 10 18:31:13.267469 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.267419 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-166.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 18:31:13.267469 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.267425 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-166.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 18:31:13.267469 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.267434 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-166.ec2.internal" event="NodeHasSufficientPID" Sep 10 18:31:13.267469 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.267435 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-166.ec2.internal" event="NodeHasSufficientPID" Sep 10 18:31:13.268632 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.268593 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-166.ec2.internal" Sep 10 18:31:13.268708 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.268642 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 18:31:13.269270 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.269248 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-166.ec2.internal" event="NodeHasSufficientMemory" Sep 10 18:31:13.269343 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.269282 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-166.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 18:31:13.269343 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.269297 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-166.ec2.internal" event="NodeHasSufficientPID" Sep 10 18:31:13.298150 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:13.298128 2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-166.ec2.internal\" not found" node="ip-10-0-134-166.ec2.internal" Sep 10 18:31:13.301392 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:13.301376 2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-166.ec2.internal\" not found" node="ip-10-0-134-166.ec2.internal" Sep 10 18:31:13.335717 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:13.335699 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-166.ec2.internal\" not found" Sep 10 18:31:13.435983 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:13.435963 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-166.ec2.internal\" not found" Sep 10 18:31:13.453295 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.453265 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/28f1ecabb2c30a2edb71680a33548326-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-166.ec2.internal\" (UID: \"28f1ecabb2c30a2edb71680a33548326\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-166.ec2.internal" Sep 10 18:31:13.453347 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.453302 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28f1ecabb2c30a2edb71680a33548326-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-166.ec2.internal\" (UID: \"28f1ecabb2c30a2edb71680a33548326\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-166.ec2.internal" Sep 10 18:31:13.453347 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.453329 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a6bd606f49b0efaf64eed63f0b2241b9-config\") pod \"kube-apiserver-proxy-ip-10-0-134-166.ec2.internal\" (UID: \"a6bd606f49b0efaf64eed63f0b2241b9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-166.ec2.internal" Sep 10 18:31:13.536708 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:13.536650 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-166.ec2.internal\" not found" Sep 10 18:31:13.553996 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.553978 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/28f1ecabb2c30a2edb71680a33548326-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-166.ec2.internal\" (UID: \"28f1ecabb2c30a2edb71680a33548326\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-166.ec2.internal" Sep 10 18:31:13.554038 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.554001 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28f1ecabb2c30a2edb71680a33548326-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-166.ec2.internal\" (UID: \"28f1ecabb2c30a2edb71680a33548326\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-166.ec2.internal" Sep 10 18:31:13.554038 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.554018 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a6bd606f49b0efaf64eed63f0b2241b9-config\") pod \"kube-apiserver-proxy-ip-10-0-134-166.ec2.internal\" (UID: \"a6bd606f49b0efaf64eed63f0b2241b9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-166.ec2.internal" Sep 10 18:31:13.554103 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.554060 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a6bd606f49b0efaf64eed63f0b2241b9-config\") pod \"kube-apiserver-proxy-ip-10-0-134-166.ec2.internal\" (UID: \"a6bd606f49b0efaf64eed63f0b2241b9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-166.ec2.internal" Sep 10 18:31:13.554103 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.554070 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/28f1ecabb2c30a2edb71680a33548326-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-166.ec2.internal\" (UID: \"28f1ecabb2c30a2edb71680a33548326\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-166.ec2.internal" Sep 10 18:31:13.554103 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.554078 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28f1ecabb2c30a2edb71680a33548326-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-166.ec2.internal\" (UID: \"28f1ecabb2c30a2edb71680a33548326\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-166.ec2.internal" Sep 10 18:31:13.601105 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.601081 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-166.ec2.internal" Sep 10 18:31:13.603682 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.603664 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-166.ec2.internal" Sep 10 18:31:13.637481 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:13.637459 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-166.ec2.internal\" not found" Sep 10 18:31:13.738065 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:13.738043 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-166.ec2.internal\" not found" Sep 10 18:31:13.838509 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:13.838489 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-166.ec2.internal\" not found" Sep 10 18:31:13.939082 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:13.939058 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-166.ec2.internal\" not found" Sep 10 18:31:13.995513 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.995489 2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 10 18:31:13.995915 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:13.995661 2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 18:31:14.040020 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:14.039992 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-166.ec2.internal\" not found" Sep 10 18:31:14.065997 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:14.065967 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 18:26:14 +0000 UTC" deadline="2028-05-21 07:42:53.85092674 +0000 UTC" Sep 10 18:31:14.065997 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:14.065993 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14845h11m39.78493684s" Sep 10 18:31:14.069412 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:14.069393 2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 18:31:14.103060 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:14.103005 2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 18:31:14.151846 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:14.151825 2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-166.ec2.internal" Sep 10 18:31:14.162059 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:14.162043 2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 18:31:14.163138 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:14.163126 2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-166.ec2.internal" Sep 10 18:31:14.168332 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:14.168317 2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 18:31:14.183721 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:14.183697 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28f1ecabb2c30a2edb71680a33548326.slice/crio-1e2e29a4ad265502435b4fa9a7f7d8fd6200fb89b348a79e04b5a7f972f19433 WatchSource:0}: Error finding container 1e2e29a4ad265502435b4fa9a7f7d8fd6200fb89b348a79e04b5a7f972f19433: Status 404 returned error can't find the container with id 1e2e29a4ad265502435b4fa9a7f7d8fd6200fb89b348a79e04b5a7f972f19433 Sep 10 18:31:14.184088 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:14.184067 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6bd606f49b0efaf64eed63f0b2241b9.slice/crio-f2f7a29f376c2e42a9fd2b30e8dd8cbc04797eefdab7c3054dcaf1c81d1f4a72 WatchSource:0}: Error finding container f2f7a29f376c2e42a9fd2b30e8dd8cbc04797eefdab7c3054dcaf1c81d1f4a72: Status 404 returned error can't find the container with id f2f7a29f376c2e42a9fd2b30e8dd8cbc04797eefdab7c3054dcaf1c81d1f4a72 Sep 10 18:31:14.189001 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:14.188985 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 18:31:14.216800 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:14.216779 2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 18:31:16.245794 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.245548 2581 apiserver.go:52] "Watching apiserver" Sep 10 18:31:16.250421 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.250394 2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 10 18:31:16.250797 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.250773 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-cznnw","openshift-network-operator/iptables-alerter-hxctp","kube-system/global-pull-secret-syncer-df44c","openshift-dns/node-resolver-gf28x","openshift-image-registry/node-ca-lh8gr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-166.ec2.internal","openshift-multus/multus-additional-cni-plugins-qrb6h","openshift-multus/multus-jc45h","openshift-multus/network-metrics-daemon-fqmzj","openshift-ovn-kubernetes/ovnkube-node-9j6cn","kube-system/konnectivity-agent-4g45x","kube-system/kube-apiserver-proxy-ip-10-0-134-166.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zpc6","openshift-cluster-node-tuning-operator/tuned-jcwmv"] Sep 10 18:31:16.258948 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.258928 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qrb6h" Sep 10 18:31:16.261243 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.261061 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 10 18:31:16.261243 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.261074 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 10 18:31:16.261243 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.261083 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-r9zwv\"" Sep 10 18:31:16.261243 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.261130 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 10 18:31:16.261243 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.261157 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 10 18:31:16.261512 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.261249 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 10 18:31:16.262633 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.262618 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 10 18:31:16.265503 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.265449 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zpc6" Sep 10 18:31:16.267450 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.267431 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 10 18:31:16.267556 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.267538 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fz55j\"" Sep 10 18:31:16.267556 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.267544 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 10 18:31:16.267664 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.267637 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 10 18:31:16.269289 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.269013 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-df44c" Sep 10 18:31:16.269289 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:16.269086 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-df44c" podUID="0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9" Sep 10 18:31:16.271510 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.271491 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/747cbe9d-036e-448f-b567-6ffd3c917b2e-cni-binary-copy\") pod \"multus-additional-cni-plugins-qrb6h\" (UID: \"747cbe9d-036e-448f-b567-6ffd3c917b2e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb6h" Sep 10 18:31:16.271602 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.271520 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/747cbe9d-036e-448f-b567-6ffd3c917b2e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qrb6h\" (UID: \"747cbe9d-036e-448f-b567-6ffd3c917b2e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb6h" Sep 10 18:31:16.271602 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.271537 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/898e6b12-a7af-4d34-af96-b6c1f54e66c5-sys-fs\") pod \"aws-ebs-csi-driver-node-8zpc6\" (UID: \"898e6b12-a7af-4d34-af96-b6c1f54e66c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zpc6" Sep 10 18:31:16.271602 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.271552 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/747cbe9d-036e-448f-b567-6ffd3c917b2e-system-cni-dir\") pod \"multus-additional-cni-plugins-qrb6h\" (UID: \"747cbe9d-036e-448f-b567-6ffd3c917b2e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb6h" Sep 10 18:31:16.271602 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.271572 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/747cbe9d-036e-448f-b567-6ffd3c917b2e-cnibin\") pod \"multus-additional-cni-plugins-qrb6h\" (UID: \"747cbe9d-036e-448f-b567-6ffd3c917b2e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb6h" Sep 10 18:31:16.271602 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.271593 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/747cbe9d-036e-448f-b567-6ffd3c917b2e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qrb6h\" (UID: \"747cbe9d-036e-448f-b567-6ffd3c917b2e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb6h" Sep 10 18:31:16.271843 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.271615 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npm9h\" (UniqueName: \"kubernetes.io/projected/747cbe9d-036e-448f-b567-6ffd3c917b2e-kube-api-access-npm9h\") pod \"multus-additional-cni-plugins-qrb6h\" (UID: \"747cbe9d-036e-448f-b567-6ffd3c917b2e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb6h" Sep 10 18:31:16.271843 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.271634 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/898e6b12-a7af-4d34-af96-b6c1f54e66c5-socket-dir\") pod \"aws-ebs-csi-driver-node-8zpc6\" (UID: \"898e6b12-a7af-4d34-af96-b6c1f54e66c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zpc6" Sep 10 18:31:16.271843 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.271659 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/747cbe9d-036e-448f-b567-6ffd3c917b2e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qrb6h\" (UID: \"747cbe9d-036e-448f-b567-6ffd3c917b2e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb6h" Sep 10 18:31:16.271843 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.271680 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/898e6b12-a7af-4d34-af96-b6c1f54e66c5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8zpc6\" (UID: \"898e6b12-a7af-4d34-af96-b6c1f54e66c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zpc6" Sep 10 18:31:16.271843 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.271694 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/898e6b12-a7af-4d34-af96-b6c1f54e66c5-registration-dir\") pod \"aws-ebs-csi-driver-node-8zpc6\" (UID: \"898e6b12-a7af-4d34-af96-b6c1f54e66c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zpc6" Sep 10 18:31:16.271843 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.271709 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/898e6b12-a7af-4d34-af96-b6c1f54e66c5-device-dir\") pod \"aws-ebs-csi-driver-node-8zpc6\" (UID: \"898e6b12-a7af-4d34-af96-b6c1f54e66c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zpc6" Sep 10 18:31:16.271843 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.271723 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/898e6b12-a7af-4d34-af96-b6c1f54e66c5-etc-selinux\") pod \"aws-ebs-csi-driver-node-8zpc6\" (UID: \"898e6b12-a7af-4d34-af96-b6c1f54e66c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zpc6" Sep 10 18:31:16.271843 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.271738 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxjmk\" (UniqueName: \"kubernetes.io/projected/898e6b12-a7af-4d34-af96-b6c1f54e66c5-kube-api-access-fxjmk\") pod \"aws-ebs-csi-driver-node-8zpc6\" (UID: \"898e6b12-a7af-4d34-af96-b6c1f54e66c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zpc6" Sep 10 18:31:16.272766 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.272749 2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 18:31:16.275451 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.275424 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gf28x" Sep 10 18:31:16.278063 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.277851 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 10 18:31:16.278063 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.277877 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 10 18:31:16.278063 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.277893 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6fc2f\"" Sep 10 18:31:16.282243 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.281272 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lh8gr" Sep 10 18:31:16.283629 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.283611 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.284420 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.284404 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 10 18:31:16.285790 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.285769 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 10 18:31:16.286104 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.286083 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s2hdg\"" Sep 10 18:31:16.286249 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.284412 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 10 18:31:16.286758 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.286739 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 10 18:31:16.287025 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.287010 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-sqmnq\"" Sep 10 18:31:16.287419 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.287402 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 10 18:31:16.288563 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.288246 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmzj" Sep 10 18:31:16.288563 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:16.288342 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqmzj" podUID="54ef3c93-1624-46e6-ba09-4fd3bb1fc07f" Sep 10 18:31:16.290441 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.290412 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cznnw" Sep 10 18:31:16.290518 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:16.290472 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cznnw" podUID="a43748df-f7e9-4e22-b659-34b9550d706e" Sep 10 18:31:16.295492 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.295473 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hxctp" Sep 10 18:31:16.297312 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.297294 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wnxd8\"" Sep 10 18:31:16.297456 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.297309 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 10 18:31:16.297521 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.297495 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 10 18:31:16.298199 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.298179 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 10 18:31:16.299875 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.299857 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-tk96w" Sep 10 18:31:16.303755 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.303733 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.305595 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.305575 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 10 18:31:16.306022 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.306005 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tk96w" Sep 10 18:31:16.306088 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.306074 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 10 18:31:16.306276 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.306258 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 10 18:31:16.306418 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.306265 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 10 18:31:16.306418 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.306363 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5qkpr\"" Sep 10 18:31:16.306533 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.306442 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 10 18:31:16.306533 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.306467 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 10 18:31:16.314644 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.314623 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4g45x" Sep 10 18:31:16.316561 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.316543 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 10 18:31:16.316791 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.316748 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 10 18:31:16.316861 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.316801 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-cxsx9\"" Sep 10 18:31:16.324137 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.324118 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.325741 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.325722 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-z6k6k\"" Sep 10 18:31:16.325950 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.325937 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 10 18:31:16.363573 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.363557 2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 10 18:31:16.372192 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.372172 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-run-ovn\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.372288 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.372200 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bd3aed00-8ff3-4247-abc7-4893cfae63dc-konnectivity-ca\") pod \"konnectivity-agent-4g45x\" (UID: \"bd3aed00-8ff3-4247-abc7-4893cfae63dc\") " pod="kube-system/konnectivity-agent-4g45x" Sep 10 18:31:16.372288 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.372226 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-cnibin\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.372288 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.372241 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-multus-conf-dir\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.372288 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.372257 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9eac62b3-150f-42a5-9af2-bf313bf92907-ovn-node-metrics-cert\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.372288 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.372271 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54ef3c93-1624-46e6-ba09-4fd3bb1fc07f-metrics-certs\") pod \"network-metrics-daemon-fqmzj\" (UID: \"54ef3c93-1624-46e6-ba09-4fd3bb1fc07f\") " pod="openshift-multus/network-metrics-daemon-fqmzj" Sep 10 18:31:16.372288 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.372283 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-host-cni-netd\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.372585 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.372295 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-etc-modprobe-d\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.372585 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.372312 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-etc-kubernetes\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.372585 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.372373 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-etc-systemd\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.372585 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.372408 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-host-run-k8s-cni-cncf-io\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.372585 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.372439 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/747cbe9d-036e-448f-b567-6ffd3c917b2e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qrb6h\" (UID: \"747cbe9d-036e-448f-b567-6ffd3c917b2e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb6h" Sep 10 18:31:16.372585 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.372463 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbwkl\" (UniqueName: \"kubernetes.io/projected/dc8250af-b1fc-4108-b6fd-092198a098ac-kube-api-access-nbwkl\") pod \"node-ca-lh8gr\" (UID: \"dc8250af-b1fc-4108-b6fd-092198a098ac\") " pod="openshift-image-registry/node-ca-lh8gr" Sep 10 18:31:16.372585 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.372486 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-var-lib-kubelet\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.372585 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.372527 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-etc-tuned\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.372869 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.372604 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-etc-openvswitch\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.372869 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.372629 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9eac62b3-150f-42a5-9af2-bf313bf92907-env-overrides\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.372869 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.372655 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9eac62b3-150f-42a5-9af2-bf313bf92907-ovnkube-script-lib\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.372869 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.372674 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/898e6b12-a7af-4d34-af96-b6c1f54e66c5-etc-selinux\") pod \"aws-ebs-csi-driver-node-8zpc6\" (UID: \"898e6b12-a7af-4d34-af96-b6c1f54e66c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zpc6" Sep 10 18:31:16.372869 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.372689 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-sys\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.372869 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.372713 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-lib-modules\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.372869 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.372727 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcz8h\" (UniqueName: \"kubernetes.io/projected/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-kube-api-access-jcz8h\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.372869 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.372748 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/747cbe9d-036e-448f-b567-6ffd3c917b2e-cni-binary-copy\") pod \"multus-additional-cni-plugins-qrb6h\" (UID: \"747cbe9d-036e-448f-b567-6ffd3c917b2e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb6h" Sep 10 18:31:16.372869 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.372790 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/70d9c337-84ae-485f-be5d-9a7c3919f3b0-multus-daemon-config\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.372869 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.372798 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/898e6b12-a7af-4d34-af96-b6c1f54e66c5-etc-selinux\") pod \"aws-ebs-csi-driver-node-8zpc6\" (UID: \"898e6b12-a7af-4d34-af96-b6c1f54e66c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zpc6" Sep 10 18:31:16.372869 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.372831 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dc8250af-b1fc-4108-b6fd-092198a098ac-serviceca\") pod \"node-ca-lh8gr\" (UID: \"dc8250af-b1fc-4108-b6fd-092198a098ac\") " pod="openshift-image-registry/node-ca-lh8gr" Sep 10 18:31:16.372869 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.372855 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-host-kubelet\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.373419 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.372878 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bd3aed00-8ff3-4247-abc7-4893cfae63dc-agent-certs\") pod \"konnectivity-agent-4g45x\" (UID: \"bd3aed00-8ff3-4247-abc7-4893cfae63dc\") " pod="kube-system/konnectivity-agent-4g45x" Sep 10 18:31:16.373419 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.372908 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/747cbe9d-036e-448f-b567-6ffd3c917b2e-system-cni-dir\") pod \"multus-additional-cni-plugins-qrb6h\" (UID: \"747cbe9d-036e-448f-b567-6ffd3c917b2e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb6h" Sep 10 18:31:16.373419 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.372932 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-systemd-units\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.373419 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.372954 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsdgh\" (UniqueName: \"kubernetes.io/projected/9eac62b3-150f-42a5-9af2-bf313bf92907-kube-api-access-tsdgh\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.373419 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.372978 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5l6n\" (UniqueName: \"kubernetes.io/projected/54ef3c93-1624-46e6-ba09-4fd3bb1fc07f-kube-api-access-v5l6n\") pod \"network-metrics-daemon-fqmzj\" (UID: \"54ef3c93-1624-46e6-ba09-4fd3bb1fc07f\") " pod="openshift-multus/network-metrics-daemon-fqmzj" Sep 10 18:31:16.373419 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.372988 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/747cbe9d-036e-448f-b567-6ffd3c917b2e-system-cni-dir\") pod \"multus-additional-cni-plugins-qrb6h\" (UID: \"747cbe9d-036e-448f-b567-6ffd3c917b2e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb6h" Sep 10 18:31:16.373419 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.373003 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/898e6b12-a7af-4d34-af96-b6c1f54e66c5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8zpc6\" (UID: \"898e6b12-a7af-4d34-af96-b6c1f54e66c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zpc6" Sep 10 18:31:16.373419 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.373030 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/898e6b12-a7af-4d34-af96-b6c1f54e66c5-registration-dir\") pod \"aws-ebs-csi-driver-node-8zpc6\" (UID: \"898e6b12-a7af-4d34-af96-b6c1f54e66c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zpc6" Sep 10 18:31:16.373419 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.373034 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/747cbe9d-036e-448f-b567-6ffd3c917b2e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qrb6h\" (UID: \"747cbe9d-036e-448f-b567-6ffd3c917b2e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb6h" Sep 10 18:31:16.373419 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.373053 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-host-run-netns\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.373419 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.373079 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/898e6b12-a7af-4d34-af96-b6c1f54e66c5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8zpc6\" (UID: \"898e6b12-a7af-4d34-af96-b6c1f54e66c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zpc6" Sep 10 18:31:16.373419 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.373089 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-log-socket\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.373419 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.373123 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/898e6b12-a7af-4d34-af96-b6c1f54e66c5-registration-dir\") pod \"aws-ebs-csi-driver-node-8zpc6\" (UID: \"898e6b12-a7af-4d34-af96-b6c1f54e66c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zpc6" Sep 10 18:31:16.373419 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.373169 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-var-lib-openvswitch\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.373419 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.373196 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9eac62b3-150f-42a5-9af2-bf313bf92907-ovnkube-config\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.373419 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.373209 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/747cbe9d-036e-448f-b567-6ffd3c917b2e-cni-binary-copy\") pod \"multus-additional-cni-plugins-qrb6h\" (UID: \"747cbe9d-036e-448f-b567-6ffd3c917b2e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb6h" Sep 10 18:31:16.374078 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.373220 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-etc-sysctl-d\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.374078 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.373244 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-host-var-lib-cni-bin\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.374078 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.373267 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-host-var-lib-cni-multus\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.374078 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.373291 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-hostroot\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.374078 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.373313 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-etc-kubernetes\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.374078 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.373351 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-host-run-netns\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.374078 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.373401 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-host-run-ovn-kubernetes\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.374078 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.373432 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-etc-sysctl-conf\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.374078 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.373469 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-multus-socket-dir-parent\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.374078 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.373516 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-node-log\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.374078 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.373539 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-tmp\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.374078 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.373563 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhxb6\" (UniqueName: \"kubernetes.io/projected/a43748df-f7e9-4e22-b659-34b9550d706e-kube-api-access-xhxb6\") pod \"network-check-target-cznnw\" (UID: \"a43748df-f7e9-4e22-b659-34b9550d706e\") " pod="openshift-network-diagnostics/network-check-target-cznnw" Sep 10 18:31:16.374078 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.373622 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/747cbe9d-036e-448f-b567-6ffd3c917b2e-cnibin\") pod \"multus-additional-cni-plugins-qrb6h\" (UID: \"747cbe9d-036e-448f-b567-6ffd3c917b2e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb6h" Sep 10 18:31:16.374078 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.373646 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvg8v\" (UniqueName: \"kubernetes.io/projected/d84d2e56-1f36-4691-8608-a1af00d9cce5-kube-api-access-gvg8v\") pod \"iptables-alerter-hxctp\" (UID: \"d84d2e56-1f36-4691-8608-a1af00d9cce5\") " pod="openshift-network-operator/iptables-alerter-hxctp" Sep 10 18:31:16.374078 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.373686 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70d9c337-84ae-485f-be5d-9a7c3919f3b0-cni-binary-copy\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.374078 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.373680 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/747cbe9d-036e-448f-b567-6ffd3c917b2e-cnibin\") pod \"multus-additional-cni-plugins-qrb6h\" (UID: \"747cbe9d-036e-448f-b567-6ffd3c917b2e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb6h" Sep 10 18:31:16.374078 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.373711 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcxb5\" (UniqueName: \"kubernetes.io/projected/70d9c337-84ae-485f-be5d-9a7c3919f3b0-kube-api-access-pcxb5\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.374868 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.373743 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-run-openvswitch\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.374868 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.373777 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.374868 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.373806 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-run\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.374868 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.373832 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/747cbe9d-036e-448f-b567-6ffd3c917b2e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qrb6h\" (UID: \"747cbe9d-036e-448f-b567-6ffd3c917b2e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb6h" Sep 10 18:31:16.374868 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.373857 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d84d2e56-1f36-4691-8608-a1af00d9cce5-iptables-alerter-script\") pod \"iptables-alerter-hxctp\" (UID: \"d84d2e56-1f36-4691-8608-a1af00d9cce5\") " pod="openshift-network-operator/iptables-alerter-hxctp" Sep 10 18:31:16.374868 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.373968 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn42b\" (UniqueName: \"kubernetes.io/projected/4d70e39b-2b94-4593-828b-75d5d071560e-kube-api-access-nn42b\") pod \"node-resolver-gf28x\" (UID: \"4d70e39b-2b94-4593-828b-75d5d071560e\") " pod="openshift-dns/node-resolver-gf28x" Sep 10 18:31:16.374868 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.373994 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9-dbus\") pod \"global-pull-secret-syncer-df44c\" (UID: \"0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9\") " pod="kube-system/global-pull-secret-syncer-df44c" Sep 10 18:31:16.374868 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.374020 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/898e6b12-a7af-4d34-af96-b6c1f54e66c5-device-dir\") pod \"aws-ebs-csi-driver-node-8zpc6\" (UID: \"898e6b12-a7af-4d34-af96-b6c1f54e66c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zpc6" Sep 10 18:31:16.374868 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.374044 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxjmk\" (UniqueName: \"kubernetes.io/projected/898e6b12-a7af-4d34-af96-b6c1f54e66c5-kube-api-access-fxjmk\") pod \"aws-ebs-csi-driver-node-8zpc6\" (UID: \"898e6b12-a7af-4d34-af96-b6c1f54e66c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zpc6" Sep 10 18:31:16.374868 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.374067 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-system-cni-dir\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.374868 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.374087 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-multus-cni-dir\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.374868 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.374110 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4d70e39b-2b94-4593-828b-75d5d071560e-hosts-file\") pod \"node-resolver-gf28x\" (UID: \"4d70e39b-2b94-4593-828b-75d5d071560e\") " pod="openshift-dns/node-resolver-gf28x" Sep 10 18:31:16.374868 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.374134 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9-original-pull-secret\") pod \"global-pull-secret-syncer-df44c\" (UID: \"0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9\") " pod="kube-system/global-pull-secret-syncer-df44c" Sep 10 18:31:16.374868 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.374155 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-host\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.374868 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.374179 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/747cbe9d-036e-448f-b567-6ffd3c917b2e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qrb6h\" (UID: \"747cbe9d-036e-448f-b567-6ffd3c917b2e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb6h" Sep 10 18:31:16.374868 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.374217 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/898e6b12-a7af-4d34-af96-b6c1f54e66c5-sys-fs\") pod \"aws-ebs-csi-driver-node-8zpc6\" (UID: \"898e6b12-a7af-4d34-af96-b6c1f54e66c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zpc6" Sep 10 18:31:16.375483 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.374247 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dc8250af-b1fc-4108-b6fd-092198a098ac-host\") pod \"node-ca-lh8gr\" (UID: \"dc8250af-b1fc-4108-b6fd-092198a098ac\") " pod="openshift-image-registry/node-ca-lh8gr" Sep 10 18:31:16.375483 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.374262 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-etc-sysconfig\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.375483 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.374279 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npm9h\" (UniqueName: \"kubernetes.io/projected/747cbe9d-036e-448f-b567-6ffd3c917b2e-kube-api-access-npm9h\") pod \"multus-additional-cni-plugins-qrb6h\" (UID: \"747cbe9d-036e-448f-b567-6ffd3c917b2e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb6h" Sep 10 18:31:16.375483 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.374285 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/747cbe9d-036e-448f-b567-6ffd3c917b2e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qrb6h\" (UID: \"747cbe9d-036e-448f-b567-6ffd3c917b2e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb6h" Sep 10 18:31:16.375483 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.374294 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/898e6b12-a7af-4d34-af96-b6c1f54e66c5-socket-dir\") pod \"aws-ebs-csi-driver-node-8zpc6\" (UID: \"898e6b12-a7af-4d34-af96-b6c1f54e66c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zpc6" Sep 10 18:31:16.375483 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.374309 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-host-var-lib-kubelet\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.375483 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.374345 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-host-cni-bin\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.375483 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.374376 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9-kubelet-config\") pod \"global-pull-secret-syncer-df44c\" (UID: \"0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9\") " pod="kube-system/global-pull-secret-syncer-df44c" Sep 10 18:31:16.375483 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.374385 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/747cbe9d-036e-448f-b567-6ffd3c917b2e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qrb6h\" (UID: \"747cbe9d-036e-448f-b567-6ffd3c917b2e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb6h" Sep 10 18:31:16.375483 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.374431 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/898e6b12-a7af-4d34-af96-b6c1f54e66c5-device-dir\") pod \"aws-ebs-csi-driver-node-8zpc6\" (UID: \"898e6b12-a7af-4d34-af96-b6c1f54e66c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zpc6" Sep 10 18:31:16.375483 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.374535 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d84d2e56-1f36-4691-8608-a1af00d9cce5-host-slash\") pod \"iptables-alerter-hxctp\" (UID: \"d84d2e56-1f36-4691-8608-a1af00d9cce5\") " pod="openshift-network-operator/iptables-alerter-hxctp" Sep 10 18:31:16.375483 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.374545 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/898e6b12-a7af-4d34-af96-b6c1f54e66c5-sys-fs\") pod \"aws-ebs-csi-driver-node-8zpc6\" (UID: \"898e6b12-a7af-4d34-af96-b6c1f54e66c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zpc6" Sep 10 18:31:16.375483 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.374563 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-host-run-multus-certs\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.375483 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.374588 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d70e39b-2b94-4593-828b-75d5d071560e-tmp-dir\") pod \"node-resolver-gf28x\" (UID: \"4d70e39b-2b94-4593-828b-75d5d071560e\") " pod="openshift-dns/node-resolver-gf28x" Sep 10 18:31:16.375483 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.374633 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-host-slash\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.375483 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.374662 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-run-systemd\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.375483 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.374762 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/898e6b12-a7af-4d34-af96-b6c1f54e66c5-socket-dir\") pod \"aws-ebs-csi-driver-node-8zpc6\" (UID: \"898e6b12-a7af-4d34-af96-b6c1f54e66c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zpc6" Sep 10 18:31:16.380583 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.380236 2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 10 18:31:16.381392 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.381347 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-166.ec2.internal" event={"ID":"28f1ecabb2c30a2edb71680a33548326","Type":"ContainerStarted","Data":"1e2e29a4ad265502435b4fa9a7f7d8fd6200fb89b348a79e04b5a7f972f19433"} Sep 10 18:31:16.382839 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.382814 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-166.ec2.internal" event={"ID":"a6bd606f49b0efaf64eed63f0b2241b9","Type":"ContainerStarted","Data":"f2f7a29f376c2e42a9fd2b30e8dd8cbc04797eefdab7c3054dcaf1c81d1f4a72"} Sep 10 18:31:16.384417 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.384388 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxjmk\" (UniqueName: \"kubernetes.io/projected/898e6b12-a7af-4d34-af96-b6c1f54e66c5-kube-api-access-fxjmk\") pod \"aws-ebs-csi-driver-node-8zpc6\" (UID: \"898e6b12-a7af-4d34-af96-b6c1f54e66c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zpc6" Sep 10 18:31:16.384496 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.384427 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npm9h\" (UniqueName: \"kubernetes.io/projected/747cbe9d-036e-448f-b567-6ffd3c917b2e-kube-api-access-npm9h\") pod \"multus-additional-cni-plugins-qrb6h\" (UID: \"747cbe9d-036e-448f-b567-6ffd3c917b2e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb6h" Sep 10 18:31:16.475043 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475010 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-host-cni-netd\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.475043 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475047 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-etc-modprobe-d\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.475306 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475070 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-etc-kubernetes\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.475306 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475095 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-etc-systemd\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.475306 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475119 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-host-run-k8s-cni-cncf-io\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.475306 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475125 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-host-cni-netd\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.475306 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475142 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbwkl\" (UniqueName: \"kubernetes.io/projected/dc8250af-b1fc-4108-b6fd-092198a098ac-kube-api-access-nbwkl\") pod \"node-ca-lh8gr\" (UID: \"dc8250af-b1fc-4108-b6fd-092198a098ac\") " pod="openshift-image-registry/node-ca-lh8gr" Sep 10 18:31:16.475306 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475155 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-etc-kubernetes\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.475306 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475166 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-etc-systemd\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.475306 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475180 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-var-lib-kubelet\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.475306 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475184 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-etc-modprobe-d\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.475306 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475206 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-etc-tuned\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.475306 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475215 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-host-run-k8s-cni-cncf-io\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.475306 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475231 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-etc-openvswitch\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.475306 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475237 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-var-lib-kubelet\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.475306 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475268 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9eac62b3-150f-42a5-9af2-bf313bf92907-env-overrides\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.475306 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475273 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-etc-openvswitch\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.475306 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475294 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9eac62b3-150f-42a5-9af2-bf313bf92907-ovnkube-script-lib\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.476105 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475336 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-sys\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.476105 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475363 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-lib-modules\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.476105 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475386 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcz8h\" (UniqueName: \"kubernetes.io/projected/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-kube-api-access-jcz8h\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.476105 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475409 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/70d9c337-84ae-485f-be5d-9a7c3919f3b0-multus-daemon-config\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.476105 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475427 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dc8250af-b1fc-4108-b6fd-092198a098ac-serviceca\") pod \"node-ca-lh8gr\" (UID: \"dc8250af-b1fc-4108-b6fd-092198a098ac\") " pod="openshift-image-registry/node-ca-lh8gr" Sep 10 18:31:16.476105 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475445 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-host-kubelet\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.476105 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475465 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bd3aed00-8ff3-4247-abc7-4893cfae63dc-agent-certs\") pod \"konnectivity-agent-4g45x\" (UID: \"bd3aed00-8ff3-4247-abc7-4893cfae63dc\") " pod="kube-system/konnectivity-agent-4g45x" Sep 10 18:31:16.476105 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475490 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-systemd-units\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.476105 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475503 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-lib-modules\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.476105 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475512 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsdgh\" (UniqueName: \"kubernetes.io/projected/9eac62b3-150f-42a5-9af2-bf313bf92907-kube-api-access-tsdgh\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.476105 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475558 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5l6n\" (UniqueName: \"kubernetes.io/projected/54ef3c93-1624-46e6-ba09-4fd3bb1fc07f-kube-api-access-v5l6n\") pod \"network-metrics-daemon-fqmzj\" (UID: \"54ef3c93-1624-46e6-ba09-4fd3bb1fc07f\") " pod="openshift-multus/network-metrics-daemon-fqmzj" Sep 10 18:31:16.476105 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475588 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-host-run-netns\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.476105 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475616 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-log-socket\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.476105 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475633 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-sys\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.476105 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475642 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-var-lib-openvswitch\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.476105 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475666 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9eac62b3-150f-42a5-9af2-bf313bf92907-ovnkube-config\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.476105 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475691 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-etc-sysctl-d\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.476105 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475718 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-host-var-lib-cni-bin\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.477061 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475740 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-host-var-lib-cni-multus\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.477061 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475763 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-hostroot\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.477061 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475775 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-host-kubelet\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.477061 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475783 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-etc-kubernetes\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.477061 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475810 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-host-run-netns\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.477061 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475826 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-host-var-lib-cni-multus\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.477061 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475834 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-host-run-ovn-kubernetes\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.477061 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475875 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-etc-sysctl-conf\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.477061 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475903 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-multus-socket-dir-parent\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.477061 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475873 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-host-run-ovn-kubernetes\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.477061 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475930 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dc8250af-b1fc-4108-b6fd-092198a098ac-serviceca\") pod \"node-ca-lh8gr\" (UID: \"dc8250af-b1fc-4108-b6fd-092198a098ac\") " pod="openshift-image-registry/node-ca-lh8gr" Sep 10 18:31:16.477061 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475970 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-host-run-netns\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.477061 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475945 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-node-log\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.477061 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475999 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-etc-kubernetes\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.477061 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476007 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-tmp\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.477061 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476020 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-multus-socket-dir-parent\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.477061 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476044 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-var-lib-openvswitch\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.477061 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475741 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-systemd-units\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.477927 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476045 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhxb6\" (UniqueName: \"kubernetes.io/projected/a43748df-f7e9-4e22-b659-34b9550d706e-kube-api-access-xhxb6\") pod \"network-check-target-cznnw\" (UID: \"a43748df-f7e9-4e22-b659-34b9550d706e\") " pod="openshift-network-diagnostics/network-check-target-cznnw" Sep 10 18:31:16.477927 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476086 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-hostroot\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.477927 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476096 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvg8v\" (UniqueName: \"kubernetes.io/projected/d84d2e56-1f36-4691-8608-a1af00d9cce5-kube-api-access-gvg8v\") pod \"iptables-alerter-hxctp\" (UID: \"d84d2e56-1f36-4691-8608-a1af00d9cce5\") " pod="openshift-network-operator/iptables-alerter-hxctp" Sep 10 18:31:16.477927 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476107 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-etc-sysctl-d\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.477927 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476122 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70d9c337-84ae-485f-be5d-9a7c3919f3b0-cni-binary-copy\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.477927 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476149 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-host-var-lib-cni-bin\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.477927 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476148 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcxb5\" (UniqueName: \"kubernetes.io/projected/70d9c337-84ae-485f-be5d-9a7c3919f3b0-kube-api-access-pcxb5\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.477927 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476191 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-run-openvswitch\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.477927 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476217 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.477927 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476245 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-run\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.477927 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476269 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d84d2e56-1f36-4691-8608-a1af00d9cce5-iptables-alerter-script\") pod \"iptables-alerter-hxctp\" (UID: \"d84d2e56-1f36-4691-8608-a1af00d9cce5\") " pod="openshift-network-operator/iptables-alerter-hxctp" Sep 10 18:31:16.477927 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476309 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn42b\" (UniqueName: \"kubernetes.io/projected/4d70e39b-2b94-4593-828b-75d5d071560e-kube-api-access-nn42b\") pod \"node-resolver-gf28x\" (UID: \"4d70e39b-2b94-4593-828b-75d5d071560e\") " pod="openshift-dns/node-resolver-gf28x" Sep 10 18:31:16.477927 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476368 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9-dbus\") pod \"global-pull-secret-syncer-df44c\" (UID: \"0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9\") " pod="kube-system/global-pull-secret-syncer-df44c" Sep 10 18:31:16.477927 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476003 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-log-socket\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.477927 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476422 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-system-cni-dir\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.477927 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476449 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9-dbus\") pod \"global-pull-secret-syncer-df44c\" (UID: \"0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9\") " pod="kube-system/global-pull-secret-syncer-df44c" Sep 10 18:31:16.477927 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476461 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/70d9c337-84ae-485f-be5d-9a7c3919f3b0-multus-daemon-config\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.477927 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476494 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-system-cni-dir\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.478581 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476534 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-run\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.478581 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476666 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-etc-sysctl-conf\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.478581 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476705 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-run-openvswitch\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.478581 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476743 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.478581 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.475781 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9eac62b3-150f-42a5-9af2-bf313bf92907-env-overrides\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.478581 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476451 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-multus-cni-dir\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.478581 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476800 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4d70e39b-2b94-4593-828b-75d5d071560e-hosts-file\") pod \"node-resolver-gf28x\" (UID: \"4d70e39b-2b94-4593-828b-75d5d071560e\") " pod="openshift-dns/node-resolver-gf28x" Sep 10 18:31:16.478581 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476825 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9-original-pull-secret\") pod \"global-pull-secret-syncer-df44c\" (UID: \"0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9\") " pod="kube-system/global-pull-secret-syncer-df44c" Sep 10 18:31:16.478581 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476851 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-host\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.478581 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476875 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70d9c337-84ae-485f-be5d-9a7c3919f3b0-cni-binary-copy\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.478581 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476877 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dc8250af-b1fc-4108-b6fd-092198a098ac-host\") pod \"node-ca-lh8gr\" (UID: \"dc8250af-b1fc-4108-b6fd-092198a098ac\") " pod="openshift-image-registry/node-ca-lh8gr" Sep 10 18:31:16.478581 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476876 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d84d2e56-1f36-4691-8608-a1af00d9cce5-iptables-alerter-script\") pod \"iptables-alerter-hxctp\" (UID: \"d84d2e56-1f36-4691-8608-a1af00d9cce5\") " pod="openshift-network-operator/iptables-alerter-hxctp" Sep 10 18:31:16.478581 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476902 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-etc-sysconfig\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.478581 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476929 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-host-var-lib-kubelet\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.478581 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476938 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-host-run-netns\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.478581 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476951 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-host-cni-bin\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.478581 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476957 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-multus-cni-dir\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.478581 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476976 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9-kubelet-config\") pod \"global-pull-secret-syncer-df44c\" (UID: \"0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9\") " pod="kube-system/global-pull-secret-syncer-df44c" Sep 10 18:31:16.479271 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476981 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-host\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.479271 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.476995 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4d70e39b-2b94-4593-828b-75d5d071560e-hosts-file\") pod \"node-resolver-gf28x\" (UID: \"4d70e39b-2b94-4593-828b-75d5d071560e\") " pod="openshift-dns/node-resolver-gf28x" Sep 10 18:31:16.479271 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.477002 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d84d2e56-1f36-4691-8608-a1af00d9cce5-host-slash\") pod \"iptables-alerter-hxctp\" (UID: \"d84d2e56-1f36-4691-8608-a1af00d9cce5\") " pod="openshift-network-operator/iptables-alerter-hxctp" Sep 10 18:31:16.479271 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.477026 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-host-run-multus-certs\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.479271 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.477049 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d70e39b-2b94-4593-828b-75d5d071560e-tmp-dir\") pod \"node-resolver-gf28x\" (UID: \"4d70e39b-2b94-4593-828b-75d5d071560e\") " pod="openshift-dns/node-resolver-gf28x" Sep 10 18:31:16.479271 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:16.477057 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 18:31:16.479271 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.477071 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-host-slash\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.479271 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.477092 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-run-systemd\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.479271 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.477119 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-run-ovn\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.479271 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.477150 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-node-log\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.479271 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.477167 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-run-ovn\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.479271 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:16.477173 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9-original-pull-secret podName:0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9 nodeName:}" failed. No retries permitted until 2026-09-10 18:31:16.977139876 +0000 UTC m=+3.063477350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9-original-pull-secret") pod "global-pull-secret-syncer-df44c" (UID: "0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9") : object "kube-system"/"original-pull-secret" not registered Sep 10 18:31:16.479271 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.477194 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dc8250af-b1fc-4108-b6fd-092198a098ac-host\") pod \"node-ca-lh8gr\" (UID: \"dc8250af-b1fc-4108-b6fd-092198a098ac\") " pod="openshift-image-registry/node-ca-lh8gr" Sep 10 18:31:16.479271 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.477193 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bd3aed00-8ff3-4247-abc7-4893cfae63dc-konnectivity-ca\") pod \"konnectivity-agent-4g45x\" (UID: \"bd3aed00-8ff3-4247-abc7-4893cfae63dc\") " pod="kube-system/konnectivity-agent-4g45x" Sep 10 18:31:16.479271 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.477212 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9-kubelet-config\") pod \"global-pull-secret-syncer-df44c\" (UID: \"0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9\") " pod="kube-system/global-pull-secret-syncer-df44c" Sep 10 18:31:16.479271 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.477220 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-cnibin\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.479271 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.477252 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d84d2e56-1f36-4691-8608-a1af00d9cce5-host-slash\") pod \"iptables-alerter-hxctp\" (UID: \"d84d2e56-1f36-4691-8608-a1af00d9cce5\") " pod="openshift-network-operator/iptables-alerter-hxctp" Sep 10 18:31:16.479271 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.477256 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-multus-conf-dir\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.479890 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.477274 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-multus-conf-dir\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.479890 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.477282 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9eac62b3-150f-42a5-9af2-bf313bf92907-ovn-node-metrics-cert\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.479890 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.477307 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-etc-sysconfig\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.479890 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.477308 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54ef3c93-1624-46e6-ba09-4fd3bb1fc07f-metrics-certs\") pod \"network-metrics-daemon-fqmzj\" (UID: \"54ef3c93-1624-46e6-ba09-4fd3bb1fc07f\") " pod="openshift-multus/network-metrics-daemon-fqmzj" Sep 10 18:31:16.479890 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.477355 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-host-var-lib-kubelet\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.479890 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.477382 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-host-cni-bin\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.479890 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.477411 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-cnibin\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.479890 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:16.477425 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 18:31:16.479890 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.477444 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-host-slash\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.479890 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:16.477483 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54ef3c93-1624-46e6-ba09-4fd3bb1fc07f-metrics-certs podName:54ef3c93-1624-46e6-ba09-4fd3bb1fc07f nodeName:}" failed. No retries permitted until 2026-09-10 18:31:16.977467516 +0000 UTC m=+3.063804970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54ef3c93-1624-46e6-ba09-4fd3bb1fc07f-metrics-certs") pod "network-metrics-daemon-fqmzj" (UID: "54ef3c93-1624-46e6-ba09-4fd3bb1fc07f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 18:31:16.479890 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.477504 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9eac62b3-150f-42a5-9af2-bf313bf92907-run-systemd\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.479890 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.477556 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/70d9c337-84ae-485f-be5d-9a7c3919f3b0-host-run-multus-certs\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.479890 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.477687 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d70e39b-2b94-4593-828b-75d5d071560e-tmp-dir\") pod \"node-resolver-gf28x\" (UID: \"4d70e39b-2b94-4593-828b-75d5d071560e\") " pod="openshift-dns/node-resolver-gf28x" Sep 10 18:31:16.479890 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.478133 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-etc-tuned\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.479890 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.478657 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bd3aed00-8ff3-4247-abc7-4893cfae63dc-agent-certs\") pod \"konnectivity-agent-4g45x\" (UID: \"bd3aed00-8ff3-4247-abc7-4893cfae63dc\") " pod="kube-system/konnectivity-agent-4g45x" Sep 10 18:31:16.479890 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.479735 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-tmp\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.480491 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.479990 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9eac62b3-150f-42a5-9af2-bf313bf92907-ovn-node-metrics-cert\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.480491 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.480454 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bd3aed00-8ff3-4247-abc7-4893cfae63dc-konnectivity-ca\") pod \"konnectivity-agent-4g45x\" (UID: \"bd3aed00-8ff3-4247-abc7-4893cfae63dc\") " pod="kube-system/konnectivity-agent-4g45x" Sep 10 18:31:16.480794 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.480772 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9eac62b3-150f-42a5-9af2-bf313bf92907-ovnkube-config\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.480886 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.480794 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9eac62b3-150f-42a5-9af2-bf313bf92907-ovnkube-script-lib\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.482298 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:16.482277 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 18:31:16.482415 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:16.482307 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 18:31:16.482415 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:16.482372 2581 projected.go:196] Error preparing data for projected volume kube-api-access-xhxb6 for pod openshift-network-diagnostics/network-check-target-cznnw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 18:31:16.482539 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:16.482445 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a43748df-f7e9-4e22-b659-34b9550d706e-kube-api-access-xhxb6 podName:a43748df-f7e9-4e22-b659-34b9550d706e nodeName:}" failed. No retries permitted until 2026-09-10 18:31:16.982429623 +0000 UTC m=+3.068767077 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xhxb6" (UniqueName: "kubernetes.io/projected/a43748df-f7e9-4e22-b659-34b9550d706e-kube-api-access-xhxb6") pod "network-check-target-cznnw" (UID: "a43748df-f7e9-4e22-b659-34b9550d706e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 18:31:16.484285 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.484258 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbwkl\" (UniqueName: \"kubernetes.io/projected/dc8250af-b1fc-4108-b6fd-092198a098ac-kube-api-access-nbwkl\") pod \"node-ca-lh8gr\" (UID: \"dc8250af-b1fc-4108-b6fd-092198a098ac\") " pod="openshift-image-registry/node-ca-lh8gr" Sep 10 18:31:16.484766 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.484743 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcxb5\" (UniqueName: \"kubernetes.io/projected/70d9c337-84ae-485f-be5d-9a7c3919f3b0-kube-api-access-pcxb5\") pod \"multus-jc45h\" (UID: \"70d9c337-84ae-485f-be5d-9a7c3919f3b0\") " pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.485201 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.485178 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcz8h\" (UniqueName: \"kubernetes.io/projected/85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1-kube-api-access-jcz8h\") pod \"tuned-jcwmv\" (UID: \"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1\") " pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.485273 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.485208 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvg8v\" (UniqueName: \"kubernetes.io/projected/d84d2e56-1f36-4691-8608-a1af00d9cce5-kube-api-access-gvg8v\") pod \"iptables-alerter-hxctp\" (UID: \"d84d2e56-1f36-4691-8608-a1af00d9cce5\") " pod="openshift-network-operator/iptables-alerter-hxctp" Sep 10 18:31:16.485273 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.485259 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5l6n\" (UniqueName: \"kubernetes.io/projected/54ef3c93-1624-46e6-ba09-4fd3bb1fc07f-kube-api-access-v5l6n\") pod \"network-metrics-daemon-fqmzj\" (UID: \"54ef3c93-1624-46e6-ba09-4fd3bb1fc07f\") " pod="openshift-multus/network-metrics-daemon-fqmzj" Sep 10 18:31:16.486013 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.485992 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn42b\" (UniqueName: \"kubernetes.io/projected/4d70e39b-2b94-4593-828b-75d5d071560e-kube-api-access-nn42b\") pod \"node-resolver-gf28x\" (UID: \"4d70e39b-2b94-4593-828b-75d5d071560e\") " pod="openshift-dns/node-resolver-gf28x" Sep 10 18:31:16.486616 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.486600 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsdgh\" (UniqueName: \"kubernetes.io/projected/9eac62b3-150f-42a5-9af2-bf313bf92907-kube-api-access-tsdgh\") pod \"ovnkube-node-9j6cn\" (UID: \"9eac62b3-150f-42a5-9af2-bf313bf92907\") " pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.569242 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.569217 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qrb6h" Sep 10 18:31:16.576712 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.576692 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zpc6" Sep 10 18:31:16.586228 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.586209 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gf28x" Sep 10 18:31:16.598693 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.598676 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lh8gr" Sep 10 18:31:16.606197 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.606178 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" Sep 10 18:31:16.616600 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.616581 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hxctp" Sep 10 18:31:16.623142 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.623125 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:16.630424 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.630405 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4g45x" Sep 10 18:31:16.636572 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.636552 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jc45h" Sep 10 18:31:16.809013 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.808989 2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 18:31:16.980716 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.980641 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9-original-pull-secret\") pod \"global-pull-secret-syncer-df44c\" (UID: \"0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9\") " pod="kube-system/global-pull-secret-syncer-df44c" Sep 10 18:31:16.980716 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:16.980680 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54ef3c93-1624-46e6-ba09-4fd3bb1fc07f-metrics-certs\") pod \"network-metrics-daemon-fqmzj\" (UID: \"54ef3c93-1624-46e6-ba09-4fd3bb1fc07f\") " pod="openshift-multus/network-metrics-daemon-fqmzj" Sep 10 18:31:16.980893 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:16.980793 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 18:31:16.980893 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:16.980794 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 18:31:16.980893 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:16.980853 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54ef3c93-1624-46e6-ba09-4fd3bb1fc07f-metrics-certs podName:54ef3c93-1624-46e6-ba09-4fd3bb1fc07f nodeName:}" failed. No retries permitted until 2026-09-10 18:31:17.980839144 +0000 UTC m=+4.067176597 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54ef3c93-1624-46e6-ba09-4fd3bb1fc07f-metrics-certs") pod "network-metrics-daemon-fqmzj" (UID: "54ef3c93-1624-46e6-ba09-4fd3bb1fc07f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 18:31:16.980893 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:16.980868 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9-original-pull-secret podName:0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9 nodeName:}" failed. No retries permitted until 2026-09-10 18:31:17.980860502 +0000 UTC m=+4.067197954 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9-original-pull-secret") pod "global-pull-secret-syncer-df44c" (UID: "0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9") : object "kube-system"/"original-pull-secret" not registered Sep 10 18:31:17.081006 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:17.080975 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhxb6\" (UniqueName: \"kubernetes.io/projected/a43748df-f7e9-4e22-b659-34b9550d706e-kube-api-access-xhxb6\") pod \"network-check-target-cznnw\" (UID: \"a43748df-f7e9-4e22-b659-34b9550d706e\") " pod="openshift-network-diagnostics/network-check-target-cznnw" Sep 10 18:31:17.081176 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:17.081097 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 18:31:17.081176 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:17.081117 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 18:31:17.081176 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:17.081129 2581 projected.go:196] Error preparing data for projected volume kube-api-access-xhxb6 for pod openshift-network-diagnostics/network-check-target-cznnw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 18:31:17.081289 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:17.081183 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a43748df-f7e9-4e22-b659-34b9550d706e-kube-api-access-xhxb6 podName:a43748df-f7e9-4e22-b659-34b9550d706e nodeName:}" failed. No retries permitted until 2026-09-10 18:31:18.08116403 +0000 UTC m=+4.167501503 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xhxb6" (UniqueName: "kubernetes.io/projected/a43748df-f7e9-4e22-b659-34b9550d706e-kube-api-access-xhxb6") pod "network-check-target-cznnw" (UID: "a43748df-f7e9-4e22-b659-34b9550d706e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 18:31:17.245533 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:17.245439 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod898e6b12_a7af_4d34_af96_b6c1f54e66c5.slice/crio-b8f8ecff1b0e800d2bf59d6b1251965ae3987d86b3438000c7de5763a7fa923e WatchSource:0}: Error finding container b8f8ecff1b0e800d2bf59d6b1251965ae3987d86b3438000c7de5763a7fa923e: Status 404 returned error can't find the container with id b8f8ecff1b0e800d2bf59d6b1251965ae3987d86b3438000c7de5763a7fa923e Sep 10 18:31:17.247060 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:17.246956 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd84d2e56_1f36_4691_8608_a1af00d9cce5.slice/crio-40a8e7a31007685e63086c02c2f4d2dc6c56aa750655ce20617d5d4ef7cebe6e WatchSource:0}: Error finding container 40a8e7a31007685e63086c02c2f4d2dc6c56aa750655ce20617d5d4ef7cebe6e: Status 404 returned error can't find the container with id 40a8e7a31007685e63086c02c2f4d2dc6c56aa750655ce20617d5d4ef7cebe6e Sep 10 18:31:17.248241 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:17.248213 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod747cbe9d_036e_448f_b567_6ffd3c917b2e.slice/crio-cb3c9e33b61aab817539f2fe8703f21ac0db0a690ab2d2fdc4de32d9610999a0 WatchSource:0}: Error finding container cb3c9e33b61aab817539f2fe8703f21ac0db0a690ab2d2fdc4de32d9610999a0: Status 404 returned error can't find the container with id cb3c9e33b61aab817539f2fe8703f21ac0db0a690ab2d2fdc4de32d9610999a0 Sep 10 18:31:17.250880 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:17.250863 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc8250af_b1fc_4108_b6fd_092198a098ac.slice/crio-da36a2b50640229c235fa172023a409fde54988b0f8047a8017b32e2cf474a5b WatchSource:0}: Error finding container da36a2b50640229c235fa172023a409fde54988b0f8047a8017b32e2cf474a5b: Status 404 returned error can't find the container with id da36a2b50640229c235fa172023a409fde54988b0f8047a8017b32e2cf474a5b Sep 10 18:31:17.251534 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:17.251512 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3aed00_8ff3_4247_abc7_4893cfae63dc.slice/crio-c03327e9e53ec434f7ba23c8c2c6c9de3b8402d8f29b33e28cd8a7c63d3a2f1c WatchSource:0}: Error finding container c03327e9e53ec434f7ba23c8c2c6c9de3b8402d8f29b33e28cd8a7c63d3a2f1c: Status 404 returned error can't find the container with id c03327e9e53ec434f7ba23c8c2c6c9de3b8402d8f29b33e28cd8a7c63d3a2f1c Sep 10 18:31:17.253433 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:17.253412 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d70e39b_2b94_4593_828b_75d5d071560e.slice/crio-c3e603c45555a97f2053fc10687f0127224c12b6b6c7878e7c24cff1bcd24c59 WatchSource:0}: Error finding container c3e603c45555a97f2053fc10687f0127224c12b6b6c7878e7c24cff1bcd24c59: Status 404 returned error can't find the container with id c3e603c45555a97f2053fc10687f0127224c12b6b6c7878e7c24cff1bcd24c59 Sep 10 18:31:17.253715 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:17.253687 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eac62b3_150f_42a5_9af2_bf313bf92907.slice/crio-2b4df2bfaf65aa38bb7d6eb4fe1015d23bd653b4ebebbe1c59e11b96720fd6c2 WatchSource:0}: Error finding container 2b4df2bfaf65aa38bb7d6eb4fe1015d23bd653b4ebebbe1c59e11b96720fd6c2: Status 404 returned error can't find the container with id 2b4df2bfaf65aa38bb7d6eb4fe1015d23bd653b4ebebbe1c59e11b96720fd6c2 Sep 10 18:31:17.255183 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:17.255158 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70d9c337_84ae_485f_be5d_9a7c3919f3b0.slice/crio-5d14dae26183389422f675ca71c3ca77fd46da31027d359982ca0175c0b6fe04 WatchSource:0}: Error finding container 5d14dae26183389422f675ca71c3ca77fd46da31027d359982ca0175c0b6fe04: Status 404 returned error can't find the container with id 5d14dae26183389422f675ca71c3ca77fd46da31027d359982ca0175c0b6fe04 Sep 10 18:31:17.256435 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:17.256416 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85c327e7_271d_4c17_b0fd_a8d4d5cfd2a1.slice/crio-046a125f3baf17d1d199ba277d4f2c9c3798bd89c6a35387a7a29ea50037b6ba WatchSource:0}: Error finding container 046a125f3baf17d1d199ba277d4f2c9c3798bd89c6a35387a7a29ea50037b6ba: Status 404 returned error can't find the container with id 046a125f3baf17d1d199ba277d4f2c9c3798bd89c6a35387a7a29ea50037b6ba Sep 10 18:31:17.318593 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:17.318539 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 18:26:16 +0000 UTC" deadline="2028-06-12 02:20:06.304615512 +0000 UTC" Sep 10 18:31:17.318680 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:17.318594 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15367h48m48.986026549s" Sep 10 18:31:17.385464 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:17.385432 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4g45x" event={"ID":"bd3aed00-8ff3-4247-abc7-4893cfae63dc","Type":"ContainerStarted","Data":"c03327e9e53ec434f7ba23c8c2c6c9de3b8402d8f29b33e28cd8a7c63d3a2f1c"} Sep 10 18:31:17.387307 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:17.387281 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" event={"ID":"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1","Type":"ContainerStarted","Data":"046a125f3baf17d1d199ba277d4f2c9c3798bd89c6a35387a7a29ea50037b6ba"} Sep 10 18:31:17.388505 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:17.388480 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jc45h" event={"ID":"70d9c337-84ae-485f-be5d-9a7c3919f3b0","Type":"ContainerStarted","Data":"5d14dae26183389422f675ca71c3ca77fd46da31027d359982ca0175c0b6fe04"} Sep 10 18:31:17.389741 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:17.389718 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" event={"ID":"9eac62b3-150f-42a5-9af2-bf313bf92907","Type":"ContainerStarted","Data":"2b4df2bfaf65aa38bb7d6eb4fe1015d23bd653b4ebebbe1c59e11b96720fd6c2"} Sep 10 18:31:17.391180 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:17.391152 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gf28x" event={"ID":"4d70e39b-2b94-4593-828b-75d5d071560e","Type":"ContainerStarted","Data":"c3e603c45555a97f2053fc10687f0127224c12b6b6c7878e7c24cff1bcd24c59"} Sep 10 18:31:17.392417 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:17.392396 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qrb6h" event={"ID":"747cbe9d-036e-448f-b567-6ffd3c917b2e","Type":"ContainerStarted","Data":"cb3c9e33b61aab817539f2fe8703f21ac0db0a690ab2d2fdc4de32d9610999a0"} Sep 10 18:31:17.393946 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:17.393924 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hxctp" event={"ID":"d84d2e56-1f36-4691-8608-a1af00d9cce5","Type":"ContainerStarted","Data":"40a8e7a31007685e63086c02c2f4d2dc6c56aa750655ce20617d5d4ef7cebe6e"} Sep 10 18:31:17.396034 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:17.396015 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-166.ec2.internal" event={"ID":"a6bd606f49b0efaf64eed63f0b2241b9","Type":"ContainerStarted","Data":"0cfddc244adea846c93b899b7325d50735dcc0d8637fe1357c270f885aaed7d3"} Sep 10 18:31:17.397239 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:17.397221 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lh8gr" event={"ID":"dc8250af-b1fc-4108-b6fd-092198a098ac","Type":"ContainerStarted","Data":"da36a2b50640229c235fa172023a409fde54988b0f8047a8017b32e2cf474a5b"} Sep 10 18:31:17.398423 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:17.398402 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zpc6" event={"ID":"898e6b12-a7af-4d34-af96-b6c1f54e66c5","Type":"ContainerStarted","Data":"b8f8ecff1b0e800d2bf59d6b1251965ae3987d86b3438000c7de5763a7fa923e"} Sep 10 18:31:17.408368 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:17.408312 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-166.ec2.internal" podStartSLOduration=2.408300115 podStartE2EDuration="2.408300115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 18:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:31:17.407814848 +0000 UTC m=+3.494152325" watchObservedRunningTime="2026-09-10 18:31:17.408300115 +0000 UTC m=+3.494637589" Sep 10 18:31:17.988587 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:17.988550 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9-original-pull-secret\") pod \"global-pull-secret-syncer-df44c\" (UID: \"0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9\") " pod="kube-system/global-pull-secret-syncer-df44c" Sep 10 18:31:17.988764 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:17.988610 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54ef3c93-1624-46e6-ba09-4fd3bb1fc07f-metrics-certs\") pod \"network-metrics-daemon-fqmzj\" (UID: \"54ef3c93-1624-46e6-ba09-4fd3bb1fc07f\") " pod="openshift-multus/network-metrics-daemon-fqmzj" Sep 10 18:31:17.988764 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:17.988723 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 18:31:17.988898 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:17.988780 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54ef3c93-1624-46e6-ba09-4fd3bb1fc07f-metrics-certs podName:54ef3c93-1624-46e6-ba09-4fd3bb1fc07f nodeName:}" failed. No retries permitted until 2026-09-10 18:31:19.988761982 +0000 UTC m=+6.075099442 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54ef3c93-1624-46e6-ba09-4fd3bb1fc07f-metrics-certs") pod "network-metrics-daemon-fqmzj" (UID: "54ef3c93-1624-46e6-ba09-4fd3bb1fc07f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 18:31:17.988898 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:17.988839 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 18:31:17.988898 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:17.988869 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9-original-pull-secret podName:0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9 nodeName:}" failed. No retries permitted until 2026-09-10 18:31:19.988858157 +0000 UTC m=+6.075195610 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9-original-pull-secret") pod "global-pull-secret-syncer-df44c" (UID: "0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9") : object "kube-system"/"original-pull-secret" not registered Sep 10 18:31:18.089401 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:18.089164 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhxb6\" (UniqueName: \"kubernetes.io/projected/a43748df-f7e9-4e22-b659-34b9550d706e-kube-api-access-xhxb6\") pod \"network-check-target-cznnw\" (UID: \"a43748df-f7e9-4e22-b659-34b9550d706e\") " pod="openshift-network-diagnostics/network-check-target-cznnw" Sep 10 18:31:18.089401 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:18.089393 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 18:31:18.089628 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:18.089427 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 18:31:18.089628 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:18.089442 2581 projected.go:196] Error preparing data for projected volume kube-api-access-xhxb6 for pod openshift-network-diagnostics/network-check-target-cznnw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 18:31:18.089628 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:18.089497 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a43748df-f7e9-4e22-b659-34b9550d706e-kube-api-access-xhxb6 podName:a43748df-f7e9-4e22-b659-34b9550d706e nodeName:}" failed. No retries permitted until 2026-09-10 18:31:20.089479788 +0000 UTC m=+6.175817256 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xhxb6" (UniqueName: "kubernetes.io/projected/a43748df-f7e9-4e22-b659-34b9550d706e-kube-api-access-xhxb6") pod "network-check-target-cznnw" (UID: "a43748df-f7e9-4e22-b659-34b9550d706e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 18:31:18.319548 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:18.319475 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 18:26:16 +0000 UTC" deadline="2028-04-18 00:01:47.222086156 +0000 UTC" Sep 10 18:31:18.319548 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:18.319514 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14045h30m28.902577188s" Sep 10 18:31:18.375947 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:18.374697 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmzj" Sep 10 18:31:18.375947 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:18.374709 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cznnw" Sep 10 18:31:18.375947 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:18.374847 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqmzj" podUID="54ef3c93-1624-46e6-ba09-4fd3bb1fc07f" Sep 10 18:31:18.375947 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:18.374924 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-df44c" Sep 10 18:31:18.375947 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:18.375024 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-df44c" podUID="0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9" Sep 10 18:31:18.375947 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:18.375100 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cznnw" podUID="a43748df-f7e9-4e22-b659-34b9550d706e" Sep 10 18:31:18.409900 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:18.409866 2581 generic.go:356] "Generic (PLEG): container finished" podID="28f1ecabb2c30a2edb71680a33548326" containerID="e712f23c52bf857fc6174d6f900c41e3e48f39dd433ccc788a54a3cb8010e2c6" exitCode=0 Sep 10 18:31:18.410832 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:18.410804 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-166.ec2.internal" event={"ID":"28f1ecabb2c30a2edb71680a33548326","Type":"ContainerDied","Data":"e712f23c52bf857fc6174d6f900c41e3e48f39dd433ccc788a54a3cb8010e2c6"} Sep 10 18:31:19.422554 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:19.422516 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-166.ec2.internal" event={"ID":"28f1ecabb2c30a2edb71680a33548326","Type":"ContainerStarted","Data":"d1ef282842802e46c69047dda3c018949c465fbd06be3eb04e22e7c4b37783e1"} Sep 10 18:31:19.436381 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:19.436312 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-166.ec2.internal" podStartSLOduration=4.436293782 podStartE2EDuration="4.436293782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 18:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:31:19.435836 +0000 UTC m=+5.522173475" watchObservedRunningTime="2026-09-10 18:31:19.436293782 +0000 UTC m=+5.522631258" Sep 10 18:31:20.005656 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:20.005387 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9-original-pull-secret\") pod \"global-pull-secret-syncer-df44c\" (UID: \"0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9\") " pod="kube-system/global-pull-secret-syncer-df44c" Sep 10 18:31:20.005823 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:20.005683 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54ef3c93-1624-46e6-ba09-4fd3bb1fc07f-metrics-certs\") pod \"network-metrics-daemon-fqmzj\" (UID: \"54ef3c93-1624-46e6-ba09-4fd3bb1fc07f\") " pod="openshift-multus/network-metrics-daemon-fqmzj" Sep 10 18:31:20.005823 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:20.005596 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 18:31:20.005943 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:20.005853 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9-original-pull-secret podName:0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9 nodeName:}" failed. No retries permitted until 2026-09-10 18:31:24.005833857 +0000 UTC m=+10.092171335 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9-original-pull-secret") pod "global-pull-secret-syncer-df44c" (UID: "0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9") : object "kube-system"/"original-pull-secret" not registered Sep 10 18:31:20.006177 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:20.005792 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 18:31:20.006278 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:20.006225 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54ef3c93-1624-46e6-ba09-4fd3bb1fc07f-metrics-certs podName:54ef3c93-1624-46e6-ba09-4fd3bb1fc07f nodeName:}" failed. No retries permitted until 2026-09-10 18:31:24.006200653 +0000 UTC m=+10.092538121 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54ef3c93-1624-46e6-ba09-4fd3bb1fc07f-metrics-certs") pod "network-metrics-daemon-fqmzj" (UID: "54ef3c93-1624-46e6-ba09-4fd3bb1fc07f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 18:31:20.106246 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:20.106207 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhxb6\" (UniqueName: \"kubernetes.io/projected/a43748df-f7e9-4e22-b659-34b9550d706e-kube-api-access-xhxb6\") pod \"network-check-target-cznnw\" (UID: \"a43748df-f7e9-4e22-b659-34b9550d706e\") " pod="openshift-network-diagnostics/network-check-target-cznnw" Sep 10 18:31:20.106436 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:20.106425 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 18:31:20.106505 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:20.106446 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 18:31:20.106505 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:20.106459 2581 projected.go:196] Error preparing data for projected volume kube-api-access-xhxb6 for pod openshift-network-diagnostics/network-check-target-cznnw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 18:31:20.106606 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:20.106521 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a43748df-f7e9-4e22-b659-34b9550d706e-kube-api-access-xhxb6 podName:a43748df-f7e9-4e22-b659-34b9550d706e nodeName:}" failed. No retries permitted until 2026-09-10 18:31:24.106494156 +0000 UTC m=+10.192831614 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xhxb6" (UniqueName: "kubernetes.io/projected/a43748df-f7e9-4e22-b659-34b9550d706e-kube-api-access-xhxb6") pod "network-check-target-cznnw" (UID: "a43748df-f7e9-4e22-b659-34b9550d706e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 18:31:20.380515 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:20.380485 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-df44c" Sep 10 18:31:20.380741 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:20.380677 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-df44c" podUID="0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9" Sep 10 18:31:20.381619 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:20.381151 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmzj" Sep 10 18:31:20.381619 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:20.381369 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqmzj" podUID="54ef3c93-1624-46e6-ba09-4fd3bb1fc07f" Sep 10 18:31:20.381619 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:20.381456 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cznnw" Sep 10 18:31:20.381973 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:20.381922 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cznnw" podUID="a43748df-f7e9-4e22-b659-34b9550d706e" Sep 10 18:31:22.374550 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:22.374518 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-df44c" Sep 10 18:31:22.375013 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:22.374647 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-df44c" podUID="0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9" Sep 10 18:31:22.375092 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:22.375021 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmzj" Sep 10 18:31:22.375151 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:22.375121 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqmzj" podUID="54ef3c93-1624-46e6-ba09-4fd3bb1fc07f" Sep 10 18:31:22.375199 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:22.375169 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cznnw" Sep 10 18:31:22.375260 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:22.375236 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cznnw" podUID="a43748df-f7e9-4e22-b659-34b9550d706e" Sep 10 18:31:24.037573 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:24.036702 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54ef3c93-1624-46e6-ba09-4fd3bb1fc07f-metrics-certs\") pod \"network-metrics-daemon-fqmzj\" (UID: \"54ef3c93-1624-46e6-ba09-4fd3bb1fc07f\") " pod="openshift-multus/network-metrics-daemon-fqmzj" Sep 10 18:31:24.037573 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:24.036801 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9-original-pull-secret\") pod \"global-pull-secret-syncer-df44c\" (UID: \"0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9\") " pod="kube-system/global-pull-secret-syncer-df44c" Sep 10 18:31:24.037573 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:24.036961 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 18:31:24.037573 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:24.037027 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9-original-pull-secret podName:0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9 nodeName:}" failed. No retries permitted until 2026-09-10 18:31:32.037006818 +0000 UTC m=+18.123344293 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9-original-pull-secret") pod "global-pull-secret-syncer-df44c" (UID: "0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9") : object "kube-system"/"original-pull-secret" not registered Sep 10 18:31:24.037573 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:24.037489 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 18:31:24.037573 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:24.037539 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54ef3c93-1624-46e6-ba09-4fd3bb1fc07f-metrics-certs podName:54ef3c93-1624-46e6-ba09-4fd3bb1fc07f nodeName:}" failed. No retries permitted until 2026-09-10 18:31:32.037524328 +0000 UTC m=+18.123861796 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54ef3c93-1624-46e6-ba09-4fd3bb1fc07f-metrics-certs") pod "network-metrics-daemon-fqmzj" (UID: "54ef3c93-1624-46e6-ba09-4fd3bb1fc07f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 18:31:24.138543 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:24.137929 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhxb6\" (UniqueName: \"kubernetes.io/projected/a43748df-f7e9-4e22-b659-34b9550d706e-kube-api-access-xhxb6\") pod \"network-check-target-cznnw\" (UID: \"a43748df-f7e9-4e22-b659-34b9550d706e\") " pod="openshift-network-diagnostics/network-check-target-cznnw" Sep 10 18:31:24.138543 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:24.138105 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 18:31:24.138543 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:24.138130 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 18:31:24.138543 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:24.138143 2581 projected.go:196] Error preparing data for projected volume kube-api-access-xhxb6 for pod openshift-network-diagnostics/network-check-target-cznnw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 18:31:24.138543 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:24.138198 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a43748df-f7e9-4e22-b659-34b9550d706e-kube-api-access-xhxb6 podName:a43748df-f7e9-4e22-b659-34b9550d706e nodeName:}" failed. No retries permitted until 2026-09-10 18:31:32.13818052 +0000 UTC m=+18.224517987 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xhxb6" (UniqueName: "kubernetes.io/projected/a43748df-f7e9-4e22-b659-34b9550d706e-kube-api-access-xhxb6") pod "network-check-target-cznnw" (UID: "a43748df-f7e9-4e22-b659-34b9550d706e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 18:31:24.374890 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:24.374855 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cznnw" Sep 10 18:31:24.375058 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:24.374976 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cznnw" podUID="a43748df-f7e9-4e22-b659-34b9550d706e" Sep 10 18:31:24.375058 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:24.375030 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-df44c" Sep 10 18:31:24.375241 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:24.375096 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-df44c" podUID="0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9" Sep 10 18:31:24.375241 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:24.375136 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmzj" Sep 10 18:31:24.375241 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:24.375208 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqmzj" podUID="54ef3c93-1624-46e6-ba09-4fd3bb1fc07f" Sep 10 18:31:26.375352 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:26.375026 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-df44c" Sep 10 18:31:26.375352 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:26.375035 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmzj" Sep 10 18:31:26.375352 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:26.375159 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-df44c" podUID="0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9" Sep 10 18:31:26.375352 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:26.375192 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cznnw" Sep 10 18:31:26.375352 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:26.375259 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cznnw" podUID="a43748df-f7e9-4e22-b659-34b9550d706e" Sep 10 18:31:26.375877 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:26.375704 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqmzj" podUID="54ef3c93-1624-46e6-ba09-4fd3bb1fc07f" Sep 10 18:31:28.375171 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:28.374692 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-df44c" Sep 10 18:31:28.375171 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:28.374724 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cznnw" Sep 10 18:31:28.375171 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:28.374692 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmzj" Sep 10 18:31:28.375171 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:28.374811 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-df44c" podUID="0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9" Sep 10 18:31:28.375171 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:28.374917 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqmzj" podUID="54ef3c93-1624-46e6-ba09-4fd3bb1fc07f" Sep 10 18:31:28.375171 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:28.374989 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cznnw" podUID="a43748df-f7e9-4e22-b659-34b9550d706e" Sep 10 18:31:30.374817 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:30.374709 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmzj" Sep 10 18:31:30.374817 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:30.374709 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-df44c" Sep 10 18:31:30.375215 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:30.374853 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqmzj" podUID="54ef3c93-1624-46e6-ba09-4fd3bb1fc07f" Sep 10 18:31:30.375215 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:30.374864 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cznnw" Sep 10 18:31:30.375215 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:30.374934 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-df44c" podUID="0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9" Sep 10 18:31:30.375215 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:30.374990 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cznnw" podUID="a43748df-f7e9-4e22-b659-34b9550d706e" Sep 10 18:31:32.096642 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:32.096611 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9-original-pull-secret\") pod \"global-pull-secret-syncer-df44c\" (UID: \"0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9\") " pod="kube-system/global-pull-secret-syncer-df44c" Sep 10 18:31:32.096642 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:32.096649 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54ef3c93-1624-46e6-ba09-4fd3bb1fc07f-metrics-certs\") pod \"network-metrics-daemon-fqmzj\" (UID: \"54ef3c93-1624-46e6-ba09-4fd3bb1fc07f\") " pod="openshift-multus/network-metrics-daemon-fqmzj" Sep 10 18:31:32.097168 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:32.096748 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 18:31:32.097168 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:32.096748 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 18:31:32.097168 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:32.096808 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9-original-pull-secret podName:0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9 nodeName:}" failed. No retries permitted until 2026-09-10 18:31:48.096789475 +0000 UTC m=+34.183126942 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9-original-pull-secret") pod "global-pull-secret-syncer-df44c" (UID: "0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9") : object "kube-system"/"original-pull-secret" not registered Sep 10 18:31:32.097168 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:32.096825 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54ef3c93-1624-46e6-ba09-4fd3bb1fc07f-metrics-certs podName:54ef3c93-1624-46e6-ba09-4fd3bb1fc07f nodeName:}" failed. No retries permitted until 2026-09-10 18:31:48.096816529 +0000 UTC m=+34.183153982 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54ef3c93-1624-46e6-ba09-4fd3bb1fc07f-metrics-certs") pod "network-metrics-daemon-fqmzj" (UID: "54ef3c93-1624-46e6-ba09-4fd3bb1fc07f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 18:31:32.197165 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:32.197136 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhxb6\" (UniqueName: \"kubernetes.io/projected/a43748df-f7e9-4e22-b659-34b9550d706e-kube-api-access-xhxb6\") pod \"network-check-target-cznnw\" (UID: \"a43748df-f7e9-4e22-b659-34b9550d706e\") " pod="openshift-network-diagnostics/network-check-target-cznnw" Sep 10 18:31:32.197361 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:32.197342 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 18:31:32.197418 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:32.197373 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 18:31:32.197418 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:32.197387 2581 projected.go:196] Error preparing data for projected volume kube-api-access-xhxb6 for pod openshift-network-diagnostics/network-check-target-cznnw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 18:31:32.197506 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:32.197444 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a43748df-f7e9-4e22-b659-34b9550d706e-kube-api-access-xhxb6 podName:a43748df-f7e9-4e22-b659-34b9550d706e nodeName:}" failed. No retries permitted until 2026-09-10 18:31:48.197425957 +0000 UTC m=+34.283763410 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xhxb6" (UniqueName: "kubernetes.io/projected/a43748df-f7e9-4e22-b659-34b9550d706e-kube-api-access-xhxb6") pod "network-check-target-cznnw" (UID: "a43748df-f7e9-4e22-b659-34b9550d706e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 18:31:32.374916 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:32.374842 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-df44c" Sep 10 18:31:32.374916 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:32.374881 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cznnw" Sep 10 18:31:32.375129 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:32.374842 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmzj" Sep 10 18:31:32.375129 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:32.374958 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-df44c" podUID="0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9" Sep 10 18:31:32.375129 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:32.375023 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqmzj" podUID="54ef3c93-1624-46e6-ba09-4fd3bb1fc07f" Sep 10 18:31:32.375129 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:32.375102 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cznnw" podUID="a43748df-f7e9-4e22-b659-34b9550d706e" Sep 10 18:31:34.375496 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:34.375466 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-df44c" Sep 10 18:31:34.375923 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:34.375565 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cznnw" Sep 10 18:31:34.375923 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:34.375592 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-df44c" podUID="0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9" Sep 10 18:31:34.375923 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:34.375637 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cznnw" podUID="a43748df-f7e9-4e22-b659-34b9550d706e" Sep 10 18:31:34.375923 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:34.375666 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmzj" Sep 10 18:31:34.375923 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:34.375782 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqmzj" podUID="54ef3c93-1624-46e6-ba09-4fd3bb1fc07f" Sep 10 18:31:35.455224 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:35.455049 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" event={"ID":"9eac62b3-150f-42a5-9af2-bf313bf92907","Type":"ContainerStarted","Data":"a596dc117ea03c11e95eea098201e0f06f6c7832e64da120f0597d2eac2bd7e3"} Sep 10 18:31:35.455899 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:35.455305 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" event={"ID":"9eac62b3-150f-42a5-9af2-bf313bf92907","Type":"ContainerStarted","Data":"967a4c00fb5a5a1f34d49f252e45fb8d0e592e46d2402ab84e45e3e1dc6b1fb9"} Sep 10 18:31:35.456695 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:35.456663 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gf28x" event={"ID":"4d70e39b-2b94-4593-828b-75d5d071560e","Type":"ContainerStarted","Data":"c7bca011335324ce39d9664dd021f3e91191a2d5900ec145032759be355a3a74"} Sep 10 18:31:35.458345 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:35.458299 2581 generic.go:356] "Generic (PLEG): container finished" podID="747cbe9d-036e-448f-b567-6ffd3c917b2e" containerID="b4e3cdb37324664211113ad3f1627ab569eeb887d0d8caafa12fee3a7ab335d2" exitCode=0 Sep 10 18:31:35.458448 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:35.458359 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qrb6h" event={"ID":"747cbe9d-036e-448f-b567-6ffd3c917b2e","Type":"ContainerDied","Data":"b4e3cdb37324664211113ad3f1627ab569eeb887d0d8caafa12fee3a7ab335d2"} Sep 10 18:31:35.460409 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:35.460272 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lh8gr" event={"ID":"dc8250af-b1fc-4108-b6fd-092198a098ac","Type":"ContainerStarted","Data":"039b93067829dd06fb02d57a899f01f6990434e8500d22c3e64f2acf1c60c16f"} Sep 10 18:31:35.462311 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:35.462286 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zpc6" event={"ID":"898e6b12-a7af-4d34-af96-b6c1f54e66c5","Type":"ContainerStarted","Data":"5e9c5b826acf57111f19ad3a9efb1fd2164e52d151486b0bb43f28a9197d3361"} Sep 10 18:31:35.463953 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:35.463933 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4g45x" event={"ID":"bd3aed00-8ff3-4247-abc7-4893cfae63dc","Type":"ContainerStarted","Data":"65109cc2e0e0c7fdaef90e6edc6b7b4fd64476c2f7ef335d05da6e92d1429267"} Sep 10 18:31:35.465679 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:35.465655 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" event={"ID":"85c327e7-271d-4c17-b0fd-a8d4d5cfd2a1","Type":"ContainerStarted","Data":"b9e64009bd2fc84126f492eb0ac2d8cc8f9e98c5f39b5e5032ffa9e712f64d6a"} Sep 10 18:31:35.467098 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:35.467079 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jc45h" event={"ID":"70d9c337-84ae-485f-be5d-9a7c3919f3b0","Type":"ContainerStarted","Data":"0d8e454f51c9ce8479c006a7653370e54211fecb85c0f9d0b6a5eb8c8c5ac0c6"} Sep 10 18:31:35.470254 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:35.470220 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gf28x" podStartSLOduration=7.497318375 podStartE2EDuration="21.470210193s" totalImagesPullingTime="13.972891818s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 18:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:31:35.470162959 +0000 UTC m=+21.556500435" watchObservedRunningTime="2026-09-10 18:31:35.470210193 +0000 UTC m=+21.556547668" Sep 10 18:31:35.486249 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:35.486210 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jcwmv" podStartSLOduration=3.638516441 podStartE2EDuration="21.486198411s" totalImagesPullingTime="17.84768197s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 18:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:31:35.485727796 +0000 UTC m=+21.572065272" watchObservedRunningTime="2026-09-10 18:31:35.486198411 +0000 UTC m=+21.572535886" Sep 10 18:31:35.518101 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:35.518061 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lh8gr" podStartSLOduration=3.665934691 podStartE2EDuration="21.518046502s" totalImagesPullingTime="17.852111811s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 18:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:31:35.517497185 +0000 UTC m=+21.603834660" watchObservedRunningTime="2026-09-10 18:31:35.518046502 +0000 UTC m=+21.604384373" Sep 10 18:31:35.547130 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:35.547090 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jc45h" podStartSLOduration=3.678493542 podStartE2EDuration="21.547076844s" totalImagesPullingTime="17.868583302s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 18:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:31:35.546845362 +0000 UTC m=+21.633182836" watchObservedRunningTime="2026-09-10 18:31:35.547076844 +0000 UTC m=+21.633414319" Sep 10 18:31:35.547339 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:35.547304 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4g45x" podStartSLOduration=7.5739701329999995 podStartE2EDuration="21.54729899s" totalImagesPullingTime="13.973328857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:31:35.531545046 +0000 UTC m=+21.617882520" watchObservedRunningTime="2026-09-10 18:31:35.54729899 +0000 UTC m=+21.633636464" Sep 10 18:31:36.309116 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:36.308908 2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 10 18:31:36.334450 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:36.334354 2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T18:31:36.309106957Z","UUID":"6cc9e441-0457-46cc-8a9f-96a73c329c6e","Handler":null,"Name":"","Endpoint":""} Sep 10 18:31:36.337460 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:36.337435 2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 10 18:31:36.337573 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:36.337474 2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 10 18:31:36.375029 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:36.375001 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmzj" Sep 10 18:31:36.375179 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:36.375001 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-df44c" Sep 10 18:31:36.375179 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:36.375127 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cznnw" Sep 10 18:31:36.375179 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:36.375133 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqmzj" podUID="54ef3c93-1624-46e6-ba09-4fd3bb1fc07f" Sep 10 18:31:36.375307 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:36.375196 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-df44c" podUID="0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9" Sep 10 18:31:36.375307 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:36.375287 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cznnw" podUID="a43748df-f7e9-4e22-b659-34b9550d706e" Sep 10 18:31:36.473202 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:36.473084 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" event={"ID":"9eac62b3-150f-42a5-9af2-bf313bf92907","Type":"ContainerStarted","Data":"f07391d0bb8b5f2560963529e02e32e67fdf45c1d4ca7ad66df3f16d3081bccf"} Sep 10 18:31:36.473202 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:36.473183 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" event={"ID":"9eac62b3-150f-42a5-9af2-bf313bf92907","Type":"ContainerStarted","Data":"36b95f8cf9a0883234cf1f9cee15318bd1fec54eeee0c46c382685311061c4d5"} Sep 10 18:31:36.473816 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:36.473237 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" event={"ID":"9eac62b3-150f-42a5-9af2-bf313bf92907","Type":"ContainerStarted","Data":"c33e8c159d9652e3fd10bd3fa53b2c12386bccbf7da7e50e5c71430c56556298"} Sep 10 18:31:36.473816 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:36.473283 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" event={"ID":"9eac62b3-150f-42a5-9af2-bf313bf92907","Type":"ContainerStarted","Data":"24ea0b51538bf5d336e14d65085c4c59b853f7039555d0030e7ad50aed597d6d"} Sep 10 18:31:36.475127 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:36.475060 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zpc6" event={"ID":"898e6b12-a7af-4d34-af96-b6c1f54e66c5","Type":"ContainerStarted","Data":"009236b56ca678c11ee55f3b55ecd57e9c1a0a494819233634e70cd54e682468"} Sep 10 18:31:37.124087 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:37.124053 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4g45x" Sep 10 18:31:37.124695 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:37.124672 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4g45x" Sep 10 18:31:37.480300 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:37.480263 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zpc6" event={"ID":"898e6b12-a7af-4d34-af96-b6c1f54e66c5","Type":"ContainerStarted","Data":"d5a19e643930b31e95481c5f5ba8ce74f0a5c525eaaf53866cf4f3e2dc6ca2ea"} Sep 10 18:31:37.481968 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:37.481943 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hxctp" event={"ID":"d84d2e56-1f36-4691-8608-a1af00d9cce5","Type":"ContainerStarted","Data":"1faeabc2d21ca3e1ab8ce6e2996bf5bd6e21107b690e588f29485f5da7bf1b39"} Sep 10 18:31:37.500241 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:37.500203 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zpc6" podStartSLOduration=3.764134287 podStartE2EDuration="23.500189064s" totalImagesPullingTime="19.736054777s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 18:31:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:31:37.499235011 +0000 UTC m=+23.585572487" watchObservedRunningTime="2026-09-10 18:31:37.500189064 +0000 UTC m=+23.586526536" Sep 10 18:31:38.374553 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:38.374521 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-df44c" Sep 10 18:31:38.374731 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:38.374521 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmzj" Sep 10 18:31:38.374731 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:38.374650 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-df44c" podUID="0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9" Sep 10 18:31:38.374731 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:38.374697 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cznnw" Sep 10 18:31:38.374885 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:38.374828 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqmzj" podUID="54ef3c93-1624-46e6-ba09-4fd3bb1fc07f" Sep 10 18:31:38.374948 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:38.374926 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cznnw" podUID="a43748df-f7e9-4e22-b659-34b9550d706e" Sep 10 18:31:38.489971 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:38.489937 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" event={"ID":"9eac62b3-150f-42a5-9af2-bf313bf92907","Type":"ContainerStarted","Data":"6064a16c153c901353900bd8a3afbd381b2e5d14e4fd2a9133a0eca324d12cf7"} Sep 10 18:31:38.490434 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:38.489986 2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 10 18:31:40.374653 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:40.374434 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cznnw" Sep 10 18:31:40.375069 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:40.374456 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-df44c" Sep 10 18:31:40.375069 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:40.374468 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmzj" Sep 10 18:31:40.375069 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:40.374796 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-df44c" podUID="0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9" Sep 10 18:31:40.375069 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:40.374885 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqmzj" podUID="54ef3c93-1624-46e6-ba09-4fd3bb1fc07f" Sep 10 18:31:40.375069 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:40.374679 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cznnw" podUID="a43748df-f7e9-4e22-b659-34b9550d706e" Sep 10 18:31:40.497339 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:40.497295 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" event={"ID":"9eac62b3-150f-42a5-9af2-bf313bf92907","Type":"ContainerStarted","Data":"5ed44036c6f41dfbf4305375dd6ae19f0c05ac113405775b0f48b6d94a8204a4"} Sep 10 18:31:40.497532 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:40.497510 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:40.497624 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:40.497584 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:40.503078 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:40.503054 2581 generic.go:356] "Generic (PLEG): container finished" podID="747cbe9d-036e-448f-b567-6ffd3c917b2e" containerID="e0262660af6ba28332fbaa57ef6f1f0111865898745c496f0befbd3b90aeb121" exitCode=0 Sep 10 18:31:40.503204 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:40.503096 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qrb6h" event={"ID":"747cbe9d-036e-448f-b567-6ffd3c917b2e","Type":"ContainerDied","Data":"e0262660af6ba28332fbaa57ef6f1f0111865898745c496f0befbd3b90aeb121"} Sep 10 18:31:40.513961 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:40.513942 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:40.519785 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:40.519768 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:40.529384 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:40.529350 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" podStartSLOduration=8.584462861 podStartE2EDuration="26.529319015s" totalImagesPullingTime="17.944856154s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 18:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:31:40.52897292 +0000 UTC m=+26.615310398" watchObservedRunningTime="2026-09-10 18:31:40.529319015 +0000 UTC m=+26.615656507" Sep 10 18:31:40.529686 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:40.529666 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hxctp" podStartSLOduration=9.037025602 podStartE2EDuration="26.529660325s" totalImagesPullingTime="17.492634723s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 18:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:31:37.513201845 +0000 UTC m=+23.599539320" watchObservedRunningTime="2026-09-10 18:31:40.529660325 +0000 UTC m=+26.615997863" Sep 10 18:31:41.033669 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:41.033640 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:31:41.139117 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:41.139083 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4g45x" Sep 10 18:31:41.139289 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:41.139215 2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 10 18:31:41.140286 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:41.140256 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4g45x" Sep 10 18:31:41.770999 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:41.770772 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-df44c"] Sep 10 18:31:41.771622 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:41.771050 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-df44c" Sep 10 18:31:41.771622 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:41.771137 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-df44c" podUID="0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9" Sep 10 18:31:41.774007 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:41.773970 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cznnw"] Sep 10 18:31:41.774139 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:41.774083 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cznnw" Sep 10 18:31:41.774209 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:41.774170 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cznnw" podUID="a43748df-f7e9-4e22-b659-34b9550d706e" Sep 10 18:31:41.774806 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:41.774689 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fqmzj"] Sep 10 18:31:41.774806 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:41.774792 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmzj" Sep 10 18:31:41.774979 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:41.774893 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqmzj" podUID="54ef3c93-1624-46e6-ba09-4fd3bb1fc07f" Sep 10 18:31:42.512520 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:42.512486 2581 generic.go:356] "Generic (PLEG): container finished" podID="747cbe9d-036e-448f-b567-6ffd3c917b2e" containerID="39582959a53de2e93bf268822a31901ceeb7e6d39420edd4358c457b331649d7" exitCode=0 Sep 10 18:31:42.512691 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:42.512582 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qrb6h" event={"ID":"747cbe9d-036e-448f-b567-6ffd3c917b2e","Type":"ContainerDied","Data":"39582959a53de2e93bf268822a31901ceeb7e6d39420edd4358c457b331649d7"} Sep 10 18:31:43.374823 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:43.374791 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cznnw" Sep 10 18:31:43.375168 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:43.374912 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cznnw" podUID="a43748df-f7e9-4e22-b659-34b9550d706e" Sep 10 18:31:43.375168 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:43.374966 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-df44c" Sep 10 18:31:43.375168 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:43.375030 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-df44c" podUID="0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9" Sep 10 18:31:43.375168 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:43.375090 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmzj" Sep 10 18:31:43.375395 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:43.375169 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqmzj" podUID="54ef3c93-1624-46e6-ba09-4fd3bb1fc07f" Sep 10 18:31:43.517851 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:43.517822 2581 generic.go:356] "Generic (PLEG): container finished" podID="747cbe9d-036e-448f-b567-6ffd3c917b2e" containerID="f6aff0a4fc5cdfac641f69ac1558f65aae19aa6374c7f36415b36c48bd4a3dc2" exitCode=0 Sep 10 18:31:43.518066 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:43.517864 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qrb6h" event={"ID":"747cbe9d-036e-448f-b567-6ffd3c917b2e","Type":"ContainerDied","Data":"f6aff0a4fc5cdfac641f69ac1558f65aae19aa6374c7f36415b36c48bd4a3dc2"} Sep 10 18:31:44.897303 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:44.897256 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:31:44.983899 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:44.983873 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:31:45.374841 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:45.374811 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cznnw" Sep 10 18:31:45.375026 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:45.374856 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmzj" Sep 10 18:31:45.375026 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:45.374922 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-df44c" Sep 10 18:31:45.375026 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:45.374937 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cznnw" podUID="a43748df-f7e9-4e22-b659-34b9550d706e" Sep 10 18:31:45.375182 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:45.375040 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-df44c" podUID="0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9" Sep 10 18:31:45.375182 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:45.375093 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqmzj" podUID="54ef3c93-1624-46e6-ba09-4fd3bb1fc07f" Sep 10 18:31:47.374286 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:47.374252 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cznnw" Sep 10 18:31:47.374767 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:47.374369 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cznnw" podUID="a43748df-f7e9-4e22-b659-34b9550d706e" Sep 10 18:31:47.374767 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:47.374376 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-df44c" Sep 10 18:31:47.374767 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:47.374463 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-df44c" podUID="0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9" Sep 10 18:31:47.374767 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:47.374383 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmzj" Sep 10 18:31:47.374767 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:47.374535 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqmzj" podUID="54ef3c93-1624-46e6-ba09-4fd3bb1fc07f" Sep 10 18:31:47.715494 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:47.715467 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-166.ec2.internal" event="NodeReady" Sep 10 18:31:47.715657 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:47.715620 2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 10 18:31:47.756382 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:47.756355 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4t7vn"] Sep 10 18:31:47.774774 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:47.774744 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-drn5f"] Sep 10 18:31:47.774929 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:47.774882 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4t7vn" Sep 10 18:31:47.776685 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:47.776664 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4lj58\"" Sep 10 18:31:47.776952 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:47.776817 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 10 18:31:47.777056 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:47.776955 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 10 18:31:47.786429 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:47.786412 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4t7vn"] Sep 10 18:31:47.786515 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:47.786442 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-drn5f"] Sep 10 18:31:47.786569 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:47.786525 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-drn5f" Sep 10 18:31:47.788829 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:47.788552 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 10 18:31:47.788829 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:47.788763 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 10 18:31:47.789130 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:47.789111 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 10 18:31:47.789239 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:47.789155 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-h28kp\"" Sep 10 18:31:47.907124 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:47.907057 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-config-volume\") pod \"dns-default-4t7vn\" (UID: \"2c41b6ab-b71c-4b51-9cf7-4deba7a3a113\") " pod="openshift-dns/dns-default-4t7vn" Sep 10 18:31:47.907124 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:47.907090 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd225\" (UniqueName: \"kubernetes.io/projected/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-kube-api-access-wd225\") pod \"dns-default-4t7vn\" (UID: \"2c41b6ab-b71c-4b51-9cf7-4deba7a3a113\") " pod="openshift-dns/dns-default-4t7vn" Sep 10 18:31:47.907314 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:47.907129 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7470a24c-2464-4678-9015-4689638c6b57-cert\") pod \"ingress-canary-drn5f\" (UID: \"7470a24c-2464-4678-9015-4689638c6b57\") " pod="openshift-ingress-canary/ingress-canary-drn5f" Sep 10 18:31:47.907314 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:47.907155 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sxk5\" (UniqueName: \"kubernetes.io/projected/7470a24c-2464-4678-9015-4689638c6b57-kube-api-access-5sxk5\") pod \"ingress-canary-drn5f\" (UID: \"7470a24c-2464-4678-9015-4689638c6b57\") " pod="openshift-ingress-canary/ingress-canary-drn5f" Sep 10 18:31:47.907314 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:47.907255 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-tmp-dir\") pod \"dns-default-4t7vn\" (UID: \"2c41b6ab-b71c-4b51-9cf7-4deba7a3a113\") " pod="openshift-dns/dns-default-4t7vn" Sep 10 18:31:47.907314 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:47.907293 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-metrics-tls\") pod \"dns-default-4t7vn\" (UID: \"2c41b6ab-b71c-4b51-9cf7-4deba7a3a113\") " pod="openshift-dns/dns-default-4t7vn" Sep 10 18:31:48.008376 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:48.008348 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-tmp-dir\") pod \"dns-default-4t7vn\" (UID: \"2c41b6ab-b71c-4b51-9cf7-4deba7a3a113\") " pod="openshift-dns/dns-default-4t7vn" Sep 10 18:31:48.008567 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:48.008387 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-metrics-tls\") pod \"dns-default-4t7vn\" (UID: \"2c41b6ab-b71c-4b51-9cf7-4deba7a3a113\") " pod="openshift-dns/dns-default-4t7vn" Sep 10 18:31:48.008567 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:48.008503 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 18:31:48.008567 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:48.008526 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-config-volume\") pod \"dns-default-4t7vn\" (UID: \"2c41b6ab-b71c-4b51-9cf7-4deba7a3a113\") " pod="openshift-dns/dns-default-4t7vn" Sep 10 18:31:48.008567 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:48.008562 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-metrics-tls podName:2c41b6ab-b71c-4b51-9cf7-4deba7a3a113 nodeName:}" failed. No retries permitted until 2026-09-10 18:31:48.508542802 +0000 UTC m=+34.594880278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-metrics-tls") pod "dns-default-4t7vn" (UID: "2c41b6ab-b71c-4b51-9cf7-4deba7a3a113") : secret "dns-default-metrics-tls" not found Sep 10 18:31:48.008745 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:48.008579 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd225\" (UniqueName: \"kubernetes.io/projected/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-kube-api-access-wd225\") pod \"dns-default-4t7vn\" (UID: \"2c41b6ab-b71c-4b51-9cf7-4deba7a3a113\") " pod="openshift-dns/dns-default-4t7vn" Sep 10 18:31:48.008745 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:48.008622 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7470a24c-2464-4678-9015-4689638c6b57-cert\") pod \"ingress-canary-drn5f\" (UID: \"7470a24c-2464-4678-9015-4689638c6b57\") " pod="openshift-ingress-canary/ingress-canary-drn5f" Sep 10 18:31:48.008745 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:48.008650 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sxk5\" (UniqueName: \"kubernetes.io/projected/7470a24c-2464-4678-9015-4689638c6b57-kube-api-access-5sxk5\") pod \"ingress-canary-drn5f\" (UID: \"7470a24c-2464-4678-9015-4689638c6b57\") " pod="openshift-ingress-canary/ingress-canary-drn5f" Sep 10 18:31:48.008887 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:48.008746 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-tmp-dir\") pod \"dns-default-4t7vn\" (UID: \"2c41b6ab-b71c-4b51-9cf7-4deba7a3a113\") " pod="openshift-dns/dns-default-4t7vn" Sep 10 18:31:48.009004 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:48.008985 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-config-volume\") pod \"dns-default-4t7vn\" (UID: \"2c41b6ab-b71c-4b51-9cf7-4deba7a3a113\") " pod="openshift-dns/dns-default-4t7vn" Sep 10 18:31:48.014158 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:48.014140 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7470a24c-2464-4678-9015-4689638c6b57-cert\") pod \"ingress-canary-drn5f\" (UID: \"7470a24c-2464-4678-9015-4689638c6b57\") " pod="openshift-ingress-canary/ingress-canary-drn5f" Sep 10 18:31:48.020847 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:48.020816 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sxk5\" (UniqueName: \"kubernetes.io/projected/7470a24c-2464-4678-9015-4689638c6b57-kube-api-access-5sxk5\") pod \"ingress-canary-drn5f\" (UID: \"7470a24c-2464-4678-9015-4689638c6b57\") " pod="openshift-ingress-canary/ingress-canary-drn5f" Sep 10 18:31:48.021510 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:48.021484 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd225\" (UniqueName: \"kubernetes.io/projected/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-kube-api-access-wd225\") pod \"dns-default-4t7vn\" (UID: \"2c41b6ab-b71c-4b51-9cf7-4deba7a3a113\") " pod="openshift-dns/dns-default-4t7vn" Sep 10 18:31:48.095641 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:48.095613 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-drn5f" Sep 10 18:31:48.109972 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:48.109948 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9-original-pull-secret\") pod \"global-pull-secret-syncer-df44c\" (UID: \"0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9\") " pod="kube-system/global-pull-secret-syncer-df44c" Sep 10 18:31:48.110079 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:48.109997 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54ef3c93-1624-46e6-ba09-4fd3bb1fc07f-metrics-certs\") pod \"network-metrics-daemon-fqmzj\" (UID: \"54ef3c93-1624-46e6-ba09-4fd3bb1fc07f\") " pod="openshift-multus/network-metrics-daemon-fqmzj" Sep 10 18:31:48.110142 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:48.110105 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 18:31:48.110142 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:48.110113 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 18:31:48.110241 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:48.110167 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54ef3c93-1624-46e6-ba09-4fd3bb1fc07f-metrics-certs podName:54ef3c93-1624-46e6-ba09-4fd3bb1fc07f nodeName:}" failed. No retries permitted until 2026-09-10 18:32:20.110149394 +0000 UTC m=+66.196486862 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54ef3c93-1624-46e6-ba09-4fd3bb1fc07f-metrics-certs") pod "network-metrics-daemon-fqmzj" (UID: "54ef3c93-1624-46e6-ba09-4fd3bb1fc07f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 18:31:48.110241 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:48.110187 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9-original-pull-secret podName:0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9 nodeName:}" failed. No retries permitted until 2026-09-10 18:32:20.110177365 +0000 UTC m=+66.196514824 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9-original-pull-secret") pod "global-pull-secret-syncer-df44c" (UID: "0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9") : object "kube-system"/"original-pull-secret" not registered Sep 10 18:31:48.210819 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:48.210739 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhxb6\" (UniqueName: \"kubernetes.io/projected/a43748df-f7e9-4e22-b659-34b9550d706e-kube-api-access-xhxb6\") pod \"network-check-target-cznnw\" (UID: \"a43748df-f7e9-4e22-b659-34b9550d706e\") " pod="openshift-network-diagnostics/network-check-target-cznnw" Sep 10 18:31:48.210972 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:48.210914 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 18:31:48.210972 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:48.210941 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 18:31:48.210972 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:48.210956 2581 projected.go:196] Error preparing data for projected volume kube-api-access-xhxb6 for pod openshift-network-diagnostics/network-check-target-cznnw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 18:31:48.211091 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:48.211022 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a43748df-f7e9-4e22-b659-34b9550d706e-kube-api-access-xhxb6 podName:a43748df-f7e9-4e22-b659-34b9550d706e nodeName:}" failed. No retries permitted until 2026-09-10 18:32:20.211003414 +0000 UTC m=+66.297340887 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xhxb6" (UniqueName: "kubernetes.io/projected/a43748df-f7e9-4e22-b659-34b9550d706e-kube-api-access-xhxb6") pod "network-check-target-cznnw" (UID: "a43748df-f7e9-4e22-b659-34b9550d706e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 18:31:48.513341 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:48.513042 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-metrics-tls\") pod \"dns-default-4t7vn\" (UID: \"2c41b6ab-b71c-4b51-9cf7-4deba7a3a113\") " pod="openshift-dns/dns-default-4t7vn" Sep 10 18:31:48.513837 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:48.513221 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 18:31:48.513837 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:48.513421 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-metrics-tls podName:2c41b6ab-b71c-4b51-9cf7-4deba7a3a113 nodeName:}" failed. No retries permitted until 2026-09-10 18:31:49.513396062 +0000 UTC m=+35.599733536 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-metrics-tls") pod "dns-default-4t7vn" (UID: "2c41b6ab-b71c-4b51-9cf7-4deba7a3a113") : secret "dns-default-metrics-tls" not found Sep 10 18:31:49.369921 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:49.369843 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-drn5f"] Sep 10 18:31:49.374090 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:49.374060 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cznnw" Sep 10 18:31:49.374090 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:49.374073 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmzj" Sep 10 18:31:49.374216 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:49.374078 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-df44c" Sep 10 18:31:49.376121 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:49.376102 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-thvkz\"" Sep 10 18:31:49.376624 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:49.376606 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 18:31:49.376717 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:49.376634 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 18:31:49.376717 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:49.376672 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dxnbp\"" Sep 10 18:31:49.376717 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:49.376681 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 18:31:49.377383 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:49.377365 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 18:31:49.395892 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:31:49.395873 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7470a24c_2464_4678_9015_4689638c6b57.slice/crio-be8b4e805dc64eb56050f65f55ca6ee71ac003588c964544caf84e6e8bf88f72 WatchSource:0}: Error finding container be8b4e805dc64eb56050f65f55ca6ee71ac003588c964544caf84e6e8bf88f72: Status 404 returned error can't find the container with id be8b4e805dc64eb56050f65f55ca6ee71ac003588c964544caf84e6e8bf88f72 Sep 10 18:31:49.521809 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:49.521782 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-metrics-tls\") pod \"dns-default-4t7vn\" (UID: \"2c41b6ab-b71c-4b51-9cf7-4deba7a3a113\") " pod="openshift-dns/dns-default-4t7vn" Sep 10 18:31:49.522239 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:49.521909 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 18:31:49.522239 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:49.521977 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-metrics-tls podName:2c41b6ab-b71c-4b51-9cf7-4deba7a3a113 nodeName:}" failed. No retries permitted until 2026-09-10 18:31:51.521957968 +0000 UTC m=+37.608295441 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-metrics-tls") pod "dns-default-4t7vn" (UID: "2c41b6ab-b71c-4b51-9cf7-4deba7a3a113") : secret "dns-default-metrics-tls" not found Sep 10 18:31:49.532270 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:49.532250 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-drn5f" event={"ID":"7470a24c-2464-4678-9015-4689638c6b57","Type":"ContainerStarted","Data":"be8b4e805dc64eb56050f65f55ca6ee71ac003588c964544caf84e6e8bf88f72"} Sep 10 18:31:49.689134 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:49.689101 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:31:49.790152 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:49.790125 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:31:50.539549 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:50.539478 2581 generic.go:356] "Generic (PLEG): container finished" podID="747cbe9d-036e-448f-b567-6ffd3c917b2e" containerID="95481b02890d2e59651cb4d82c3d426ce7ddd7df44263aace9f8b975cef4c88a" exitCode=0 Sep 10 18:31:50.539549 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:50.539530 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qrb6h" event={"ID":"747cbe9d-036e-448f-b567-6ffd3c917b2e","Type":"ContainerDied","Data":"95481b02890d2e59651cb4d82c3d426ce7ddd7df44263aace9f8b975cef4c88a"} Sep 10 18:31:51.537754 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:51.537718 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-metrics-tls\") pod \"dns-default-4t7vn\" (UID: \"2c41b6ab-b71c-4b51-9cf7-4deba7a3a113\") " pod="openshift-dns/dns-default-4t7vn" Sep 10 18:31:51.537928 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:51.537887 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 18:31:51.537982 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:51.537968 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-metrics-tls podName:2c41b6ab-b71c-4b51-9cf7-4deba7a3a113 nodeName:}" failed. No retries permitted until 2026-09-10 18:31:55.537947249 +0000 UTC m=+41.624284721 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-metrics-tls") pod "dns-default-4t7vn" (UID: "2c41b6ab-b71c-4b51-9cf7-4deba7a3a113") : secret "dns-default-metrics-tls" not found Sep 10 18:31:51.545547 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:51.545519 2581 generic.go:356] "Generic (PLEG): container finished" podID="747cbe9d-036e-448f-b567-6ffd3c917b2e" containerID="ee1eb4636988868499d032f9cc0e7dccf38c3b77979ba4f38ba62825fbdad7f3" exitCode=0 Sep 10 18:31:51.546044 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:51.545591 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qrb6h" event={"ID":"747cbe9d-036e-448f-b567-6ffd3c917b2e","Type":"ContainerDied","Data":"ee1eb4636988868499d032f9cc0e7dccf38c3b77979ba4f38ba62825fbdad7f3"} Sep 10 18:31:52.285735 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:52.285509 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:31:52.352665 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:52.352635 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:31:52.551242 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:52.551214 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qrb6h" event={"ID":"747cbe9d-036e-448f-b567-6ffd3c917b2e","Type":"ContainerStarted","Data":"38e88029440e23ec8e0a43217f930c929888a7ff8be53c9672eacd46fd870ec5"} Sep 10 18:31:52.552769 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:52.552749 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-drn5f" event={"ID":"7470a24c-2464-4678-9015-4689638c6b57","Type":"ContainerStarted","Data":"22f3518891988cdc640877373f69485ffb0ea4722cb4a97617ecf7c153d42f85"} Sep 10 18:31:52.572009 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:52.571969 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qrb6h" podStartSLOduration=8.603352231 podStartE2EDuration="38.571956489s" totalImagesPullingTime="29.968604258s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 18:31:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:31:52.571008704 +0000 UTC m=+38.657346224" watchObservedRunningTime="2026-09-10 18:31:52.571956489 +0000 UTC m=+38.658293964" Sep 10 18:31:52.586109 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:52.586072 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-drn5f" podStartSLOduration=3.021181067 podStartE2EDuration="5.58606422s" totalImagesPullingTime="2.564883153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:31:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:31:52.585693426 +0000 UTC m=+38.672030903" watchObservedRunningTime="2026-09-10 18:31:52.58606422 +0000 UTC m=+38.672401694" Sep 10 18:31:52.687656 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:52.687626 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:31:52.812551 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:52.812462 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:31:52.871004 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:52.870978 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:31:53.561557 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:53.561530 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:31:53.617502 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:53.617475 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:31:54.223695 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:54.223660 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:31:54.287238 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:54.287214 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:31:55.567794 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:55.567759 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-metrics-tls\") pod \"dns-default-4t7vn\" (UID: \"2c41b6ab-b71c-4b51-9cf7-4deba7a3a113\") " pod="openshift-dns/dns-default-4t7vn" Sep 10 18:31:55.568235 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:55.567891 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 18:31:55.568235 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:31:55.567959 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-metrics-tls podName:2c41b6ab-b71c-4b51-9cf7-4deba7a3a113 nodeName:}" failed. No retries permitted until 2026-09-10 18:32:03.567941862 +0000 UTC m=+49.654279525 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-metrics-tls") pod "dns-default-4t7vn" (UID: "2c41b6ab-b71c-4b51-9cf7-4deba7a3a113") : secret "dns-default-metrics-tls" not found Sep 10 18:31:56.165127 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:56.165091 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:31:56.265633 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:31:56.265607 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:32:03.622293 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:03.622262 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-metrics-tls\") pod \"dns-default-4t7vn\" (UID: \"2c41b6ab-b71c-4b51-9cf7-4deba7a3a113\") " pod="openshift-dns/dns-default-4t7vn" Sep 10 18:32:03.622716 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:32:03.622401 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 18:32:03.622716 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:32:03.622451 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-metrics-tls podName:2c41b6ab-b71c-4b51-9cf7-4deba7a3a113 nodeName:}" failed. No retries permitted until 2026-09-10 18:32:19.622438672 +0000 UTC m=+65.708776138 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-metrics-tls") pod "dns-default-4t7vn" (UID: "2c41b6ab-b71c-4b51-9cf7-4deba7a3a113") : secret "dns-default-metrics-tls" not found Sep 10 18:32:11.774467 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:11.774434 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:32:11.874094 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:11.874064 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:32:12.523181 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:12.523154 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9j6cn" Sep 10 18:32:12.674466 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:12.674436 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:32:12.728580 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:12.728555 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:32:17.170216 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:17.170166 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:32:17.230607 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:17.230579 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:32:17.286160 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:17.286124 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:32:19.630536 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:19.630508 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-metrics-tls\") pod \"dns-default-4t7vn\" (UID: \"2c41b6ab-b71c-4b51-9cf7-4deba7a3a113\") " pod="openshift-dns/dns-default-4t7vn" Sep 10 18:32:19.630919 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:32:19.630674 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 18:32:19.630919 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:32:19.630751 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-metrics-tls podName:2c41b6ab-b71c-4b51-9cf7-4deba7a3a113 nodeName:}" failed. No retries permitted until 2026-09-10 18:32:51.63073158 +0000 UTC m=+97.717069057 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-metrics-tls") pod "dns-default-4t7vn" (UID: "2c41b6ab-b71c-4b51-9cf7-4deba7a3a113") : secret "dns-default-metrics-tls" not found Sep 10 18:32:20.134304 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:20.134259 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9-original-pull-secret\") pod \"global-pull-secret-syncer-df44c\" (UID: \"0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9\") " pod="kube-system/global-pull-secret-syncer-df44c" Sep 10 18:32:20.134527 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:20.134339 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54ef3c93-1624-46e6-ba09-4fd3bb1fc07f-metrics-certs\") pod \"network-metrics-daemon-fqmzj\" (UID: \"54ef3c93-1624-46e6-ba09-4fd3bb1fc07f\") " pod="openshift-multus/network-metrics-daemon-fqmzj" Sep 10 18:32:20.136359 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:20.136339 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 18:32:20.136475 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:20.136456 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 18:32:20.145241 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:32:20.145220 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 10 18:32:20.145369 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:32:20.145288 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54ef3c93-1624-46e6-ba09-4fd3bb1fc07f-metrics-certs podName:54ef3c93-1624-46e6-ba09-4fd3bb1fc07f nodeName:}" failed. No retries permitted until 2026-09-10 18:33:24.145272499 +0000 UTC m=+130.231609966 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54ef3c93-1624-46e6-ba09-4fd3bb1fc07f-metrics-certs") pod "network-metrics-daemon-fqmzj" (UID: "54ef3c93-1624-46e6-ba09-4fd3bb1fc07f") : secret "metrics-daemon-secret" not found Sep 10 18:32:20.148569 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:20.148547 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9-original-pull-secret\") pod \"global-pull-secret-syncer-df44c\" (UID: \"0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9\") " pod="kube-system/global-pull-secret-syncer-df44c" Sep 10 18:32:20.235082 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:20.235054 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhxb6\" (UniqueName: \"kubernetes.io/projected/a43748df-f7e9-4e22-b659-34b9550d706e-kube-api-access-xhxb6\") pod \"network-check-target-cznnw\" (UID: \"a43748df-f7e9-4e22-b659-34b9550d706e\") " pod="openshift-network-diagnostics/network-check-target-cznnw" Sep 10 18:32:20.237053 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:20.237037 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 18:32:20.247166 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:20.247146 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 18:32:20.257875 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:20.257855 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhxb6\" (UniqueName: \"kubernetes.io/projected/a43748df-f7e9-4e22-b659-34b9550d706e-kube-api-access-xhxb6\") pod \"network-check-target-cznnw\" (UID: \"a43748df-f7e9-4e22-b659-34b9550d706e\") " pod="openshift-network-diagnostics/network-check-target-cznnw" Sep 10 18:32:20.290875 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:20.290855 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-df44c" Sep 10 18:32:20.296301 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:20.296287 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dxnbp\"" Sep 10 18:32:20.304886 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:20.304869 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cznnw" Sep 10 18:32:20.559857 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:20.559827 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-df44c"] Sep 10 18:32:20.563465 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:32:20.563435 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b6bd408_6ab0_48c2_bbb3_04e7e1146ff9.slice/crio-c8555c30a8f1c2f84afdf428584e72e1d5394f6eacbf013aa649d1999452738b WatchSource:0}: Error finding container c8555c30a8f1c2f84afdf428584e72e1d5394f6eacbf013aa649d1999452738b: Status 404 returned error can't find the container with id c8555c30a8f1c2f84afdf428584e72e1d5394f6eacbf013aa649d1999452738b Sep 10 18:32:20.579226 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:20.579203 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cznnw"] Sep 10 18:32:20.582481 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:32:20.582458 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda43748df_f7e9_4e22_b659_34b9550d706e.slice/crio-80ee6e13745837438f73f86bcd4ba2ebf8ef823b95ad428442a5d97876140050 WatchSource:0}: Error finding container 80ee6e13745837438f73f86bcd4ba2ebf8ef823b95ad428442a5d97876140050: Status 404 returned error can't find the container with id 80ee6e13745837438f73f86bcd4ba2ebf8ef823b95ad428442a5d97876140050 Sep 10 18:32:20.618108 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:20.618075 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cznnw" event={"ID":"a43748df-f7e9-4e22-b659-34b9550d706e","Type":"ContainerStarted","Data":"80ee6e13745837438f73f86bcd4ba2ebf8ef823b95ad428442a5d97876140050"} Sep 10 18:32:20.619241 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:20.619222 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-df44c" event={"ID":"0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9","Type":"ContainerStarted","Data":"c8555c30a8f1c2f84afdf428584e72e1d5394f6eacbf013aa649d1999452738b"} Sep 10 18:32:24.932897 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:24.932860 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:32:24.995625 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:24.995556 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:32:25.634949 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:25.634852 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cznnw" event={"ID":"a43748df-f7e9-4e22-b659-34b9550d706e","Type":"ContainerStarted","Data":"e642e82dc5732cb2a635228e1a7e1c248a255b9fe277a752861dd0268fde4ade"} Sep 10 18:32:25.634949 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:25.634918 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cznnw" Sep 10 18:32:25.636466 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:25.636440 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-df44c" event={"ID":"0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9","Type":"ContainerStarted","Data":"f4b84a6de9ff51250cb1103d29b550719d739c2df8b2136030834ae690485b4c"} Sep 10 18:32:25.649343 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:25.649289 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cznnw" podStartSLOduration=66.83991082 podStartE2EDuration="1m11.649276888s" totalImagesPullingTime="4.809366068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:32:25.648830884 +0000 UTC m=+71.735168360" watchObservedRunningTime="2026-09-10 18:32:25.649276888 +0000 UTC m=+71.735614362" Sep 10 18:32:25.660713 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:25.660584 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-df44c" podStartSLOduration=66.821814012 podStartE2EDuration="1m11.660567939s" totalImagesPullingTime="4.838753927s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 18:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:32:25.66025393 +0000 UTC m=+71.746591404" watchObservedRunningTime="2026-09-10 18:32:25.660567939 +0000 UTC m=+71.746905414" Sep 10 18:32:49.103763 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:49.103640 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:32:49.152334 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:49.152294 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:32:51.728088 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:51.728007 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-metrics-tls\") pod \"dns-default-4t7vn\" (UID: \"2c41b6ab-b71c-4b51-9cf7-4deba7a3a113\") " pod="openshift-dns/dns-default-4t7vn" Sep 10 18:32:51.728460 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:32:51.728112 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 18:32:51.728460 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:32:51.728162 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-metrics-tls podName:2c41b6ab-b71c-4b51-9cf7-4deba7a3a113 nodeName:}" failed. No retries permitted until 2026-09-10 18:33:55.728148289 +0000 UTC m=+161.814485742 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-metrics-tls") pod "dns-default-4t7vn" (UID: "2c41b6ab-b71c-4b51-9cf7-4deba7a3a113") : secret "dns-default-metrics-tls" not found Sep 10 18:32:52.950166 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:52.950138 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fff5d7858-jxwdj"] Sep 10 18:32:52.954317 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:52.954301 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fff5d7858-jxwdj" Sep 10 18:32:52.956316 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:52.956288 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 10 18:32:52.956924 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:52.956896 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 10 18:32:52.957028 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:52.956934 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 10 18:32:52.957028 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:52.956933 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 10 18:32:52.959696 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:52.959664 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fff5d7858-jxwdj"] Sep 10 18:32:52.978915 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:52.978892 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-857d768544-bvllr"] Sep 10 18:32:52.983593 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:52.983578 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-857d768544-bvllr" Sep 10 18:32:52.985617 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:52.985596 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 10 18:32:52.985617 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:52.985610 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 10 18:32:52.985753 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:52.985617 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 10 18:32:52.985753 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:52.985647 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 10 18:32:52.991784 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:52.991767 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-857d768544-bvllr"] Sep 10 18:32:53.034977 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:53.034949 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/763222b7-e3e3-4ff4-9ef8-65240f20246a-ca\") pod \"cluster-proxy-proxy-agent-857d768544-bvllr\" (UID: \"763222b7-e3e3-4ff4-9ef8-65240f20246a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-857d768544-bvllr" Sep 10 18:32:53.035109 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:53.034984 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/763222b7-e3e3-4ff4-9ef8-65240f20246a-hub\") pod \"cluster-proxy-proxy-agent-857d768544-bvllr\" (UID: \"763222b7-e3e3-4ff4-9ef8-65240f20246a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-857d768544-bvllr" Sep 10 18:32:53.035109 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:53.035012 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ce1097c4-8d40-4aa3-99b9-dc0125515b9b-klusterlet-config\") pod \"klusterlet-addon-workmgr-5fff5d7858-jxwdj\" (UID: \"ce1097c4-8d40-4aa3-99b9-dc0125515b9b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fff5d7858-jxwdj" Sep 10 18:32:53.035109 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:53.035063 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/763222b7-e3e3-4ff4-9ef8-65240f20246a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-857d768544-bvllr\" (UID: \"763222b7-e3e3-4ff4-9ef8-65240f20246a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-857d768544-bvllr" Sep 10 18:32:53.035109 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:53.035099 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntzdf\" (UniqueName: \"kubernetes.io/projected/ce1097c4-8d40-4aa3-99b9-dc0125515b9b-kube-api-access-ntzdf\") pod \"klusterlet-addon-workmgr-5fff5d7858-jxwdj\" (UID: \"ce1097c4-8d40-4aa3-99b9-dc0125515b9b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fff5d7858-jxwdj" Sep 10 18:32:53.035317 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:53.035133 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/763222b7-e3e3-4ff4-9ef8-65240f20246a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-857d768544-bvllr\" (UID: \"763222b7-e3e3-4ff4-9ef8-65240f20246a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-857d768544-bvllr" Sep 10 18:32:53.035317 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:53.035163 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mfn4\" (UniqueName: \"kubernetes.io/projected/763222b7-e3e3-4ff4-9ef8-65240f20246a-kube-api-access-6mfn4\") pod \"cluster-proxy-proxy-agent-857d768544-bvllr\" (UID: \"763222b7-e3e3-4ff4-9ef8-65240f20246a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-857d768544-bvllr" Sep 10 18:32:53.035317 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:53.035202 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce1097c4-8d40-4aa3-99b9-dc0125515b9b-tmp\") pod \"klusterlet-addon-workmgr-5fff5d7858-jxwdj\" (UID: \"ce1097c4-8d40-4aa3-99b9-dc0125515b9b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fff5d7858-jxwdj" Sep 10 18:32:53.035317 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:53.035230 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/763222b7-e3e3-4ff4-9ef8-65240f20246a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-857d768544-bvllr\" (UID: \"763222b7-e3e3-4ff4-9ef8-65240f20246a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-857d768544-bvllr" Sep 10 18:32:53.136315 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:53.136280 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/763222b7-e3e3-4ff4-9ef8-65240f20246a-hub\") pod \"cluster-proxy-proxy-agent-857d768544-bvllr\" (UID: \"763222b7-e3e3-4ff4-9ef8-65240f20246a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-857d768544-bvllr" Sep 10 18:32:53.136315 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:53.136337 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ce1097c4-8d40-4aa3-99b9-dc0125515b9b-klusterlet-config\") pod \"klusterlet-addon-workmgr-5fff5d7858-jxwdj\" (UID: \"ce1097c4-8d40-4aa3-99b9-dc0125515b9b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fff5d7858-jxwdj" Sep 10 18:32:53.136486 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:53.136461 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/763222b7-e3e3-4ff4-9ef8-65240f20246a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-857d768544-bvllr\" (UID: \"763222b7-e3e3-4ff4-9ef8-65240f20246a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-857d768544-bvllr" Sep 10 18:32:53.136540 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:53.136494 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntzdf\" (UniqueName: \"kubernetes.io/projected/ce1097c4-8d40-4aa3-99b9-dc0125515b9b-kube-api-access-ntzdf\") pod \"klusterlet-addon-workmgr-5fff5d7858-jxwdj\" (UID: \"ce1097c4-8d40-4aa3-99b9-dc0125515b9b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fff5d7858-jxwdj" Sep 10 18:32:53.136540 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:53.136517 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/763222b7-e3e3-4ff4-9ef8-65240f20246a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-857d768544-bvllr\" (UID: \"763222b7-e3e3-4ff4-9ef8-65240f20246a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-857d768544-bvllr" Sep 10 18:32:53.136540 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:53.136536 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mfn4\" (UniqueName: \"kubernetes.io/projected/763222b7-e3e3-4ff4-9ef8-65240f20246a-kube-api-access-6mfn4\") pod \"cluster-proxy-proxy-agent-857d768544-bvllr\" (UID: \"763222b7-e3e3-4ff4-9ef8-65240f20246a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-857d768544-bvllr" Sep 10 18:32:53.136691 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:53.136563 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce1097c4-8d40-4aa3-99b9-dc0125515b9b-tmp\") pod \"klusterlet-addon-workmgr-5fff5d7858-jxwdj\" (UID: \"ce1097c4-8d40-4aa3-99b9-dc0125515b9b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fff5d7858-jxwdj" Sep 10 18:32:53.136691 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:53.136588 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/763222b7-e3e3-4ff4-9ef8-65240f20246a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-857d768544-bvllr\" (UID: \"763222b7-e3e3-4ff4-9ef8-65240f20246a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-857d768544-bvllr" Sep 10 18:32:53.136691 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:53.136621 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/763222b7-e3e3-4ff4-9ef8-65240f20246a-ca\") pod \"cluster-proxy-proxy-agent-857d768544-bvllr\" (UID: \"763222b7-e3e3-4ff4-9ef8-65240f20246a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-857d768544-bvllr" Sep 10 18:32:53.137302 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:53.137256 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce1097c4-8d40-4aa3-99b9-dc0125515b9b-tmp\") pod \"klusterlet-addon-workmgr-5fff5d7858-jxwdj\" (UID: \"ce1097c4-8d40-4aa3-99b9-dc0125515b9b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fff5d7858-jxwdj" Sep 10 18:32:53.137432 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:53.137399 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/763222b7-e3e3-4ff4-9ef8-65240f20246a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-857d768544-bvllr\" (UID: \"763222b7-e3e3-4ff4-9ef8-65240f20246a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-857d768544-bvllr" Sep 10 18:32:53.139146 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:53.139115 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/763222b7-e3e3-4ff4-9ef8-65240f20246a-ca\") pod \"cluster-proxy-proxy-agent-857d768544-bvllr\" (UID: \"763222b7-e3e3-4ff4-9ef8-65240f20246a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-857d768544-bvllr" Sep 10 18:32:53.139230 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:53.139163 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ce1097c4-8d40-4aa3-99b9-dc0125515b9b-klusterlet-config\") pod \"klusterlet-addon-workmgr-5fff5d7858-jxwdj\" (UID: \"ce1097c4-8d40-4aa3-99b9-dc0125515b9b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fff5d7858-jxwdj" Sep 10 18:32:53.139281 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:53.139264 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/763222b7-e3e3-4ff4-9ef8-65240f20246a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-857d768544-bvllr\" (UID: \"763222b7-e3e3-4ff4-9ef8-65240f20246a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-857d768544-bvllr" Sep 10 18:32:53.139534 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:53.139518 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/763222b7-e3e3-4ff4-9ef8-65240f20246a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-857d768544-bvllr\" (UID: \"763222b7-e3e3-4ff4-9ef8-65240f20246a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-857d768544-bvllr" Sep 10 18:32:53.139592 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:53.139573 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/763222b7-e3e3-4ff4-9ef8-65240f20246a-hub\") pod \"cluster-proxy-proxy-agent-857d768544-bvllr\" (UID: \"763222b7-e3e3-4ff4-9ef8-65240f20246a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-857d768544-bvllr" Sep 10 18:32:53.143406 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:53.143381 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntzdf\" (UniqueName: \"kubernetes.io/projected/ce1097c4-8d40-4aa3-99b9-dc0125515b9b-kube-api-access-ntzdf\") pod \"klusterlet-addon-workmgr-5fff5d7858-jxwdj\" (UID: \"ce1097c4-8d40-4aa3-99b9-dc0125515b9b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fff5d7858-jxwdj" Sep 10 18:32:53.143757 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:53.143738 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mfn4\" (UniqueName: \"kubernetes.io/projected/763222b7-e3e3-4ff4-9ef8-65240f20246a-kube-api-access-6mfn4\") pod \"cluster-proxy-proxy-agent-857d768544-bvllr\" (UID: \"763222b7-e3e3-4ff4-9ef8-65240f20246a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-857d768544-bvllr" Sep 10 18:32:53.263697 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:53.263641 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fff5d7858-jxwdj" Sep 10 18:32:53.303917 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:53.303884 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-857d768544-bvllr" Sep 10 18:32:53.530829 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:53.530727 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fff5d7858-jxwdj"] Sep 10 18:32:53.534366 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:32:53.534335 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce1097c4_8d40_4aa3_99b9_dc0125515b9b.slice/crio-419d4c63fbc64c0ed7f46c0ebccc312a670a497225bff17fa969fe3b48569b00 WatchSource:0}: Error finding container 419d4c63fbc64c0ed7f46c0ebccc312a670a497225bff17fa969fe3b48569b00: Status 404 returned error can't find the container with id 419d4c63fbc64c0ed7f46c0ebccc312a670a497225bff17fa969fe3b48569b00 Sep 10 18:32:53.574100 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:53.574080 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-857d768544-bvllr"] Sep 10 18:32:53.576826 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:32:53.576806 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod763222b7_e3e3_4ff4_9ef8_65240f20246a.slice/crio-55d438cb50ddcc536f2d6ce3722fa7f6884614452e196b0fa6e700fab4d313bd WatchSource:0}: Error finding container 55d438cb50ddcc536f2d6ce3722fa7f6884614452e196b0fa6e700fab4d313bd: Status 404 returned error can't find the container with id 55d438cb50ddcc536f2d6ce3722fa7f6884614452e196b0fa6e700fab4d313bd Sep 10 18:32:53.703953 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:53.703923 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-857d768544-bvllr" event={"ID":"763222b7-e3e3-4ff4-9ef8-65240f20246a","Type":"ContainerStarted","Data":"55d438cb50ddcc536f2d6ce3722fa7f6884614452e196b0fa6e700fab4d313bd"} Sep 10 18:32:53.704970 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:53.704946 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fff5d7858-jxwdj" event={"ID":"ce1097c4-8d40-4aa3-99b9-dc0125515b9b","Type":"ContainerStarted","Data":"419d4c63fbc64c0ed7f46c0ebccc312a670a497225bff17fa969fe3b48569b00"} Sep 10 18:32:56.641975 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:32:56.641941 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cznnw" Sep 10 18:33:00.728280 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:33:00.728242 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-857d768544-bvllr" event={"ID":"763222b7-e3e3-4ff4-9ef8-65240f20246a","Type":"ContainerStarted","Data":"f4cbecaa469fb986e57bcc362a30cd817132ed1a042abc7e2bc113dce85a6b9e"} Sep 10 18:33:00.728743 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:33:00.728289 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-857d768544-bvllr" event={"ID":"763222b7-e3e3-4ff4-9ef8-65240f20246a","Type":"ContainerStarted","Data":"75daebb490401f187fca45179a3248f9bd5a277053927f956c47587cde3f3126"} Sep 10 18:33:00.728743 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:33:00.728309 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-857d768544-bvllr" event={"ID":"763222b7-e3e3-4ff4-9ef8-65240f20246a","Type":"ContainerStarted","Data":"a15faf8acaed99fac80adf606571c528403471f45b6c2e8db99e70297a37e1a0"} Sep 10 18:33:00.729545 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:33:00.729526 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fff5d7858-jxwdj" event={"ID":"ce1097c4-8d40-4aa3-99b9-dc0125515b9b","Type":"ContainerStarted","Data":"6f6e1b39fc6d169106afb3b5393782c67a92aaa403eea9e9e868e8ab16a78b12"} Sep 10 18:33:00.729751 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:33:00.729732 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fff5d7858-jxwdj" Sep 10 18:33:00.731354 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:33:00.731312 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fff5d7858-jxwdj" Sep 10 18:33:00.744404 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:33:00.744366 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-857d768544-bvllr" podStartSLOduration=2.584941245 podStartE2EDuration="8.744354998s" totalImagesPullingTime="6.159413753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:32:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:33:00.743064496 +0000 UTC m=+106.829401970" watchObservedRunningTime="2026-09-10 18:33:00.744354998 +0000 UTC m=+106.830692476" Sep 10 18:33:00.755805 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:33:00.755766 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fff5d7858-jxwdj" podStartSLOduration=2.539649497 podStartE2EDuration="8.755753497s" totalImagesPullingTime="6.216104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:32:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:33:00.755463006 +0000 UTC m=+106.841800515" watchObservedRunningTime="2026-09-10 18:33:00.755753497 +0000 UTC m=+106.842090972" Sep 10 18:33:24.243597 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:33:24.243548 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54ef3c93-1624-46e6-ba09-4fd3bb1fc07f-metrics-certs\") pod \"network-metrics-daemon-fqmzj\" (UID: \"54ef3c93-1624-46e6-ba09-4fd3bb1fc07f\") " pod="openshift-multus/network-metrics-daemon-fqmzj" Sep 10 18:33:24.244059 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:33:24.243694 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 10 18:33:24.244059 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:33:24.243788 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54ef3c93-1624-46e6-ba09-4fd3bb1fc07f-metrics-certs podName:54ef3c93-1624-46e6-ba09-4fd3bb1fc07f nodeName:}" failed. No retries permitted until 2026-09-10 18:35:26.243748356 +0000 UTC m=+252.330085853 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54ef3c93-1624-46e6-ba09-4fd3bb1fc07f-metrics-certs") pod "network-metrics-daemon-fqmzj" (UID: "54ef3c93-1624-46e6-ba09-4fd3bb1fc07f") : secret "metrics-daemon-secret" not found Sep 10 18:33:50.786336 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:33:50.786284 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-4t7vn" podUID="2c41b6ab-b71c-4b51-9cf7-4deba7a3a113" Sep 10 18:33:50.871687 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:33:50.871654 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4t7vn" Sep 10 18:33:52.384638 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:33:52.384603 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-fqmzj" podUID="54ef3c93-1624-46e6-ba09-4fd3bb1fc07f" Sep 10 18:33:55.748046 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:33:55.748006 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-metrics-tls\") pod \"dns-default-4t7vn\" (UID: \"2c41b6ab-b71c-4b51-9cf7-4deba7a3a113\") " pod="openshift-dns/dns-default-4t7vn" Sep 10 18:33:55.748459 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:33:55.748156 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 18:33:55.748459 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:33:55.748222 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-metrics-tls podName:2c41b6ab-b71c-4b51-9cf7-4deba7a3a113 nodeName:}" failed. No retries permitted until 2026-09-10 18:35:57.748206108 +0000 UTC m=+283.834543562 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-metrics-tls") pod "dns-default-4t7vn" (UID: "2c41b6ab-b71c-4b51-9cf7-4deba7a3a113") : secret "dns-default-metrics-tls" not found Sep 10 18:34:05.374697 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:34:05.374660 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmzj" Sep 10 18:35:00.071412 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:00.071375 2581 generic.go:356] "Generic (PLEG): container finished" podID="ce1097c4-8d40-4aa3-99b9-dc0125515b9b" containerID="6f6e1b39fc6d169106afb3b5393782c67a92aaa403eea9e9e868e8ab16a78b12" exitCode=1 Sep 10 18:35:00.071892 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:00.071418 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fff5d7858-jxwdj" event={"ID":"ce1097c4-8d40-4aa3-99b9-dc0125515b9b","Type":"ContainerDied","Data":"6f6e1b39fc6d169106afb3b5393782c67a92aaa403eea9e9e868e8ab16a78b12"} Sep 10 18:35:00.071892 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:00.071686 2581 scope.go:117] "RemoveContainer" containerID="6f6e1b39fc6d169106afb3b5393782c67a92aaa403eea9e9e868e8ab16a78b12" Sep 10 18:35:00.729889 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:00.729859 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fff5d7858-jxwdj" Sep 10 18:35:01.075705 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:01.075672 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fff5d7858-jxwdj" event={"ID":"ce1097c4-8d40-4aa3-99b9-dc0125515b9b","Type":"ContainerStarted","Data":"fda9e8d8f8aa5aeb0c155d693d6d4da9300c0f7e2690d0fc8a3cd020f289f4e1"} Sep 10 18:35:01.076062 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:01.075910 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fff5d7858-jxwdj" Sep 10 18:35:01.076526 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:01.076507 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fff5d7858-jxwdj" Sep 10 18:35:08.703659 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:08.703620 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:08.788598 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:08.788569 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:11.235855 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:11.235820 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:11.283049 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:11.283019 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:12.763545 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:12.763517 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-chkc2"] Sep 10 18:35:12.768011 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:12.767988 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-chkc2" Sep 10 18:35:12.768939 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:12.768901 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:12.769812 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:12.769787 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 10 18:35:12.770472 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:12.770455 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 10 18:35:12.770472 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:12.770463 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-n67nc\"" Sep 10 18:35:12.770592 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:12.770483 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 10 18:35:12.770592 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:12.770505 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 10 18:35:12.777231 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:12.777212 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-chkc2"] Sep 10 18:35:12.832261 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:12.832231 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:12.854411 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:12.854389 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/34adc247-d318-44f5-8588-0dba849035e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-chkc2\" (UID: \"34adc247-d318-44f5-8588-0dba849035e6\") " pod="openshift-insights/insights-runtime-extractor-chkc2" Sep 10 18:35:12.854481 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:12.854434 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/34adc247-d318-44f5-8588-0dba849035e6-crio-socket\") pod \"insights-runtime-extractor-chkc2\" (UID: \"34adc247-d318-44f5-8588-0dba849035e6\") " pod="openshift-insights/insights-runtime-extractor-chkc2" Sep 10 18:35:12.854481 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:12.854462 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqhnn\" (UniqueName: \"kubernetes.io/projected/34adc247-d318-44f5-8588-0dba849035e6-kube-api-access-gqhnn\") pod \"insights-runtime-extractor-chkc2\" (UID: \"34adc247-d318-44f5-8588-0dba849035e6\") " pod="openshift-insights/insights-runtime-extractor-chkc2" Sep 10 18:35:12.854562 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:12.854487 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/34adc247-d318-44f5-8588-0dba849035e6-data-volume\") pod \"insights-runtime-extractor-chkc2\" (UID: \"34adc247-d318-44f5-8588-0dba849035e6\") " pod="openshift-insights/insights-runtime-extractor-chkc2" Sep 10 18:35:12.854562 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:12.854505 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/34adc247-d318-44f5-8588-0dba849035e6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-chkc2\" (UID: \"34adc247-d318-44f5-8588-0dba849035e6\") " pod="openshift-insights/insights-runtime-extractor-chkc2" Sep 10 18:35:12.954737 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:12.954714 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/34adc247-d318-44f5-8588-0dba849035e6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-chkc2\" (UID: \"34adc247-d318-44f5-8588-0dba849035e6\") " pod="openshift-insights/insights-runtime-extractor-chkc2" Sep 10 18:35:12.954809 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:12.954748 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/34adc247-d318-44f5-8588-0dba849035e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-chkc2\" (UID: \"34adc247-d318-44f5-8588-0dba849035e6\") " pod="openshift-insights/insights-runtime-extractor-chkc2" Sep 10 18:35:12.954809 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:12.954797 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/34adc247-d318-44f5-8588-0dba849035e6-crio-socket\") pod \"insights-runtime-extractor-chkc2\" (UID: \"34adc247-d318-44f5-8588-0dba849035e6\") " pod="openshift-insights/insights-runtime-extractor-chkc2" Sep 10 18:35:12.954945 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:35:12.954827 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 18:35:12.954945 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:12.954836 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqhnn\" (UniqueName: \"kubernetes.io/projected/34adc247-d318-44f5-8588-0dba849035e6-kube-api-access-gqhnn\") pod \"insights-runtime-extractor-chkc2\" (UID: \"34adc247-d318-44f5-8588-0dba849035e6\") " pod="openshift-insights/insights-runtime-extractor-chkc2" Sep 10 18:35:12.954945 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:12.954875 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/34adc247-d318-44f5-8588-0dba849035e6-data-volume\") pod \"insights-runtime-extractor-chkc2\" (UID: \"34adc247-d318-44f5-8588-0dba849035e6\") " pod="openshift-insights/insights-runtime-extractor-chkc2" Sep 10 18:35:12.954945 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:35:12.954899 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/34adc247-d318-44f5-8588-0dba849035e6-kube-rbac-proxy-cm podName:34adc247-d318-44f5-8588-0dba849035e6 nodeName:}" failed. No retries permitted until 2026-09-10 18:35:13.454880731 +0000 UTC m=+239.541218185 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/34adc247-d318-44f5-8588-0dba849035e6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-chkc2" (UID: "34adc247-d318-44f5-8588-0dba849035e6") : configmap "kube-rbac-proxy" not found Sep 10 18:35:12.954945 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:35:12.954926 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 18:35:12.954945 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:12.954930 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/34adc247-d318-44f5-8588-0dba849035e6-crio-socket\") pod \"insights-runtime-extractor-chkc2\" (UID: \"34adc247-d318-44f5-8588-0dba849035e6\") " pod="openshift-insights/insights-runtime-extractor-chkc2" Sep 10 18:35:12.955196 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:35:12.954981 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34adc247-d318-44f5-8588-0dba849035e6-insights-runtime-extractor-tls podName:34adc247-d318-44f5-8588-0dba849035e6 nodeName:}" failed. No retries permitted until 2026-09-10 18:35:13.454966909 +0000 UTC m=+239.541304362 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/34adc247-d318-44f5-8588-0dba849035e6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-chkc2" (UID: "34adc247-d318-44f5-8588-0dba849035e6") : secret "insights-runtime-extractor-tls" not found Sep 10 18:35:12.955196 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:12.955131 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/34adc247-d318-44f5-8588-0dba849035e6-data-volume\") pod \"insights-runtime-extractor-chkc2\" (UID: \"34adc247-d318-44f5-8588-0dba849035e6\") " pod="openshift-insights/insights-runtime-extractor-chkc2" Sep 10 18:35:12.962052 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:12.962034 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqhnn\" (UniqueName: \"kubernetes.io/projected/34adc247-d318-44f5-8588-0dba849035e6-kube-api-access-gqhnn\") pod \"insights-runtime-extractor-chkc2\" (UID: \"34adc247-d318-44f5-8588-0dba849035e6\") " pod="openshift-insights/insights-runtime-extractor-chkc2" Sep 10 18:35:13.457412 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:13.457384 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/34adc247-d318-44f5-8588-0dba849035e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-chkc2\" (UID: \"34adc247-d318-44f5-8588-0dba849035e6\") " pod="openshift-insights/insights-runtime-extractor-chkc2" Sep 10 18:35:13.457568 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:13.457453 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/34adc247-d318-44f5-8588-0dba849035e6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-chkc2\" (UID: \"34adc247-d318-44f5-8588-0dba849035e6\") " pod="openshift-insights/insights-runtime-extractor-chkc2" Sep 10 18:35:13.457568 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:35:13.457515 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 18:35:13.457568 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:35:13.457517 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 18:35:13.457666 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:35:13.457577 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34adc247-d318-44f5-8588-0dba849035e6-insights-runtime-extractor-tls podName:34adc247-d318-44f5-8588-0dba849035e6 nodeName:}" failed. No retries permitted until 2026-09-10 18:35:14.457562876 +0000 UTC m=+240.543900329 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/34adc247-d318-44f5-8588-0dba849035e6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-chkc2" (UID: "34adc247-d318-44f5-8588-0dba849035e6") : secret "insights-runtime-extractor-tls" not found Sep 10 18:35:13.457666 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:35:13.457590 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/34adc247-d318-44f5-8588-0dba849035e6-kube-rbac-proxy-cm podName:34adc247-d318-44f5-8588-0dba849035e6 nodeName:}" failed. No retries permitted until 2026-09-10 18:35:14.457583364 +0000 UTC m=+240.543920818 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/34adc247-d318-44f5-8588-0dba849035e6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-chkc2" (UID: "34adc247-d318-44f5-8588-0dba849035e6") : configmap "kube-rbac-proxy" not found Sep 10 18:35:13.807212 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:13.807190 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:13.900658 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:13.900632 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:14.001402 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:14.001378 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:14.390486 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:14.390450 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:14.463358 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:14.463318 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/34adc247-d318-44f5-8588-0dba849035e6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-chkc2\" (UID: \"34adc247-d318-44f5-8588-0dba849035e6\") " pod="openshift-insights/insights-runtime-extractor-chkc2" Sep 10 18:35:14.463480 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:14.463396 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/34adc247-d318-44f5-8588-0dba849035e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-chkc2\" (UID: \"34adc247-d318-44f5-8588-0dba849035e6\") " pod="openshift-insights/insights-runtime-extractor-chkc2" Sep 10 18:35:14.463480 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:35:14.463447 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 18:35:14.463551 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:35:14.463503 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/34adc247-d318-44f5-8588-0dba849035e6-kube-rbac-proxy-cm podName:34adc247-d318-44f5-8588-0dba849035e6 nodeName:}" failed. No retries permitted until 2026-09-10 18:35:16.46348634 +0000 UTC m=+242.549823813 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/34adc247-d318-44f5-8588-0dba849035e6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-chkc2" (UID: "34adc247-d318-44f5-8588-0dba849035e6") : configmap "kube-rbac-proxy" not found Sep 10 18:35:14.463599 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:35:14.463550 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 18:35:14.463631 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:35:14.463606 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34adc247-d318-44f5-8588-0dba849035e6-insights-runtime-extractor-tls podName:34adc247-d318-44f5-8588-0dba849035e6 nodeName:}" failed. No retries permitted until 2026-09-10 18:35:16.463592606 +0000 UTC m=+242.549930059 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/34adc247-d318-44f5-8588-0dba849035e6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-chkc2" (UID: "34adc247-d318-44f5-8588-0dba849035e6") : secret "insights-runtime-extractor-tls" not found Sep 10 18:35:14.815026 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:14.814993 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:14.864453 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:14.864425 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:14.882991 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:14.882974 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gf28x_4d70e39b-2b94-4593-828b-75d5d071560e/dns-node-resolver/0.log" Sep 10 18:35:15.095113 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:15.095052 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:15.190577 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:15.190555 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:15.885536 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:15.885513 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lh8gr_dc8250af-b1fc-4108-b6fd-092198a098ac/node-ca/0.log" Sep 10 18:35:16.268924 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:16.268853 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:16.369172 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:16.369147 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:16.476514 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:16.476485 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/34adc247-d318-44f5-8588-0dba849035e6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-chkc2\" (UID: \"34adc247-d318-44f5-8588-0dba849035e6\") " pod="openshift-insights/insights-runtime-extractor-chkc2" Sep 10 18:35:16.476652 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:16.476520 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/34adc247-d318-44f5-8588-0dba849035e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-chkc2\" (UID: \"34adc247-d318-44f5-8588-0dba849035e6\") " pod="openshift-insights/insights-runtime-extractor-chkc2" Sep 10 18:35:16.476652 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:35:16.476617 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 18:35:16.476652 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:35:16.476642 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 18:35:16.476815 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:35:16.476690 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/34adc247-d318-44f5-8588-0dba849035e6-kube-rbac-proxy-cm podName:34adc247-d318-44f5-8588-0dba849035e6 nodeName:}" failed. No retries permitted until 2026-09-10 18:35:20.476671331 +0000 UTC m=+246.563008805 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/34adc247-d318-44f5-8588-0dba849035e6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-chkc2" (UID: "34adc247-d318-44f5-8588-0dba849035e6") : configmap "kube-rbac-proxy" not found Sep 10 18:35:16.476815 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:35:16.476710 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34adc247-d318-44f5-8588-0dba849035e6-insights-runtime-extractor-tls podName:34adc247-d318-44f5-8588-0dba849035e6 nodeName:}" failed. No retries permitted until 2026-09-10 18:35:20.476700121 +0000 UTC m=+246.563037581 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/34adc247-d318-44f5-8588-0dba849035e6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-chkc2" (UID: "34adc247-d318-44f5-8588-0dba849035e6") : secret "insights-runtime-extractor-tls" not found Sep 10 18:35:16.483517 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:16.483497 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-drn5f_7470a24c-2464-4678-9015-4689638c6b57/serve-healthcheck-canary/0.log" Sep 10 18:35:16.724065 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:16.724033 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:16.774995 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:16.774966 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:17.102490 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:17.102467 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:17.155566 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:17.155540 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:17.605556 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:17.605522 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:20.503832 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:20.503789 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/34adc247-d318-44f5-8588-0dba849035e6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-chkc2\" (UID: \"34adc247-d318-44f5-8588-0dba849035e6\") " pod="openshift-insights/insights-runtime-extractor-chkc2" Sep 10 18:35:20.503832 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:20.503829 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/34adc247-d318-44f5-8588-0dba849035e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-chkc2\" (UID: \"34adc247-d318-44f5-8588-0dba849035e6\") " pod="openshift-insights/insights-runtime-extractor-chkc2" Sep 10 18:35:20.504475 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:35:20.503896 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 18:35:20.504475 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:35:20.503941 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 18:35:20.504475 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:35:20.503976 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/34adc247-d318-44f5-8588-0dba849035e6-kube-rbac-proxy-cm podName:34adc247-d318-44f5-8588-0dba849035e6 nodeName:}" failed. No retries permitted until 2026-09-10 18:35:28.503956368 +0000 UTC m=+254.590293873 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/34adc247-d318-44f5-8588-0dba849035e6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-chkc2" (UID: "34adc247-d318-44f5-8588-0dba849035e6") : configmap "kube-rbac-proxy" not found Sep 10 18:35:20.504475 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:35:20.503994 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34adc247-d318-44f5-8588-0dba849035e6-insights-runtime-extractor-tls podName:34adc247-d318-44f5-8588-0dba849035e6 nodeName:}" failed. No retries permitted until 2026-09-10 18:35:28.503986996 +0000 UTC m=+254.590324449 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/34adc247-d318-44f5-8588-0dba849035e6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-chkc2" (UID: "34adc247-d318-44f5-8588-0dba849035e6") : secret "insights-runtime-extractor-tls" not found Sep 10 18:35:20.511971 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:20.511931 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:20.611278 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:20.611247 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:20.711070 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:20.711040 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:20.811932 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:20.811909 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:20.912895 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:20.912868 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:21.019188 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:21.019166 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:21.213125 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:21.213066 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:22.411002 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:22.410966 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:22.618116 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:22.618082 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:22.820460 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:22.820429 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:23.009135 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:23.009105 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:23.217191 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:23.217125 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:24.009427 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:24.009383 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:24.411753 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:24.411717 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:24.613070 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:24.613042 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:24.713368 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:24.713287 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:24.813709 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:24.813689 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:25.209877 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:25.209848 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:25.407643 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:25.407611 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:25.811044 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:25.811015 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:25.926213 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:25.926176 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:26.340983 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:26.340961 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54ef3c93-1624-46e6-ba09-4fd3bb1fc07f-metrics-certs\") pod \"network-metrics-daemon-fqmzj\" (UID: \"54ef3c93-1624-46e6-ba09-4fd3bb1fc07f\") " pod="openshift-multus/network-metrics-daemon-fqmzj" Sep 10 18:35:26.341371 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:35:26.341080 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 10 18:35:26.341371 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:35:26.341134 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54ef3c93-1624-46e6-ba09-4fd3bb1fc07f-metrics-certs podName:54ef3c93-1624-46e6-ba09-4fd3bb1fc07f nodeName:}" failed. No retries permitted until 2026-09-10 18:37:28.341119617 +0000 UTC m=+374.427457070 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54ef3c93-1624-46e6-ba09-4fd3bb1fc07f-metrics-certs") pod "network-metrics-daemon-fqmzj" (UID: "54ef3c93-1624-46e6-ba09-4fd3bb1fc07f") : secret "metrics-daemon-secret" not found Sep 10 18:35:26.626105 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:26.626028 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:26.999389 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:26.999301 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:27.099717 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:27.099685 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:27.209028 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:27.209002 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:27.413250 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:27.413222 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:27.613183 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:27.613159 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:28.554161 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:28.554127 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/34adc247-d318-44f5-8588-0dba849035e6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-chkc2\" (UID: \"34adc247-d318-44f5-8588-0dba849035e6\") " pod="openshift-insights/insights-runtime-extractor-chkc2" Sep 10 18:35:28.554161 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:28.554162 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/34adc247-d318-44f5-8588-0dba849035e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-chkc2\" (UID: \"34adc247-d318-44f5-8588-0dba849035e6\") " pod="openshift-insights/insights-runtime-extractor-chkc2" Sep 10 18:35:28.554650 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:35:28.554304 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 18:35:28.554650 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:35:28.554391 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34adc247-d318-44f5-8588-0dba849035e6-insights-runtime-extractor-tls podName:34adc247-d318-44f5-8588-0dba849035e6 nodeName:}" failed. No retries permitted until 2026-09-10 18:35:44.554371902 +0000 UTC m=+270.640709356 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/34adc247-d318-44f5-8588-0dba849035e6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-chkc2" (UID: "34adc247-d318-44f5-8588-0dba849035e6") : secret "insights-runtime-extractor-tls" not found Sep 10 18:35:28.554753 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:28.554700 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/34adc247-d318-44f5-8588-0dba849035e6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-chkc2\" (UID: \"34adc247-d318-44f5-8588-0dba849035e6\") " pod="openshift-insights/insights-runtime-extractor-chkc2" Sep 10 18:35:28.615014 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:28.614981 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:28.982373 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:28.982281 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:29.083453 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:29.083423 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:29.212303 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:29.212274 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:29.416866 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:29.416837 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:29.824301 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:29.824271 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:30.409027 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:30.408996 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:30.615806 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:30.615779 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:31.008219 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:31.008184 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:31.212872 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:31.212836 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:35.079157 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:35.079126 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5678c4d879-pgdx7"] Sep 10 18:35:35.082106 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:35.082091 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" Sep 10 18:35:35.084125 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:35.084094 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 10 18:35:35.084125 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:35.084114 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 10 18:35:35.084291 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:35.084127 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xgqmb\"" Sep 10 18:35:35.084291 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:35.084215 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 10 18:35:35.089372 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:35.089354 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 10 18:35:35.093836 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:35.093814 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5678c4d879-pgdx7"] Sep 10 18:35:35.128535 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:35.128510 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:35.198442 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:35.198420 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b082e08-bb40-4b42-aff0-ffc0ccbf2f02-trusted-ca\") pod \"image-registry-5678c4d879-pgdx7\" (UID: \"6b082e08-bb40-4b42-aff0-ffc0ccbf2f02\") " pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" Sep 10 18:35:35.198535 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:35.198446 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b082e08-bb40-4b42-aff0-ffc0ccbf2f02-registry-certificates\") pod \"image-registry-5678c4d879-pgdx7\" (UID: \"6b082e08-bb40-4b42-aff0-ffc0ccbf2f02\") " pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" Sep 10 18:35:35.198535 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:35.198472 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b082e08-bb40-4b42-aff0-ffc0ccbf2f02-image-registry-private-configuration\") pod \"image-registry-5678c4d879-pgdx7\" (UID: \"6b082e08-bb40-4b42-aff0-ffc0ccbf2f02\") " pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" Sep 10 18:35:35.198535 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:35.198490 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b082e08-bb40-4b42-aff0-ffc0ccbf2f02-ca-trust-extracted\") pod \"image-registry-5678c4d879-pgdx7\" (UID: \"6b082e08-bb40-4b42-aff0-ffc0ccbf2f02\") " pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" Sep 10 18:35:35.198661 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:35.198551 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b082e08-bb40-4b42-aff0-ffc0ccbf2f02-installation-pull-secrets\") pod \"image-registry-5678c4d879-pgdx7\" (UID: \"6b082e08-bb40-4b42-aff0-ffc0ccbf2f02\") " pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" Sep 10 18:35:35.198661 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:35.198579 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b082e08-bb40-4b42-aff0-ffc0ccbf2f02-registry-tls\") pod \"image-registry-5678c4d879-pgdx7\" (UID: \"6b082e08-bb40-4b42-aff0-ffc0ccbf2f02\") " pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" Sep 10 18:35:35.198661 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:35.198631 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b082e08-bb40-4b42-aff0-ffc0ccbf2f02-bound-sa-token\") pod \"image-registry-5678c4d879-pgdx7\" (UID: \"6b082e08-bb40-4b42-aff0-ffc0ccbf2f02\") " pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" Sep 10 18:35:35.198661 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:35.198647 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25jnh\" (UniqueName: \"kubernetes.io/projected/6b082e08-bb40-4b42-aff0-ffc0ccbf2f02-kube-api-access-25jnh\") pod \"image-registry-5678c4d879-pgdx7\" (UID: \"6b082e08-bb40-4b42-aff0-ffc0ccbf2f02\") " pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" Sep 10 18:35:35.226926 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:35.226900 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:35.299719 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:35.299695 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b082e08-bb40-4b42-aff0-ffc0ccbf2f02-installation-pull-secrets\") pod \"image-registry-5678c4d879-pgdx7\" (UID: \"6b082e08-bb40-4b42-aff0-ffc0ccbf2f02\") " pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" Sep 10 18:35:35.299796 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:35.299726 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b082e08-bb40-4b42-aff0-ffc0ccbf2f02-registry-tls\") pod \"image-registry-5678c4d879-pgdx7\" (UID: \"6b082e08-bb40-4b42-aff0-ffc0ccbf2f02\") " pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" Sep 10 18:35:35.299796 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:35.299768 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b082e08-bb40-4b42-aff0-ffc0ccbf2f02-bound-sa-token\") pod \"image-registry-5678c4d879-pgdx7\" (UID: \"6b082e08-bb40-4b42-aff0-ffc0ccbf2f02\") " pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" Sep 10 18:35:35.299796 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:35.299792 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25jnh\" (UniqueName: \"kubernetes.io/projected/6b082e08-bb40-4b42-aff0-ffc0ccbf2f02-kube-api-access-25jnh\") pod \"image-registry-5678c4d879-pgdx7\" (UID: \"6b082e08-bb40-4b42-aff0-ffc0ccbf2f02\") " pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" Sep 10 18:35:35.299950 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:35.299827 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b082e08-bb40-4b42-aff0-ffc0ccbf2f02-trusted-ca\") pod \"image-registry-5678c4d879-pgdx7\" (UID: \"6b082e08-bb40-4b42-aff0-ffc0ccbf2f02\") " pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" Sep 10 18:35:35.299950 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:35.299853 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b082e08-bb40-4b42-aff0-ffc0ccbf2f02-registry-certificates\") pod \"image-registry-5678c4d879-pgdx7\" (UID: \"6b082e08-bb40-4b42-aff0-ffc0ccbf2f02\") " pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" Sep 10 18:35:35.299950 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:35.299912 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b082e08-bb40-4b42-aff0-ffc0ccbf2f02-image-registry-private-configuration\") pod \"image-registry-5678c4d879-pgdx7\" (UID: \"6b082e08-bb40-4b42-aff0-ffc0ccbf2f02\") " pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" Sep 10 18:35:35.299950 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:35.299932 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b082e08-bb40-4b42-aff0-ffc0ccbf2f02-ca-trust-extracted\") pod \"image-registry-5678c4d879-pgdx7\" (UID: \"6b082e08-bb40-4b42-aff0-ffc0ccbf2f02\") " pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" Sep 10 18:35:35.300401 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:35.300381 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b082e08-bb40-4b42-aff0-ffc0ccbf2f02-ca-trust-extracted\") pod \"image-registry-5678c4d879-pgdx7\" (UID: \"6b082e08-bb40-4b42-aff0-ffc0ccbf2f02\") " pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" Sep 10 18:35:35.300784 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:35.300758 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b082e08-bb40-4b42-aff0-ffc0ccbf2f02-registry-certificates\") pod \"image-registry-5678c4d879-pgdx7\" (UID: \"6b082e08-bb40-4b42-aff0-ffc0ccbf2f02\") " pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" Sep 10 18:35:35.300867 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:35.300783 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b082e08-bb40-4b42-aff0-ffc0ccbf2f02-trusted-ca\") pod \"image-registry-5678c4d879-pgdx7\" (UID: \"6b082e08-bb40-4b42-aff0-ffc0ccbf2f02\") " pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" Sep 10 18:35:35.302245 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:35.302222 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b082e08-bb40-4b42-aff0-ffc0ccbf2f02-installation-pull-secrets\") pod \"image-registry-5678c4d879-pgdx7\" (UID: \"6b082e08-bb40-4b42-aff0-ffc0ccbf2f02\") " pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" Sep 10 18:35:35.302842 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:35.302826 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b082e08-bb40-4b42-aff0-ffc0ccbf2f02-registry-tls\") pod \"image-registry-5678c4d879-pgdx7\" (UID: \"6b082e08-bb40-4b42-aff0-ffc0ccbf2f02\") " pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" Sep 10 18:35:35.302944 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:35.302848 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b082e08-bb40-4b42-aff0-ffc0ccbf2f02-image-registry-private-configuration\") pod \"image-registry-5678c4d879-pgdx7\" (UID: \"6b082e08-bb40-4b42-aff0-ffc0ccbf2f02\") " pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" Sep 10 18:35:35.306201 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:35.306176 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b082e08-bb40-4b42-aff0-ffc0ccbf2f02-bound-sa-token\") pod \"image-registry-5678c4d879-pgdx7\" (UID: \"6b082e08-bb40-4b42-aff0-ffc0ccbf2f02\") " pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" Sep 10 18:35:35.306691 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:35.306672 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25jnh\" (UniqueName: \"kubernetes.io/projected/6b082e08-bb40-4b42-aff0-ffc0ccbf2f02-kube-api-access-25jnh\") pod \"image-registry-5678c4d879-pgdx7\" (UID: \"6b082e08-bb40-4b42-aff0-ffc0ccbf2f02\") " pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" Sep 10 18:35:35.391992 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:35.391942 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" Sep 10 18:35:35.662370 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:35.662254 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5678c4d879-pgdx7"] Sep 10 18:35:35.666262 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:35:35.666231 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b082e08_bb40_4b42_aff0_ffc0ccbf2f02.slice/crio-ed7bb81245d2374e760446170a0850ad3f67a00052addb63f7c26f3c87b1ff0f WatchSource:0}: Error finding container ed7bb81245d2374e760446170a0850ad3f67a00052addb63f7c26f3c87b1ff0f: Status 404 returned error can't find the container with id ed7bb81245d2374e760446170a0850ad3f67a00052addb63f7c26f3c87b1ff0f Sep 10 18:35:36.179785 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:36.179747 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" event={"ID":"6b082e08-bb40-4b42-aff0-ffc0ccbf2f02","Type":"ContainerStarted","Data":"0405b7b305682a94c9333ed12414f06117a0bda20ccea8ff384794b7c412f910"} Sep 10 18:35:36.180210 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:36.179823 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" event={"ID":"6b082e08-bb40-4b42-aff0-ffc0ccbf2f02","Type":"ContainerStarted","Data":"ed7bb81245d2374e760446170a0850ad3f67a00052addb63f7c26f3c87b1ff0f"} Sep 10 18:35:36.180210 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:36.179879 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" Sep 10 18:35:36.195719 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:36.195674 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" podStartSLOduration=1.19566223 podStartE2EDuration="1.19566223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:35:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:35:36.194347536 +0000 UTC m=+262.280685007" watchObservedRunningTime="2026-09-10 18:35:36.19566223 +0000 UTC m=+262.281999705" Sep 10 18:35:37.281227 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:37.281183 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:37.328132 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:37.328102 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:43.278750 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:43.278707 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:43.377585 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:43.377557 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:43.446782 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:43.446751 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:44.136967 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:44.136925 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:44.560197 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:44.560162 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/34adc247-d318-44f5-8588-0dba849035e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-chkc2\" (UID: \"34adc247-d318-44f5-8588-0dba849035e6\") " pod="openshift-insights/insights-runtime-extractor-chkc2" Sep 10 18:35:44.562531 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:44.562512 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/34adc247-d318-44f5-8588-0dba849035e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-chkc2\" (UID: \"34adc247-d318-44f5-8588-0dba849035e6\") " pod="openshift-insights/insights-runtime-extractor-chkc2" Sep 10 18:35:44.579482 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:44.579460 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-n67nc\"" Sep 10 18:35:44.587908 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:44.587896 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-chkc2" Sep 10 18:35:44.856877 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:44.856794 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-chkc2"] Sep 10 18:35:44.861107 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:35:44.861075 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34adc247_d318_44f5_8588_0dba849035e6.slice/crio-0b5989a141c6adf74ca8b697c68237ab696c22165630000fffd13333609f57b8 WatchSource:0}: Error finding container 0b5989a141c6adf74ca8b697c68237ab696c22165630000fffd13333609f57b8: Status 404 returned error can't find the container with id 0b5989a141c6adf74ca8b697c68237ab696c22165630000fffd13333609f57b8 Sep 10 18:35:45.205166 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:45.205080 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-chkc2" event={"ID":"34adc247-d318-44f5-8588-0dba849035e6","Type":"ContainerStarted","Data":"e8bf3a81183599871d81de5e6b6169308763571d91f5406d2617378c87ad3869"} Sep 10 18:35:45.205166 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:45.205147 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-chkc2" event={"ID":"34adc247-d318-44f5-8588-0dba849035e6","Type":"ContainerStarted","Data":"0b5989a141c6adf74ca8b697c68237ab696c22165630000fffd13333609f57b8"} Sep 10 18:35:46.209401 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:46.209314 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-chkc2" event={"ID":"34adc247-d318-44f5-8588-0dba849035e6","Type":"ContainerStarted","Data":"50c10c083cb11df45cc1e74252c4799be21b98f6e3fde13c84e0481feeb60199"} Sep 10 18:35:47.468569 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:47.468536 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:47.533936 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:47.533901 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:47.735892 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:47.735865 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:48.217243 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:48.217207 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-chkc2" event={"ID":"34adc247-d318-44f5-8588-0dba849035e6","Type":"ContainerStarted","Data":"d201ff5ae7200b7387c228aa13a6555c977a89b5bd3e841d15678fd587d80041"} Sep 10 18:35:48.233211 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:48.233166 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-chkc2" podStartSLOduration=33.487481196 podStartE2EDuration="36.233150238s" totalImagesPullingTime="2.745669042s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 18:35:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:35:48.231534928 +0000 UTC m=+274.317872402" watchObservedRunningTime="2026-09-10 18:35:48.233150238 +0000 UTC m=+274.319487712" Sep 10 18:35:50.538370 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:50.538319 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:52.605593 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:52.605559 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:52.703261 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:52.703222 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:52.818025 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:52.817998 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:52.886089 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:52.886017 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xzs2v"] Sep 10 18:35:52.889157 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:52.889143 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xzs2v" Sep 10 18:35:52.891281 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:52.891255 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 10 18:35:52.891417 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:52.891319 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 10 18:35:52.891417 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:52.891397 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 10 18:35:52.891578 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:52.891561 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rjgtz\"" Sep 10 18:35:52.892061 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:52.892043 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 10 18:35:52.892121 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:52.892047 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 10 18:35:52.892352 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:52.892337 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 10 18:35:52.905351 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:52.905307 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:53.016100 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.016068 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/591fba1e-a40f-41a3-8955-fd32464bf2ad-metrics-client-ca\") pod \"node-exporter-xzs2v\" (UID: \"591fba1e-a40f-41a3-8955-fd32464bf2ad\") " pod="openshift-monitoring/node-exporter-xzs2v" Sep 10 18:35:53.016270 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.016106 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/591fba1e-a40f-41a3-8955-fd32464bf2ad-root\") pod \"node-exporter-xzs2v\" (UID: \"591fba1e-a40f-41a3-8955-fd32464bf2ad\") " pod="openshift-monitoring/node-exporter-xzs2v" Sep 10 18:35:53.016270 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.016125 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/591fba1e-a40f-41a3-8955-fd32464bf2ad-node-exporter-wtmp\") pod \"node-exporter-xzs2v\" (UID: \"591fba1e-a40f-41a3-8955-fd32464bf2ad\") " pod="openshift-monitoring/node-exporter-xzs2v" Sep 10 18:35:53.016270 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.016163 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/591fba1e-a40f-41a3-8955-fd32464bf2ad-sys\") pod \"node-exporter-xzs2v\" (UID: \"591fba1e-a40f-41a3-8955-fd32464bf2ad\") " pod="openshift-monitoring/node-exporter-xzs2v" Sep 10 18:35:53.016270 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.016200 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/591fba1e-a40f-41a3-8955-fd32464bf2ad-node-exporter-textfile\") pod \"node-exporter-xzs2v\" (UID: \"591fba1e-a40f-41a3-8955-fd32464bf2ad\") " pod="openshift-monitoring/node-exporter-xzs2v" Sep 10 18:35:53.016459 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.016285 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsn7n\" (UniqueName: \"kubernetes.io/projected/591fba1e-a40f-41a3-8955-fd32464bf2ad-kube-api-access-tsn7n\") pod \"node-exporter-xzs2v\" (UID: \"591fba1e-a40f-41a3-8955-fd32464bf2ad\") " pod="openshift-monitoring/node-exporter-xzs2v" Sep 10 18:35:53.016459 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.016303 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/591fba1e-a40f-41a3-8955-fd32464bf2ad-node-exporter-tls\") pod \"node-exporter-xzs2v\" (UID: \"591fba1e-a40f-41a3-8955-fd32464bf2ad\") " pod="openshift-monitoring/node-exporter-xzs2v" Sep 10 18:35:53.016459 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.016345 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/591fba1e-a40f-41a3-8955-fd32464bf2ad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xzs2v\" (UID: \"591fba1e-a40f-41a3-8955-fd32464bf2ad\") " pod="openshift-monitoring/node-exporter-xzs2v" Sep 10 18:35:53.016459 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.016401 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/591fba1e-a40f-41a3-8955-fd32464bf2ad-node-exporter-accelerators-collector-config\") pod \"node-exporter-xzs2v\" (UID: \"591fba1e-a40f-41a3-8955-fd32464bf2ad\") " pod="openshift-monitoring/node-exporter-xzs2v" Sep 10 18:35:53.052273 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.052245 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:53.117623 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.117599 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsn7n\" (UniqueName: \"kubernetes.io/projected/591fba1e-a40f-41a3-8955-fd32464bf2ad-kube-api-access-tsn7n\") pod \"node-exporter-xzs2v\" (UID: \"591fba1e-a40f-41a3-8955-fd32464bf2ad\") " pod="openshift-monitoring/node-exporter-xzs2v" Sep 10 18:35:53.117720 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.117627 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/591fba1e-a40f-41a3-8955-fd32464bf2ad-node-exporter-tls\") pod \"node-exporter-xzs2v\" (UID: \"591fba1e-a40f-41a3-8955-fd32464bf2ad\") " pod="openshift-monitoring/node-exporter-xzs2v" Sep 10 18:35:53.117720 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.117653 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/591fba1e-a40f-41a3-8955-fd32464bf2ad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xzs2v\" (UID: \"591fba1e-a40f-41a3-8955-fd32464bf2ad\") " pod="openshift-monitoring/node-exporter-xzs2v" Sep 10 18:35:53.117720 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.117683 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/591fba1e-a40f-41a3-8955-fd32464bf2ad-node-exporter-accelerators-collector-config\") pod \"node-exporter-xzs2v\" (UID: \"591fba1e-a40f-41a3-8955-fd32464bf2ad\") " pod="openshift-monitoring/node-exporter-xzs2v" Sep 10 18:35:53.117720 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.117710 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/591fba1e-a40f-41a3-8955-fd32464bf2ad-metrics-client-ca\") pod \"node-exporter-xzs2v\" (UID: \"591fba1e-a40f-41a3-8955-fd32464bf2ad\") " pod="openshift-monitoring/node-exporter-xzs2v" Sep 10 18:35:53.117894 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.117732 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/591fba1e-a40f-41a3-8955-fd32464bf2ad-root\") pod \"node-exporter-xzs2v\" (UID: \"591fba1e-a40f-41a3-8955-fd32464bf2ad\") " pod="openshift-monitoring/node-exporter-xzs2v" Sep 10 18:35:53.117894 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.117757 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/591fba1e-a40f-41a3-8955-fd32464bf2ad-node-exporter-wtmp\") pod \"node-exporter-xzs2v\" (UID: \"591fba1e-a40f-41a3-8955-fd32464bf2ad\") " pod="openshift-monitoring/node-exporter-xzs2v" Sep 10 18:35:53.117894 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:35:53.117766 2581 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 10 18:35:53.117894 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:35:53.117828 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/591fba1e-a40f-41a3-8955-fd32464bf2ad-node-exporter-tls podName:591fba1e-a40f-41a3-8955-fd32464bf2ad nodeName:}" failed. No retries permitted until 2026-09-10 18:35:53.617809699 +0000 UTC m=+279.704147172 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/591fba1e-a40f-41a3-8955-fd32464bf2ad-node-exporter-tls") pod "node-exporter-xzs2v" (UID: "591fba1e-a40f-41a3-8955-fd32464bf2ad") : secret "node-exporter-tls" not found Sep 10 18:35:53.117894 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.117871 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/591fba1e-a40f-41a3-8955-fd32464bf2ad-sys\") pod \"node-exporter-xzs2v\" (UID: \"591fba1e-a40f-41a3-8955-fd32464bf2ad\") " pod="openshift-monitoring/node-exporter-xzs2v" Sep 10 18:35:53.117894 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.117892 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/591fba1e-a40f-41a3-8955-fd32464bf2ad-node-exporter-wtmp\") pod \"node-exporter-xzs2v\" (UID: \"591fba1e-a40f-41a3-8955-fd32464bf2ad\") " pod="openshift-monitoring/node-exporter-xzs2v" Sep 10 18:35:53.118200 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.117905 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/591fba1e-a40f-41a3-8955-fd32464bf2ad-node-exporter-textfile\") pod \"node-exporter-xzs2v\" (UID: \"591fba1e-a40f-41a3-8955-fd32464bf2ad\") " pod="openshift-monitoring/node-exporter-xzs2v" Sep 10 18:35:53.118200 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.117938 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/591fba1e-a40f-41a3-8955-fd32464bf2ad-sys\") pod \"node-exporter-xzs2v\" (UID: \"591fba1e-a40f-41a3-8955-fd32464bf2ad\") " pod="openshift-monitoring/node-exporter-xzs2v" Sep 10 18:35:53.118200 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.117826 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/591fba1e-a40f-41a3-8955-fd32464bf2ad-root\") pod \"node-exporter-xzs2v\" (UID: \"591fba1e-a40f-41a3-8955-fd32464bf2ad\") " pod="openshift-monitoring/node-exporter-xzs2v" Sep 10 18:35:53.118200 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.118186 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/591fba1e-a40f-41a3-8955-fd32464bf2ad-node-exporter-textfile\") pod \"node-exporter-xzs2v\" (UID: \"591fba1e-a40f-41a3-8955-fd32464bf2ad\") " pod="openshift-monitoring/node-exporter-xzs2v" Sep 10 18:35:53.118439 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.118294 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/591fba1e-a40f-41a3-8955-fd32464bf2ad-node-exporter-accelerators-collector-config\") pod \"node-exporter-xzs2v\" (UID: \"591fba1e-a40f-41a3-8955-fd32464bf2ad\") " pod="openshift-monitoring/node-exporter-xzs2v" Sep 10 18:35:53.118491 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.118441 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/591fba1e-a40f-41a3-8955-fd32464bf2ad-metrics-client-ca\") pod \"node-exporter-xzs2v\" (UID: \"591fba1e-a40f-41a3-8955-fd32464bf2ad\") " pod="openshift-monitoring/node-exporter-xzs2v" Sep 10 18:35:53.120006 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.119990 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/591fba1e-a40f-41a3-8955-fd32464bf2ad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xzs2v\" (UID: \"591fba1e-a40f-41a3-8955-fd32464bf2ad\") " pod="openshift-monitoring/node-exporter-xzs2v" Sep 10 18:35:53.125398 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.125377 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsn7n\" (UniqueName: \"kubernetes.io/projected/591fba1e-a40f-41a3-8955-fd32464bf2ad-kube-api-access-tsn7n\") pod \"node-exporter-xzs2v\" (UID: \"591fba1e-a40f-41a3-8955-fd32464bf2ad\") " pod="openshift-monitoring/node-exporter-xzs2v" Sep 10 18:35:53.152363 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.152294 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:53.260185 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.260158 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:53.305623 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.305584 2581 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-857d768544-bvllr" podUID="763222b7-e3e3-4ff4-9ef8-65240f20246a" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 10 18:35:53.316942 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.316922 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:53.417207 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.417145 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:53.517115 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.517090 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:53.568769 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.568747 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:53.622320 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.622300 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/591fba1e-a40f-41a3-8955-fd32464bf2ad-node-exporter-tls\") pod \"node-exporter-xzs2v\" (UID: \"591fba1e-a40f-41a3-8955-fd32464bf2ad\") " pod="openshift-monitoring/node-exporter-xzs2v" Sep 10 18:35:53.624658 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.624635 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/591fba1e-a40f-41a3-8955-fd32464bf2ad-node-exporter-tls\") pod \"node-exporter-xzs2v\" (UID: \"591fba1e-a40f-41a3-8955-fd32464bf2ad\") " pod="openshift-monitoring/node-exporter-xzs2v" Sep 10 18:35:53.643566 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.643544 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:53.797548 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.797530 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xzs2v" Sep 10 18:35:53.808302 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:35:53.808281 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod591fba1e_a40f_41a3_8955_fd32464bf2ad.slice/crio-e62290134f1a20194a3c808255b3138b3f6076db303e09a725fbd0e2069e02f2 WatchSource:0}: Error finding container e62290134f1a20194a3c808255b3138b3f6076db303e09a725fbd0e2069e02f2: Status 404 returned error can't find the container with id e62290134f1a20194a3c808255b3138b3f6076db303e09a725fbd0e2069e02f2 Sep 10 18:35:53.825822 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.825800 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:53.872868 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:35:53.872838 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-4t7vn" podUID="2c41b6ab-b71c-4b51-9cf7-4deba7a3a113" Sep 10 18:35:53.907474 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:53.907450 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:54.034095 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:54.034075 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:54.112220 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:54.112161 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:54.236383 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:54.236362 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4t7vn" Sep 10 18:35:54.236472 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:54.236358 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xzs2v" event={"ID":"591fba1e-a40f-41a3-8955-fd32464bf2ad","Type":"ContainerStarted","Data":"e62290134f1a20194a3c808255b3138b3f6076db303e09a725fbd0e2069e02f2"} Sep 10 18:35:54.321103 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:54.321070 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:54.922156 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:54.922117 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:55.131690 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:55.131630 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:55.240897 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:55.240864 2581 generic.go:356] "Generic (PLEG): container finished" podID="591fba1e-a40f-41a3-8955-fd32464bf2ad" containerID="3d27c33ac560840120a5a62c1bd45c5db6e6be389def9dcb54aa886cba0f56d8" exitCode=0 Sep 10 18:35:55.241019 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:55.240948 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xzs2v" event={"ID":"591fba1e-a40f-41a3-8955-fd32464bf2ad","Type":"ContainerDied","Data":"3d27c33ac560840120a5a62c1bd45c5db6e6be389def9dcb54aa886cba0f56d8"} Sep 10 18:35:55.396397 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:55.396339 2581 patch_prober.go:28] interesting pod/image-registry-5678c4d879-pgdx7 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 10 18:35:55.396397 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:55.396379 2581 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" podUID="6b082e08-bb40-4b42-aff0-ffc0ccbf2f02" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 18:35:55.406953 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:55.406923 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:55.769093 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:55.769012 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:55.839692 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:55.839661 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:55.921633 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:55.921582 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:56.144829 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:56.144780 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:56.193172 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:56.193134 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:56.245778 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:56.245756 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xzs2v" event={"ID":"591fba1e-a40f-41a3-8955-fd32464bf2ad","Type":"ContainerStarted","Data":"fbfd5ebcc0bba90b21606f7a2bc354245a1d870d5d5971a3397bad5f291c3316"} Sep 10 18:35:56.245877 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:56.245801 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xzs2v" event={"ID":"591fba1e-a40f-41a3-8955-fd32464bf2ad","Type":"ContainerStarted","Data":"b7b6af9993a2a2b5acf66a518ac94a2cea2b75fe6fce775f6dc7f723eb4a2aa9"} Sep 10 18:35:56.262184 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:56.262141 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xzs2v" podStartSLOduration=3.206763229 podStartE2EDuration="4.262125804s" totalImagesPullingTime="985.389988ms" totalInitContainerRuntime="69.972587ms" isStatefulPod=true podCreationTimestamp="2026-09-10 18:35:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:35:56.260881229 +0000 UTC m=+282.347218703" watchObservedRunningTime="2026-09-10 18:35:56.262125804 +0000 UTC m=+282.348463278" Sep 10 18:35:56.447755 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:56.447691 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:57.186685 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:57.186644 2581 patch_prober.go:28] interesting pod/image-registry-5678c4d879-pgdx7 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 10 18:35:57.187120 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:57.186705 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" podUID="6b082e08-bb40-4b42-aff0-ffc0ccbf2f02" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 18:35:57.619503 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:57.619475 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:57.669340 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:57.669295 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:57.754849 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:57.754831 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-metrics-tls\") pod \"dns-default-4t7vn\" (UID: \"2c41b6ab-b71c-4b51-9cf7-4deba7a3a113\") " pod="openshift-dns/dns-default-4t7vn" Sep 10 18:35:57.757234 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:57.757211 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c41b6ab-b71c-4b51-9cf7-4deba7a3a113-metrics-tls\") pod \"dns-default-4t7vn\" (UID: \"2c41b6ab-b71c-4b51-9cf7-4deba7a3a113\") " pod="openshift-dns/dns-default-4t7vn" Sep 10 18:35:57.787171 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:57.787153 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:57.838974 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:57.838955 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4lj58\"" Sep 10 18:35:57.847538 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:57.847521 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4t7vn" Sep 10 18:35:58.114722 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:58.114691 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4t7vn"] Sep 10 18:35:58.119904 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:35:58.119876 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c41b6ab_b71c_4b51_9cf7_4deba7a3a113.slice/crio-bb49d2e41962b2f5324852ee703ee15136ad5487cb6a479786e7f0d2a0c348ed WatchSource:0}: Error finding container bb49d2e41962b2f5324852ee703ee15136ad5487cb6a479786e7f0d2a0c348ed: Status 404 returned error can't find the container with id bb49d2e41962b2f5324852ee703ee15136ad5487cb6a479786e7f0d2a0c348ed Sep 10 18:35:58.253517 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:58.253442 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4t7vn" event={"ID":"2c41b6ab-b71c-4b51-9cf7-4deba7a3a113","Type":"ContainerStarted","Data":"bb49d2e41962b2f5324852ee703ee15136ad5487cb6a479786e7f0d2a0c348ed"} Sep 10 18:35:58.469910 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:58.469881 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:58.518263 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:58.518200 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:35:59.056911 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:35:59.056880 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:00.261277 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:00.261205 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4t7vn" event={"ID":"2c41b6ab-b71c-4b51-9cf7-4deba7a3a113","Type":"ContainerStarted","Data":"539ef2e820348780a9de199eab71edd57ba5d6079678a5f2bfc43b3dc1082d62"} Sep 10 18:36:00.261277 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:00.261249 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4t7vn" event={"ID":"2c41b6ab-b71c-4b51-9cf7-4deba7a3a113","Type":"ContainerStarted","Data":"35f8a6eb2fe722d66b0117d0368f0e64d43ecc66f60ce726e721402db55b17b9"} Sep 10 18:36:00.261694 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:00.261354 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4t7vn" Sep 10 18:36:00.276193 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:00.276141 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4t7vn" podStartSLOduration=251.555447326 podStartE2EDuration="4m13.276128672s" totalImagesPullingTime="1.720681346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:31:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:36:00.274724036 +0000 UTC m=+286.361061524" watchObservedRunningTime="2026-09-10 18:36:00.276128672 +0000 UTC m=+286.362466146" Sep 10 18:36:03.305495 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:03.305456 2581 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-857d768544-bvllr" podUID="763222b7-e3e3-4ff4-9ef8-65240f20246a" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 10 18:36:05.396958 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:05.396928 2581 patch_prober.go:28] interesting pod/image-registry-5678c4d879-pgdx7 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 10 18:36:05.397302 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:05.396980 2581 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" podUID="6b082e08-bb40-4b42-aff0-ffc0ccbf2f02" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 18:36:07.186368 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:07.186312 2581 patch_prober.go:28] interesting pod/image-registry-5678c4d879-pgdx7 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 10 18:36:07.186838 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:07.186388 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" podUID="6b082e08-bb40-4b42-aff0-ffc0ccbf2f02" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 18:36:08.375037 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:36:08.374992 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-fqmzj" podUID="54ef3c93-1624-46e6-ba09-4fd3bb1fc07f" Sep 10 18:36:09.349512 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:09.349483 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:10.266069 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:10.266041 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4t7vn" Sep 10 18:36:10.504919 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:10.504877 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:12.116028 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:12.115992 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:12.180819 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:12.180789 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:12.750434 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:12.750403 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:13.305623 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:13.305589 2581 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-857d768544-bvllr" podUID="763222b7-e3e3-4ff4-9ef8-65240f20246a" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 10 18:36:13.306068 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:13.305654 2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-857d768544-bvllr" Sep 10 18:36:13.306126 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:13.306080 2581 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"f4cbecaa469fb986e57bcc362a30cd817132ed1a042abc7e2bc113dce85a6b9e"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-857d768544-bvllr" containerMessage="Container service-proxy failed liveness probe, will be restarted" Sep 10 18:36:13.306164 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:13.306130 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-857d768544-bvllr" podUID="763222b7-e3e3-4ff4-9ef8-65240f20246a" containerName="service-proxy" containerID="cri-o://f4cbecaa469fb986e57bcc362a30cd817132ed1a042abc7e2bc113dce85a6b9e" gracePeriod=30 Sep 10 18:36:14.287374 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:14.287350 2581 kubelet.go:1648] "Image garbage collection succeeded" Sep 10 18:36:14.306661 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:14.306633 2581 generic.go:356] "Generic (PLEG): container finished" podID="763222b7-e3e3-4ff4-9ef8-65240f20246a" containerID="f4cbecaa469fb986e57bcc362a30cd817132ed1a042abc7e2bc113dce85a6b9e" exitCode=2 Sep 10 18:36:14.306953 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:14.306693 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-857d768544-bvllr" event={"ID":"763222b7-e3e3-4ff4-9ef8-65240f20246a","Type":"ContainerDied","Data":"f4cbecaa469fb986e57bcc362a30cd817132ed1a042abc7e2bc113dce85a6b9e"} Sep 10 18:36:14.306953 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:14.306725 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-857d768544-bvllr" event={"ID":"763222b7-e3e3-4ff4-9ef8-65240f20246a","Type":"ContainerStarted","Data":"a3b339cc14fc5c19bc9b41b070fa919a6baa6768d62160824cd55e16e56e0e8b"} Sep 10 18:36:15.396380 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:15.396342 2581 patch_prober.go:28] interesting pod/image-registry-5678c4d879-pgdx7 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 10 18:36:15.398855 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:15.396393 2581 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" podUID="6b082e08-bb40-4b42-aff0-ffc0ccbf2f02" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 18:36:15.398855 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:15.396435 2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" Sep 10 18:36:15.398855 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:15.396928 2581 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="registry" containerStatusID={"Type":"cri-o","ID":"0405b7b305682a94c9333ed12414f06117a0bda20ccea8ff384794b7c412f910"} pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" containerMessage="Container registry failed liveness probe, will be restarted" Sep 10 18:36:15.400575 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:15.400554 2581 patch_prober.go:28] interesting pod/image-registry-5678c4d879-pgdx7 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 10 18:36:15.400674 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:15.400599 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" podUID="6b082e08-bb40-4b42-aff0-ffc0ccbf2f02" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 18:36:16.521910 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:16.521878 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:16.569305 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:16.569280 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:17.488933 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:17.488904 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:17.747010 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:17.746930 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:17.798281 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:17.798255 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:18.550755 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:18.550696 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:18.600956 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:18.600922 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:19.374314 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:19.374274 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmzj" Sep 10 18:36:20.754937 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:20.754903 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:20.806061 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:20.806033 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:23.078161 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:23.078126 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:23.129868 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:23.129833 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:23.972044 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:23.972012 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:24.023990 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:24.023961 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:25.401031 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:25.401002 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" Sep 10 18:36:25.517364 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:25.517313 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:25.567406 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:25.567375 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:25.636750 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:25.636724 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:25.689911 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:25.689853 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:26.498730 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:26.498698 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:26.550987 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:26.550962 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:26.968763 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:26.968732 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:29.870621 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:29.870585 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:29.918871 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:29.918844 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:37.281856 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:37.281824 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:37.341778 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:37.341750 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:38.947244 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:38.947209 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:38.995359 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:38.995311 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:40.416168 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:40.416125 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" podUID="6b082e08-bb40-4b42-aff0-ffc0ccbf2f02" containerName="registry" containerID="cri-o://0405b7b305682a94c9333ed12414f06117a0bda20ccea8ff384794b7c412f910" gracePeriod=30 Sep 10 18:36:42.400947 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:42.400907 2581 generic.go:356] "Generic (PLEG): container finished" podID="6b082e08-bb40-4b42-aff0-ffc0ccbf2f02" containerID="0405b7b305682a94c9333ed12414f06117a0bda20ccea8ff384794b7c412f910" exitCode=0 Sep 10 18:36:42.401305 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:42.400990 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" event={"ID":"6b082e08-bb40-4b42-aff0-ffc0ccbf2f02","Type":"ContainerDied","Data":"0405b7b305682a94c9333ed12414f06117a0bda20ccea8ff384794b7c412f910"} Sep 10 18:36:42.401305 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:42.401035 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" event={"ID":"6b082e08-bb40-4b42-aff0-ffc0ccbf2f02","Type":"ContainerStarted","Data":"49bb1e9a41789b5b2a690535eefd7816da43cde30e07bc3f9b6b610e450471e1"} Sep 10 18:36:42.401305 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:42.401059 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" Sep 10 18:36:42.550497 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:42.550470 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:42.597524 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:42.597490 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:43.938342 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:43.938298 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5678c4d879-pgdx7_6b082e08-bb40-4b42-aff0-ffc0ccbf2f02/registry/0.log" Sep 10 18:36:43.944135 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:43.944116 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5678c4d879-pgdx7_6b082e08-bb40-4b42-aff0-ffc0ccbf2f02/registry/1.log" Sep 10 18:36:44.537440 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:44.537410 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lh8gr_dc8250af-b1fc-4108-b6fd-092198a098ac/node-ca/0.log" Sep 10 18:36:45.140187 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:45.140159 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-drn5f_7470a24c-2464-4678-9015-4689638c6b57/serve-healthcheck-canary/0.log" Sep 10 18:36:52.285010 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:52.284980 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:52.332714 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:52.332682 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:55.725400 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:55.725364 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:59.536882 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:59.536851 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:59.637958 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:59.637926 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:36:59.687237 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:36:59.687214 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:37:03.408404 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:37:03.408375 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5678c4d879-pgdx7" Sep 10 18:37:03.652057 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:37:03.652023 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:37:03.702114 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:37:03.702053 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:37:07.285868 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:37:07.285788 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:37:07.333611 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:37:07.333583 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:37:12.110269 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:37:12.110236 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:37:12.210563 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:37:12.210538 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:37:20.934644 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:37:20.934604 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:37:21.042180 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:37:21.042150 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:37:22.281946 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:37:22.281909 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:37:22.335689 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:37:22.335658 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:37:27.920367 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:37:27.920320 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:37:27.972746 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:37:27.972715 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:37:28.368976 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:37:28.368937 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54ef3c93-1624-46e6-ba09-4fd3bb1fc07f-metrics-certs\") pod \"network-metrics-daemon-fqmzj\" (UID: \"54ef3c93-1624-46e6-ba09-4fd3bb1fc07f\") " pod="openshift-multus/network-metrics-daemon-fqmzj" Sep 10 18:37:28.371405 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:37:28.371387 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54ef3c93-1624-46e6-ba09-4fd3bb1fc07f-metrics-certs\") pod \"network-metrics-daemon-fqmzj\" (UID: \"54ef3c93-1624-46e6-ba09-4fd3bb1fc07f\") " pod="openshift-multus/network-metrics-daemon-fqmzj" Sep 10 18:37:28.376616 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:37:28.376596 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-thvkz\"" Sep 10 18:37:28.385060 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:37:28.385045 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmzj" Sep 10 18:37:28.649845 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:37:28.649780 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fqmzj"] Sep 10 18:37:28.656014 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:37:28.655995 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 18:37:29.544243 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:37:29.544172 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fqmzj" event={"ID":"54ef3c93-1624-46e6-ba09-4fd3bb1fc07f","Type":"ContainerStarted","Data":"5e21aff642c2276c85ed926862438b1b04d87a490dbc641a6f843f39edbb9c1d"} Sep 10 18:37:30.549166 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:37:30.549138 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fqmzj" event={"ID":"54ef3c93-1624-46e6-ba09-4fd3bb1fc07f","Type":"ContainerStarted","Data":"147ae9380ce6d95ba006f7f2bab424babcba0bcf7dd0277e5decad2204295778"} Sep 10 18:37:30.549546 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:37:30.549183 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fqmzj" event={"ID":"54ef3c93-1624-46e6-ba09-4fd3bb1fc07f","Type":"ContainerStarted","Data":"e5b1d11ae7fb4ff3df246b92b8484cdf6cc9bfede0c3208121ebfa151c038aed"} Sep 10 18:37:30.564095 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:37:30.564050 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fqmzj" podStartSLOduration=375.716167213 podStartE2EDuration="6m16.564036561s" totalImagesPullingTime="847.869348ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:37:30.563032399 +0000 UTC m=+376.649369876" watchObservedRunningTime="2026-09-10 18:37:30.564036561 +0000 UTC m=+376.650374036" Sep 10 18:37:30.651565 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:37:30.651535 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:37:30.702758 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:37:30.702732 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:37:30.759239 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:37:30.759211 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:37:31.322829 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:37:31.322794 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:37:31.427876 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:37:31.427849 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:37:31.523826 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:37:31.523797 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:37:31.623299 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:37:31.623243 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:38:13.919805 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:38:13.919772 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:38:13.971338 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:38:13.971289 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:38:27.329754 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:38:27.329725 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:38:27.377921 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:38:27.377896 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:38:39.274682 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:38:39.274591 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:38:39.324525 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:38:39.324497 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:38:52.291003 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:38:52.290966 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:38:52.390895 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:38:52.390867 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:38:52.743441 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:38:52.743364 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:38:53.498741 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:38:53.498704 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:38:53.645301 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:38:53.645272 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:38:53.698314 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:38:53.698285 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:38:54.257718 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:38:54.257685 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:38:54.303969 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:38:54.303944 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:39:04.709436 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:39:04.709400 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:39:04.810041 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:39:04.810002 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:39:04.858428 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:39:04.858399 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:39:38.287549 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:39:38.287508 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:39:38.389159 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:39:38.389116 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:39:52.306206 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:39:52.306155 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:39:58.755046 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:39:58.754991 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:40:04.865994 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:04.865960 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:40:05.662776 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:05.662703 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:40:08.461362 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:08.461311 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:40:08.512885 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:08.512862 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:40:13.066708 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:13.066676 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:40:13.121184 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:13.121159 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:40:19.752496 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:19.752460 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:40:19.849084 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:19.849052 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:40:20.329068 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:20.329034 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:40:20.448423 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:20.448396 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:40:21.770856 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:21.770830 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:40:22.150988 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:22.150953 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:40:27.941584 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:27.941553 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:40:28.041725 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:28.041698 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:40:30.558486 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:30.558455 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:40:31.073809 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:31.073775 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:40:31.173023 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:31.172993 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:40:40.290865 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:40.290827 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-kmpvf"] Sep 10 18:40:40.293881 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:40.293859 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-kmpvf" Sep 10 18:40:40.295888 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:40.295865 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 10 18:40:40.295971 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:40.295946 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 10 18:40:40.296360 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:40.296344 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-k29fm\"" Sep 10 18:40:40.303516 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:40.303497 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-kmpvf"] Sep 10 18:40:40.306940 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:40.306921 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:40:40.407499 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:40.407462 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:40:40.418489 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:40.418468 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cfb6a3e0-d12d-43a8-a497-00b7eee2f9e7-bound-sa-token\") pod \"cert-manager-55b4d954d5-kmpvf\" (UID: \"cfb6a3e0-d12d-43a8-a497-00b7eee2f9e7\") " pod="cert-manager/cert-manager-55b4d954d5-kmpvf" Sep 10 18:40:40.418571 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:40.418531 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7d5w\" (UniqueName: \"kubernetes.io/projected/cfb6a3e0-d12d-43a8-a497-00b7eee2f9e7-kube-api-access-z7d5w\") pod \"cert-manager-55b4d954d5-kmpvf\" (UID: \"cfb6a3e0-d12d-43a8-a497-00b7eee2f9e7\") " pod="cert-manager/cert-manager-55b4d954d5-kmpvf" Sep 10 18:40:40.519646 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:40.519621 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7d5w\" (UniqueName: \"kubernetes.io/projected/cfb6a3e0-d12d-43a8-a497-00b7eee2f9e7-kube-api-access-z7d5w\") pod \"cert-manager-55b4d954d5-kmpvf\" (UID: \"cfb6a3e0-d12d-43a8-a497-00b7eee2f9e7\") " pod="cert-manager/cert-manager-55b4d954d5-kmpvf" Sep 10 18:40:40.519837 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:40.519816 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cfb6a3e0-d12d-43a8-a497-00b7eee2f9e7-bound-sa-token\") pod \"cert-manager-55b4d954d5-kmpvf\" (UID: \"cfb6a3e0-d12d-43a8-a497-00b7eee2f9e7\") " pod="cert-manager/cert-manager-55b4d954d5-kmpvf" Sep 10 18:40:40.526726 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:40.526702 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cfb6a3e0-d12d-43a8-a497-00b7eee2f9e7-bound-sa-token\") pod \"cert-manager-55b4d954d5-kmpvf\" (UID: \"cfb6a3e0-d12d-43a8-a497-00b7eee2f9e7\") " pod="cert-manager/cert-manager-55b4d954d5-kmpvf" Sep 10 18:40:40.526845 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:40.526794 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7d5w\" (UniqueName: \"kubernetes.io/projected/cfb6a3e0-d12d-43a8-a497-00b7eee2f9e7-kube-api-access-z7d5w\") pod \"cert-manager-55b4d954d5-kmpvf\" (UID: \"cfb6a3e0-d12d-43a8-a497-00b7eee2f9e7\") " pod="cert-manager/cert-manager-55b4d954d5-kmpvf" Sep 10 18:40:40.602831 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:40.602748 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-kmpvf" Sep 10 18:40:40.869510 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:40.869439 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-kmpvf"] Sep 10 18:40:40.872358 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:40:40.872336 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfb6a3e0_d12d_43a8_a497_00b7eee2f9e7.slice/crio-cb986d7b71a51037c26b70fd0ec34920e0758d179b2d87828f0caab35050d367 WatchSource:0}: Error finding container cb986d7b71a51037c26b70fd0ec34920e0758d179b2d87828f0caab35050d367: Status 404 returned error can't find the container with id cb986d7b71a51037c26b70fd0ec34920e0758d179b2d87828f0caab35050d367 Sep 10 18:40:41.155103 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:41.155022 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-kmpvf" event={"ID":"cfb6a3e0-d12d-43a8-a497-00b7eee2f9e7","Type":"ContainerStarted","Data":"cb986d7b71a51037c26b70fd0ec34920e0758d179b2d87828f0caab35050d367"} Sep 10 18:40:42.124877 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:42.124840 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:40:44.167834 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:44.167742 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-kmpvf" event={"ID":"cfb6a3e0-d12d-43a8-a497-00b7eee2f9e7","Type":"ContainerStarted","Data":"10880894d8efd30bea5e078c42a9b273671f19c2d4fe9b37722b41dbfc029603"} Sep 10 18:40:44.181578 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:44.181519 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-kmpvf" podStartSLOduration=1.169587629 podStartE2EDuration="4.181499745s" totalImagesPullingTime="3.011912116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:40:44.180060826 +0000 UTC m=+570.266398300" watchObservedRunningTime="2026-09-10 18:40:44.181499745 +0000 UTC m=+570.267837224" Sep 10 18:40:48.334698 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:48.334668 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:40:48.400206 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:48.400169 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:40:53.892638 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:53.892602 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:40:54.003747 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:54.003715 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:40:54.207592 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:54.207506 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:40:54.413752 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:54.413724 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:40:54.508229 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:54.508129 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:40:54.548607 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:54.548574 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4"] Sep 10 18:40:54.551845 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:54.551830 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4" Sep 10 18:40:54.554317 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:54.554290 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2hqf5\"" Sep 10 18:40:54.554470 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:54.554372 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 10 18:40:54.554470 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:54.554406 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 10 18:40:54.554575 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:54.554549 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 10 18:40:54.554618 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:54.554601 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 10 18:40:54.574069 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:54.574049 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4"] Sep 10 18:40:54.630504 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:54.630470 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9b8839e0-8ba6-4f99-94b3-c4202c9e1404-manifests\") pod \"opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4\" (UID: \"9b8839e0-8ba6-4f99-94b3-c4202c9e1404\") " pod="opendatahub/opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4" Sep 10 18:40:54.630504 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:54.630501 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qccb\" (UniqueName: \"kubernetes.io/projected/9b8839e0-8ba6-4f99-94b3-c4202c9e1404-kube-api-access-2qccb\") pod \"opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4\" (UID: \"9b8839e0-8ba6-4f99-94b3-c4202c9e1404\") " pod="opendatahub/opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4" Sep 10 18:40:54.630676 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:54.630522 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9b8839e0-8ba6-4f99-94b3-c4202c9e1404-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4\" (UID: \"9b8839e0-8ba6-4f99-94b3-c4202c9e1404\") " pod="opendatahub/opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4" Sep 10 18:40:54.630676 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:54.630539 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b8839e0-8ba6-4f99-94b3-c4202c9e1404-tmp\") pod \"opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4\" (UID: \"9b8839e0-8ba6-4f99-94b3-c4202c9e1404\") " pod="opendatahub/opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4" Sep 10 18:40:54.630676 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:54.630600 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b8839e0-8ba6-4f99-94b3-c4202c9e1404-webhook-cert\") pod \"opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4\" (UID: \"9b8839e0-8ba6-4f99-94b3-c4202c9e1404\") " pod="opendatahub/opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4" Sep 10 18:40:54.730931 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:54.730889 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b8839e0-8ba6-4f99-94b3-c4202c9e1404-webhook-cert\") pod \"opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4\" (UID: \"9b8839e0-8ba6-4f99-94b3-c4202c9e1404\") " pod="opendatahub/opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4" Sep 10 18:40:54.731098 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:54.730945 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9b8839e0-8ba6-4f99-94b3-c4202c9e1404-manifests\") pod \"opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4\" (UID: \"9b8839e0-8ba6-4f99-94b3-c4202c9e1404\") " pod="opendatahub/opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4" Sep 10 18:40:54.731098 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:54.730983 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qccb\" (UniqueName: \"kubernetes.io/projected/9b8839e0-8ba6-4f99-94b3-c4202c9e1404-kube-api-access-2qccb\") pod \"opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4\" (UID: \"9b8839e0-8ba6-4f99-94b3-c4202c9e1404\") " pod="opendatahub/opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4" Sep 10 18:40:54.731098 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:54.731015 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9b8839e0-8ba6-4f99-94b3-c4202c9e1404-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4\" (UID: \"9b8839e0-8ba6-4f99-94b3-c4202c9e1404\") " pod="opendatahub/opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4" Sep 10 18:40:54.731098 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:54.731040 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b8839e0-8ba6-4f99-94b3-c4202c9e1404-tmp\") pod \"opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4\" (UID: \"9b8839e0-8ba6-4f99-94b3-c4202c9e1404\") " pod="opendatahub/opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4" Sep 10 18:40:54.731427 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:54.731408 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9b8839e0-8ba6-4f99-94b3-c4202c9e1404-manifests\") pod \"opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4\" (UID: \"9b8839e0-8ba6-4f99-94b3-c4202c9e1404\") " pod="opendatahub/opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4" Sep 10 18:40:54.733468 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:54.733448 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b8839e0-8ba6-4f99-94b3-c4202c9e1404-tmp\") pod \"opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4\" (UID: \"9b8839e0-8ba6-4f99-94b3-c4202c9e1404\") " pod="opendatahub/opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4" Sep 10 18:40:54.733651 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:54.733630 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b8839e0-8ba6-4f99-94b3-c4202c9e1404-webhook-cert\") pod \"opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4\" (UID: \"9b8839e0-8ba6-4f99-94b3-c4202c9e1404\") " pod="opendatahub/opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4" Sep 10 18:40:54.733690 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:54.733648 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9b8839e0-8ba6-4f99-94b3-c4202c9e1404-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4\" (UID: \"9b8839e0-8ba6-4f99-94b3-c4202c9e1404\") " pod="opendatahub/opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4" Sep 10 18:40:54.737613 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:54.737584 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:40:54.738389 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:54.738364 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qccb\" (UniqueName: \"kubernetes.io/projected/9b8839e0-8ba6-4f99-94b3-c4202c9e1404-kube-api-access-2qccb\") pod \"opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4\" (UID: \"9b8839e0-8ba6-4f99-94b3-c4202c9e1404\") " pod="opendatahub/opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4" Sep 10 18:40:54.837725 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:54.837701 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:40:54.862145 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:54.862120 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4" Sep 10 18:40:54.938402 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:54.938304 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:40:55.043827 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:55.043795 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:40:55.111062 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:55.110998 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:40:55.141840 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:55.141814 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4"] Sep 10 18:40:55.145188 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:40:55.145164 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b8839e0_8ba6_4f99_94b3_c4202c9e1404.slice/crio-c2e4f0708920f1e0b1a39a17e2f2a1dd970b446f5e7921437fb0622338316ad7 WatchSource:0}: Error finding container c2e4f0708920f1e0b1a39a17e2f2a1dd970b446f5e7921437fb0622338316ad7: Status 404 returned error can't find the container with id c2e4f0708920f1e0b1a39a17e2f2a1dd970b446f5e7921437fb0622338316ad7 Sep 10 18:40:55.207384 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:55.207353 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4" event={"ID":"9b8839e0-8ba6-4f99-94b3-c4202c9e1404","Type":"ContainerStarted","Data":"c2e4f0708920f1e0b1a39a17e2f2a1dd970b446f5e7921437fb0622338316ad7"} Sep 10 18:40:58.221124 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:58.221097 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4" event={"ID":"9b8839e0-8ba6-4f99-94b3-c4202c9e1404","Type":"ContainerStarted","Data":"dcf9fcb0de29261fdcbdd480e60b0664dd5187f4fe2ef7e85eb4e913b55e63ed"} Sep 10 18:40:59.231127 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:59.231101 2581 generic.go:356] "Generic (PLEG): container finished" podID="9b8839e0-8ba6-4f99-94b3-c4202c9e1404" containerID="dcf9fcb0de29261fdcbdd480e60b0664dd5187f4fe2ef7e85eb4e913b55e63ed" exitCode=0 Sep 10 18:40:59.231472 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:40:59.231175 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4" event={"ID":"9b8839e0-8ba6-4f99-94b3-c4202c9e1404","Type":"ContainerDied","Data":"dcf9fcb0de29261fdcbdd480e60b0664dd5187f4fe2ef7e85eb4e913b55e63ed"} Sep 10 18:41:00.237587 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:00.237549 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4" event={"ID":"9b8839e0-8ba6-4f99-94b3-c4202c9e1404","Type":"ContainerStarted","Data":"1fe02671a728387a6ae4371df234ce8564c3be7db4e90005ba10f19ae099a569"} Sep 10 18:41:00.237960 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:00.237682 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4" Sep 10 18:41:00.255517 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:00.255475 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4" podStartSLOduration=1.7263187260000001 podStartE2EDuration="6.255459843s" totalImagesPullingTime="3.52685198s" totalInitContainerRuntime="1.002289137s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:40:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:41:00.254046675 +0000 UTC m=+586.340384151" watchObservedRunningTime="2026-09-10 18:41:00.255459843 +0000 UTC m=+586.341797317" Sep 10 18:41:03.267660 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:03.267627 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:03.320892 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:03.320853 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:05.217377 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:05.217319 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:05.269829 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:05.269784 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:07.336663 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:07.336621 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:07.390622 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:07.390599 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:11.242883 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:11.242855 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4" Sep 10 18:41:11.289605 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:11.289557 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:12.501540 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:12.501503 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:13.691181 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:13.691145 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:13.839982 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:13.839937 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:14.074720 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:14.074686 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:14.111245 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:14.111219 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-64dfbcccbf-9pgvj"] Sep 10 18:41:14.183953 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:14.183919 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-64dfbcccbf-9pgvj"] Sep 10 18:41:14.184098 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:14.184036 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-64dfbcccbf-9pgvj" Sep 10 18:41:14.187859 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:14.187837 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 10 18:41:14.187989 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:14.187845 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 10 18:41:14.188117 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:14.188105 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 10 18:41:14.188515 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:14.188498 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-vgtwx\"" Sep 10 18:41:14.188676 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:14.188659 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 10 18:41:14.261037 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:14.261012 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/828ca3e2-4140-4cb3-be58-7aa5395d90ac-tls-certs\") pod \"kube-auth-proxy-64dfbcccbf-9pgvj\" (UID: \"828ca3e2-4140-4cb3-be58-7aa5395d90ac\") " pod="openshift-ingress/kube-auth-proxy-64dfbcccbf-9pgvj" Sep 10 18:41:14.261129 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:14.261047 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/828ca3e2-4140-4cb3-be58-7aa5395d90ac-tmp\") pod \"kube-auth-proxy-64dfbcccbf-9pgvj\" (UID: \"828ca3e2-4140-4cb3-be58-7aa5395d90ac\") " pod="openshift-ingress/kube-auth-proxy-64dfbcccbf-9pgvj" Sep 10 18:41:14.261129 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:14.261078 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhlvt\" (UniqueName: \"kubernetes.io/projected/828ca3e2-4140-4cb3-be58-7aa5395d90ac-kube-api-access-rhlvt\") pod \"kube-auth-proxy-64dfbcccbf-9pgvj\" (UID: \"828ca3e2-4140-4cb3-be58-7aa5395d90ac\") " pod="openshift-ingress/kube-auth-proxy-64dfbcccbf-9pgvj" Sep 10 18:41:14.277666 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:14.277638 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:14.361777 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:14.361708 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhlvt\" (UniqueName: \"kubernetes.io/projected/828ca3e2-4140-4cb3-be58-7aa5395d90ac-kube-api-access-rhlvt\") pod \"kube-auth-proxy-64dfbcccbf-9pgvj\" (UID: \"828ca3e2-4140-4cb3-be58-7aa5395d90ac\") " pod="openshift-ingress/kube-auth-proxy-64dfbcccbf-9pgvj" Sep 10 18:41:14.361777 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:14.361747 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/828ca3e2-4140-4cb3-be58-7aa5395d90ac-tls-certs\") pod \"kube-auth-proxy-64dfbcccbf-9pgvj\" (UID: \"828ca3e2-4140-4cb3-be58-7aa5395d90ac\") " pod="openshift-ingress/kube-auth-proxy-64dfbcccbf-9pgvj" Sep 10 18:41:14.361777 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:14.361772 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/828ca3e2-4140-4cb3-be58-7aa5395d90ac-tmp\") pod \"kube-auth-proxy-64dfbcccbf-9pgvj\" (UID: \"828ca3e2-4140-4cb3-be58-7aa5395d90ac\") " pod="openshift-ingress/kube-auth-proxy-64dfbcccbf-9pgvj" Sep 10 18:41:14.363862 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:14.363829 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 10 18:41:14.364099 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:14.364081 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/828ca3e2-4140-4cb3-be58-7aa5395d90ac-tmp\") pod \"kube-auth-proxy-64dfbcccbf-9pgvj\" (UID: \"828ca3e2-4140-4cb3-be58-7aa5395d90ac\") " pod="openshift-ingress/kube-auth-proxy-64dfbcccbf-9pgvj" Sep 10 18:41:14.374108 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:14.374081 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 10 18:41:14.375042 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:14.375018 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/828ca3e2-4140-4cb3-be58-7aa5395d90ac-tls-certs\") pod \"kube-auth-proxy-64dfbcccbf-9pgvj\" (UID: \"828ca3e2-4140-4cb3-be58-7aa5395d90ac\") " pod="openshift-ingress/kube-auth-proxy-64dfbcccbf-9pgvj" Sep 10 18:41:14.383408 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:14.383388 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 10 18:41:14.394569 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:14.394549 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhlvt\" (UniqueName: \"kubernetes.io/projected/828ca3e2-4140-4cb3-be58-7aa5395d90ac-kube-api-access-rhlvt\") pod \"kube-auth-proxy-64dfbcccbf-9pgvj\" (UID: \"828ca3e2-4140-4cb3-be58-7aa5395d90ac\") " pod="openshift-ingress/kube-auth-proxy-64dfbcccbf-9pgvj" Sep 10 18:41:14.494802 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:14.494773 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-vgtwx\"" Sep 10 18:41:14.503314 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:14.503287 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-64dfbcccbf-9pgvj" Sep 10 18:41:14.800555 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:14.800527 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-64dfbcccbf-9pgvj"] Sep 10 18:41:14.803624 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:41:14.803599 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod828ca3e2_4140_4cb3_be58_7aa5395d90ac.slice/crio-5cad96cd7abf857b0c734b5c546ae6357a3343a182f622d41f70b4ee34d95cc9 WatchSource:0}: Error finding container 5cad96cd7abf857b0c734b5c546ae6357a3343a182f622d41f70b4ee34d95cc9: Status 404 returned error can't find the container with id 5cad96cd7abf857b0c734b5c546ae6357a3343a182f622d41f70b4ee34d95cc9 Sep 10 18:41:15.294468 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:15.294429 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-64dfbcccbf-9pgvj" event={"ID":"828ca3e2-4140-4cb3-be58-7aa5395d90ac","Type":"ContainerStarted","Data":"5cad96cd7abf857b0c734b5c546ae6357a3343a182f622d41f70b4ee34d95cc9"} Sep 10 18:41:15.361022 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:15.360985 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml"] Sep 10 18:41:15.363063 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:15.363043 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml" Sep 10 18:41:15.365003 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:15.364981 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 10 18:41:15.365112 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:15.364980 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 10 18:41:15.365180 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:15.365133 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 10 18:41:15.365289 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:15.365263 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 10 18:41:15.365416 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:15.365293 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rmns6\"" Sep 10 18:41:15.374308 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:15.374273 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml"] Sep 10 18:41:15.391600 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:15.391572 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:15.469527 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:15.469145 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d4p5\" (UniqueName: \"kubernetes.io/projected/5ea8ee98-b277-45e0-b74c-038215891e3c-kube-api-access-6d4p5\") pod \"istiod-openshift-gateway-7b7995d5cd-bnrml\" (UID: \"5ea8ee98-b277-45e0-b74c-038215891e3c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml" Sep 10 18:41:15.469527 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:15.469198 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ea8ee98-b277-45e0-b74c-038215891e3c-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-bnrml\" (UID: \"5ea8ee98-b277-45e0-b74c-038215891e3c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml" Sep 10 18:41:15.469527 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:15.469232 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5ea8ee98-b277-45e0-b74c-038215891e3c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-bnrml\" (UID: \"5ea8ee98-b277-45e0-b74c-038215891e3c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml" Sep 10 18:41:15.469527 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:15.469260 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5ea8ee98-b277-45e0-b74c-038215891e3c-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-bnrml\" (UID: \"5ea8ee98-b277-45e0-b74c-038215891e3c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml" Sep 10 18:41:15.469527 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:15.469297 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ea8ee98-b277-45e0-b74c-038215891e3c-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-bnrml\" (UID: \"5ea8ee98-b277-45e0-b74c-038215891e3c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml" Sep 10 18:41:15.469527 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:15.469340 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5ea8ee98-b277-45e0-b74c-038215891e3c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-bnrml\" (UID: \"5ea8ee98-b277-45e0-b74c-038215891e3c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml" Sep 10 18:41:15.469527 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:15.469396 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5ea8ee98-b277-45e0-b74c-038215891e3c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-bnrml\" (UID: \"5ea8ee98-b277-45e0-b74c-038215891e3c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml" Sep 10 18:41:15.483888 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:15.483865 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:15.570910 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:15.570834 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5ea8ee98-b277-45e0-b74c-038215891e3c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-bnrml\" (UID: \"5ea8ee98-b277-45e0-b74c-038215891e3c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml" Sep 10 18:41:15.570910 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:15.570878 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5ea8ee98-b277-45e0-b74c-038215891e3c-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-bnrml\" (UID: \"5ea8ee98-b277-45e0-b74c-038215891e3c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml" Sep 10 18:41:15.571100 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:15.570919 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ea8ee98-b277-45e0-b74c-038215891e3c-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-bnrml\" (UID: \"5ea8ee98-b277-45e0-b74c-038215891e3c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml" Sep 10 18:41:15.571100 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:15.570947 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5ea8ee98-b277-45e0-b74c-038215891e3c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-bnrml\" (UID: \"5ea8ee98-b277-45e0-b74c-038215891e3c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml" Sep 10 18:41:15.571100 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:15.570974 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5ea8ee98-b277-45e0-b74c-038215891e3c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-bnrml\" (UID: \"5ea8ee98-b277-45e0-b74c-038215891e3c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml" Sep 10 18:41:15.571100 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:15.571032 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d4p5\" (UniqueName: \"kubernetes.io/projected/5ea8ee98-b277-45e0-b74c-038215891e3c-kube-api-access-6d4p5\") pod \"istiod-openshift-gateway-7b7995d5cd-bnrml\" (UID: \"5ea8ee98-b277-45e0-b74c-038215891e3c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml" Sep 10 18:41:15.571100 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:15.571064 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ea8ee98-b277-45e0-b74c-038215891e3c-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-bnrml\" (UID: \"5ea8ee98-b277-45e0-b74c-038215891e3c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml" Sep 10 18:41:15.571986 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:15.571933 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5ea8ee98-b277-45e0-b74c-038215891e3c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-bnrml\" (UID: \"5ea8ee98-b277-45e0-b74c-038215891e3c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml" Sep 10 18:41:15.574380 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:15.574357 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5ea8ee98-b277-45e0-b74c-038215891e3c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-bnrml\" (UID: \"5ea8ee98-b277-45e0-b74c-038215891e3c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml" Sep 10 18:41:15.574562 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:15.574535 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5ea8ee98-b277-45e0-b74c-038215891e3c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-bnrml\" (UID: \"5ea8ee98-b277-45e0-b74c-038215891e3c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml" Sep 10 18:41:15.574769 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:15.574738 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ea8ee98-b277-45e0-b74c-038215891e3c-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-bnrml\" (UID: \"5ea8ee98-b277-45e0-b74c-038215891e3c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml" Sep 10 18:41:15.574989 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:15.574969 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5ea8ee98-b277-45e0-b74c-038215891e3c-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-bnrml\" (UID: \"5ea8ee98-b277-45e0-b74c-038215891e3c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml" Sep 10 18:41:15.579095 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:15.579068 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d4p5\" (UniqueName: \"kubernetes.io/projected/5ea8ee98-b277-45e0-b74c-038215891e3c-kube-api-access-6d4p5\") pod \"istiod-openshift-gateway-7b7995d5cd-bnrml\" (UID: \"5ea8ee98-b277-45e0-b74c-038215891e3c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml" Sep 10 18:41:15.580414 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:15.580375 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ea8ee98-b277-45e0-b74c-038215891e3c-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-bnrml\" (UID: \"5ea8ee98-b277-45e0-b74c-038215891e3c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml" Sep 10 18:41:15.682826 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:15.682798 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml" Sep 10 18:41:15.970506 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:15.970427 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml"] Sep 10 18:41:15.974098 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:41:15.974069 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ea8ee98_b277_45e0_b74c_038215891e3c.slice/crio-f4e3b0715fd9806ec4d4746e0941216b29aeab462d78d4abdc0160de3951c62c WatchSource:0}: Error finding container f4e3b0715fd9806ec4d4746e0941216b29aeab462d78d4abdc0160de3951c62c: Status 404 returned error can't find the container with id f4e3b0715fd9806ec4d4746e0941216b29aeab462d78d4abdc0160de3951c62c Sep 10 18:41:16.301138 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:16.301057 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml" event={"ID":"5ea8ee98-b277-45e0-b74c-038215891e3c","Type":"ContainerStarted","Data":"f4e3b0715fd9806ec4d4746e0941216b29aeab462d78d4abdc0160de3951c62c"} Sep 10 18:41:16.302194 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:16.302148 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:17.580823 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:17.580801 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 10 18:41:18.311486 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:18.311442 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-64dfbcccbf-9pgvj" event={"ID":"828ca3e2-4140-4cb3-be58-7aa5395d90ac","Type":"ContainerStarted","Data":"80b0d50948463bb0d64b9a3b30ac018c6df9e810f86cba278a36ae2cb118c259"} Sep 10 18:41:18.330310 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:18.330265 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-64dfbcccbf-9pgvj" podStartSLOduration=1.5573885729999999 podStartE2EDuration="4.330250385s" totalImagesPullingTime="2.772861812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:41:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:41:18.329660049 +0000 UTC m=+604.415997525" watchObservedRunningTime="2026-09-10 18:41:18.330250385 +0000 UTC m=+604.416587861" Sep 10 18:41:18.365736 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:18.365694 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:18.425186 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:18.425150 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:18.473509 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:18.473474 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:19.356560 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:19.356513 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 18:41:19.356829 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:19.356600 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 18:41:19.783789 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:19.783678 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:19.883733 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:19.883701 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:20.327616 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:20.327579 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml" event={"ID":"5ea8ee98-b277-45e0-b74c-038215891e3c","Type":"ContainerStarted","Data":"b8abfcdbfcd0890632dba76523126d32a58c351aac76a93faf9d11e6ab960247"} Sep 10 18:41:20.327922 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:20.327871 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml" Sep 10 18:41:20.329251 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:20.329219 2581 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-bnrml container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 18:41:20.329408 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:20.329282 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml" podUID="5ea8ee98-b277-45e0-b74c-038215891e3c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 18:41:20.351720 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:20.351676 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml" podStartSLOduration=1.971894593 podStartE2EDuration="5.351662547s" totalImagesPullingTime="3.379767954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:41:20.349855441 +0000 UTC m=+606.436192918" watchObservedRunningTime="2026-09-10 18:41:20.351662547 +0000 UTC m=+606.438000022" Sep 10 18:41:20.378141 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:20.378109 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:20.496410 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:20.496383 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:20.596670 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:20.596601 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:21.332463 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:21.332437 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml" Sep 10 18:41:24.788580 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:24.788543 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9rgkh"] Sep 10 18:41:24.792763 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:24.792746 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-9rgkh" Sep 10 18:41:24.794766 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:24.794736 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 10 18:41:24.794895 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:24.794780 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-bt65m\"" Sep 10 18:41:24.802644 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:24.802614 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9rgkh"] Sep 10 18:41:24.813718 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:24.813692 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:24.841993 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:24.841956 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqbhq\" (UniqueName: \"kubernetes.io/projected/7eba2f85-27fd-4fe5-8e01-0facd1e8bc78-kube-api-access-sqbhq\") pod \"odh-model-controller-7998c67dc4-9rgkh\" (UID: \"7eba2f85-27fd-4fe5-8e01-0facd1e8bc78\") " pod="opendatahub/odh-model-controller-7998c67dc4-9rgkh" Sep 10 18:41:24.842103 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:24.842014 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7eba2f85-27fd-4fe5-8e01-0facd1e8bc78-cert\") pod \"odh-model-controller-7998c67dc4-9rgkh\" (UID: \"7eba2f85-27fd-4fe5-8e01-0facd1e8bc78\") " pod="opendatahub/odh-model-controller-7998c67dc4-9rgkh" Sep 10 18:41:24.910566 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:24.910533 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:24.942679 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:24.942646 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqbhq\" (UniqueName: \"kubernetes.io/projected/7eba2f85-27fd-4fe5-8e01-0facd1e8bc78-kube-api-access-sqbhq\") pod \"odh-model-controller-7998c67dc4-9rgkh\" (UID: \"7eba2f85-27fd-4fe5-8e01-0facd1e8bc78\") " pod="opendatahub/odh-model-controller-7998c67dc4-9rgkh" Sep 10 18:41:24.942853 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:24.942713 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7eba2f85-27fd-4fe5-8e01-0facd1e8bc78-cert\") pod \"odh-model-controller-7998c67dc4-9rgkh\" (UID: \"7eba2f85-27fd-4fe5-8e01-0facd1e8bc78\") " pod="opendatahub/odh-model-controller-7998c67dc4-9rgkh" Sep 10 18:41:24.942916 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:41:24.942888 2581 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 10 18:41:24.942968 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:41:24.942952 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eba2f85-27fd-4fe5-8e01-0facd1e8bc78-cert podName:7eba2f85-27fd-4fe5-8e01-0facd1e8bc78 nodeName:}" failed. No retries permitted until 2026-09-10 18:41:25.44293014 +0000 UTC m=+611.529267607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7eba2f85-27fd-4fe5-8e01-0facd1e8bc78-cert") pod "odh-model-controller-7998c67dc4-9rgkh" (UID: "7eba2f85-27fd-4fe5-8e01-0facd1e8bc78") : secret "odh-model-controller-webhook-cert" not found Sep 10 18:41:24.951369 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:24.951342 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqbhq\" (UniqueName: \"kubernetes.io/projected/7eba2f85-27fd-4fe5-8e01-0facd1e8bc78-kube-api-access-sqbhq\") pod \"odh-model-controller-7998c67dc4-9rgkh\" (UID: \"7eba2f85-27fd-4fe5-8e01-0facd1e8bc78\") " pod="opendatahub/odh-model-controller-7998c67dc4-9rgkh" Sep 10 18:41:25.013822 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:25.013791 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:25.111359 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:25.111270 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:25.314690 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:25.314660 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:25.359528 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:25.359500 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-958jt"] Sep 10 18:41:25.362759 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:25.362707 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-958jt" Sep 10 18:41:25.364686 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:25.364666 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 10 18:41:25.364924 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:25.364910 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-97fms\"" Sep 10 18:41:25.373029 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:25.373005 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-958jt"] Sep 10 18:41:25.381901 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:25.381882 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:25.447962 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:25.447932 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rr59\" (UniqueName: \"kubernetes.io/projected/5bc21d47-8dd2-4012-815a-588353e3cd58-kube-api-access-9rr59\") pod \"model-serving-api-54d89bd5fc-958jt\" (UID: \"5bc21d47-8dd2-4012-815a-588353e3cd58\") " pod="opendatahub/model-serving-api-54d89bd5fc-958jt" Sep 10 18:41:25.448121 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:25.447980 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bc21d47-8dd2-4012-815a-588353e3cd58-tls-certs\") pod \"model-serving-api-54d89bd5fc-958jt\" (UID: \"5bc21d47-8dd2-4012-815a-588353e3cd58\") " pod="opendatahub/model-serving-api-54d89bd5fc-958jt" Sep 10 18:41:25.448121 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:25.448109 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7eba2f85-27fd-4fe5-8e01-0facd1e8bc78-cert\") pod \"odh-model-controller-7998c67dc4-9rgkh\" (UID: \"7eba2f85-27fd-4fe5-8e01-0facd1e8bc78\") " pod="opendatahub/odh-model-controller-7998c67dc4-9rgkh" Sep 10 18:41:25.450951 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:25.450930 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7eba2f85-27fd-4fe5-8e01-0facd1e8bc78-cert\") pod \"odh-model-controller-7998c67dc4-9rgkh\" (UID: \"7eba2f85-27fd-4fe5-8e01-0facd1e8bc78\") " pod="opendatahub/odh-model-controller-7998c67dc4-9rgkh" Sep 10 18:41:25.549016 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:25.548975 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rr59\" (UniqueName: \"kubernetes.io/projected/5bc21d47-8dd2-4012-815a-588353e3cd58-kube-api-access-9rr59\") pod \"model-serving-api-54d89bd5fc-958jt\" (UID: \"5bc21d47-8dd2-4012-815a-588353e3cd58\") " pod="opendatahub/model-serving-api-54d89bd5fc-958jt" Sep 10 18:41:25.549203 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:25.549033 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bc21d47-8dd2-4012-815a-588353e3cd58-tls-certs\") pod \"model-serving-api-54d89bd5fc-958jt\" (UID: \"5bc21d47-8dd2-4012-815a-588353e3cd58\") " pod="opendatahub/model-serving-api-54d89bd5fc-958jt" Sep 10 18:41:25.549203 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:41:25.549161 2581 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Sep 10 18:41:25.549318 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:41:25.549219 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bc21d47-8dd2-4012-815a-588353e3cd58-tls-certs podName:5bc21d47-8dd2-4012-815a-588353e3cd58 nodeName:}" failed. No retries permitted until 2026-09-10 18:41:26.04920078 +0000 UTC m=+612.135538253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5bc21d47-8dd2-4012-815a-588353e3cd58-tls-certs") pod "model-serving-api-54d89bd5fc-958jt" (UID: "5bc21d47-8dd2-4012-815a-588353e3cd58") : secret "model-serving-api-tls" not found Sep 10 18:41:25.557172 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:25.557145 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rr59\" (UniqueName: \"kubernetes.io/projected/5bc21d47-8dd2-4012-815a-588353e3cd58-kube-api-access-9rr59\") pod \"model-serving-api-54d89bd5fc-958jt\" (UID: \"5bc21d47-8dd2-4012-815a-588353e3cd58\") " pod="opendatahub/model-serving-api-54d89bd5fc-958jt" Sep 10 18:41:25.702899 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:25.702808 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-9rgkh" Sep 10 18:41:25.813175 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:25.813147 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:25.976025 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:25.975876 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9rgkh"] Sep 10 18:41:25.978500 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:41:25.978469 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eba2f85_27fd_4fe5_8e01_0facd1e8bc78.slice/crio-50601bd9118593cca12b8e45445bdb10e82c968e9ed98f7fd5c29de7f73b3dc3 WatchSource:0}: Error finding container 50601bd9118593cca12b8e45445bdb10e82c968e9ed98f7fd5c29de7f73b3dc3: Status 404 returned error can't find the container with id 50601bd9118593cca12b8e45445bdb10e82c968e9ed98f7fd5c29de7f73b3dc3 Sep 10 18:41:26.054888 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:26.054848 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bc21d47-8dd2-4012-815a-588353e3cd58-tls-certs\") pod \"model-serving-api-54d89bd5fc-958jt\" (UID: \"5bc21d47-8dd2-4012-815a-588353e3cd58\") " pod="opendatahub/model-serving-api-54d89bd5fc-958jt" Sep 10 18:41:26.057564 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:26.057530 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bc21d47-8dd2-4012-815a-588353e3cd58-tls-certs\") pod \"model-serving-api-54d89bd5fc-958jt\" (UID: \"5bc21d47-8dd2-4012-815a-588353e3cd58\") " pod="opendatahub/model-serving-api-54d89bd5fc-958jt" Sep 10 18:41:26.274420 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:26.274316 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-958jt" Sep 10 18:41:26.352691 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:26.352657 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-9rgkh" event={"ID":"7eba2f85-27fd-4fe5-8e01-0facd1e8bc78","Type":"ContainerStarted","Data":"50601bd9118593cca12b8e45445bdb10e82c968e9ed98f7fd5c29de7f73b3dc3"} Sep 10 18:41:26.546245 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:26.546212 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-958jt"] Sep 10 18:41:26.549425 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:41:26.549395 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bc21d47_8dd2_4012_815a_588353e3cd58.slice/crio-91f39e1e578c563a3f835665d1cfcfe09d2a7ce31866971402ddf514b0535638 WatchSource:0}: Error finding container 91f39e1e578c563a3f835665d1cfcfe09d2a7ce31866971402ddf514b0535638: Status 404 returned error can't find the container with id 91f39e1e578c563a3f835665d1cfcfe09d2a7ce31866971402ddf514b0535638 Sep 10 18:41:27.358087 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:27.358050 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-958jt" event={"ID":"5bc21d47-8dd2-4012-815a-588353e3cd58","Type":"ContainerStarted","Data":"91f39e1e578c563a3f835665d1cfcfe09d2a7ce31866971402ddf514b0535638"} Sep 10 18:41:30.176504 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:30.176461 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:30.379395 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:30.379361 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-958jt" Sep 10 18:41:30.379580 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:30.379409 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-958jt" event={"ID":"5bc21d47-8dd2-4012-815a-588353e3cd58","Type":"ContainerStarted","Data":"529bf2e0729a8cbc43636b2508346487009207acb919b4d5d17986ad6cf92912"} Sep 10 18:41:30.379580 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:30.379428 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-9rgkh" event={"ID":"7eba2f85-27fd-4fe5-8e01-0facd1e8bc78","Type":"ContainerStarted","Data":"3b14fa2267195b391d619923f749e6bd8b686e0b487de49554b95fdcec3a7553"} Sep 10 18:41:30.379580 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:30.379446 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-9rgkh" Sep 10 18:41:30.390399 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:30.390354 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-958jt" podStartSLOduration=2.206036174 podStartE2EDuration="5.390339693s" totalImagesPullingTime="3.184303519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:41:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:41:30.389458915 +0000 UTC m=+616.475796386" watchObservedRunningTime="2026-09-10 18:41:30.390339693 +0000 UTC m=+616.476677162" Sep 10 18:41:30.403363 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:30.403295 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-9rgkh" podStartSLOduration=2.601218323 podStartE2EDuration="6.403275509s" totalImagesPullingTime="3.802057186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:41:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:41:30.401964536 +0000 UTC m=+616.488302011" watchObservedRunningTime="2026-09-10 18:41:30.403275509 +0000 UTC m=+616.489612985" Sep 10 18:41:33.953866 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:33.953830 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:34.008509 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:34.008481 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-mbc8q"] Sep 10 18:41:34.011782 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:34.011761 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mbc8q" Sep 10 18:41:34.013920 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:34.013896 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 10 18:41:34.014038 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:34.013897 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-pqvks\"" Sep 10 18:41:34.022720 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:34.022700 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-pp5xl"] Sep 10 18:41:34.025545 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:34.025530 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pp5xl" Sep 10 18:41:34.028236 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:34.028211 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-mbc8q"] Sep 10 18:41:34.029140 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:34.029120 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 10 18:41:34.029344 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:34.029205 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-fl95b\"" Sep 10 18:41:34.038715 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:34.038696 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-pp5xl"] Sep 10 18:41:34.056230 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:34.056202 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:34.125110 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:34.125085 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b716014-647a-4159-8511-0349dda108e1-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-pp5xl\" (UID: \"5b716014-647a-4159-8511-0349dda108e1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pp5xl" Sep 10 18:41:34.125233 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:34.125117 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8pz7\" (UniqueName: \"kubernetes.io/projected/d2a51711-565b-4fac-8dfd-c1932700f12a-kube-api-access-m8pz7\") pod \"kserve-controller-manager-d6f4b65b8-mbc8q\" (UID: \"d2a51711-565b-4fac-8dfd-c1932700f12a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mbc8q" Sep 10 18:41:34.125233 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:34.125142 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2a51711-565b-4fac-8dfd-c1932700f12a-cert\") pod \"kserve-controller-manager-d6f4b65b8-mbc8q\" (UID: \"d2a51711-565b-4fac-8dfd-c1932700f12a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mbc8q" Sep 10 18:41:34.125233 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:34.125216 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d29bc\" (UniqueName: \"kubernetes.io/projected/5b716014-647a-4159-8511-0349dda108e1-kube-api-access-d29bc\") pod \"llmisvc-controller-manager-777c6bd7d5-pp5xl\" (UID: \"5b716014-647a-4159-8511-0349dda108e1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pp5xl" Sep 10 18:41:34.226385 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:34.226296 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b716014-647a-4159-8511-0349dda108e1-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-pp5xl\" (UID: \"5b716014-647a-4159-8511-0349dda108e1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pp5xl" Sep 10 18:41:34.226385 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:34.226352 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8pz7\" (UniqueName: \"kubernetes.io/projected/d2a51711-565b-4fac-8dfd-c1932700f12a-kube-api-access-m8pz7\") pod \"kserve-controller-manager-d6f4b65b8-mbc8q\" (UID: \"d2a51711-565b-4fac-8dfd-c1932700f12a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mbc8q" Sep 10 18:41:34.226534 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:34.226389 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2a51711-565b-4fac-8dfd-c1932700f12a-cert\") pod \"kserve-controller-manager-d6f4b65b8-mbc8q\" (UID: \"d2a51711-565b-4fac-8dfd-c1932700f12a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mbc8q" Sep 10 18:41:34.226534 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:34.226425 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d29bc\" (UniqueName: \"kubernetes.io/projected/5b716014-647a-4159-8511-0349dda108e1-kube-api-access-d29bc\") pod \"llmisvc-controller-manager-777c6bd7d5-pp5xl\" (UID: \"5b716014-647a-4159-8511-0349dda108e1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pp5xl" Sep 10 18:41:34.226534 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:41:34.226437 2581 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Sep 10 18:41:34.226534 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:41:34.226515 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b716014-647a-4159-8511-0349dda108e1-cert podName:5b716014-647a-4159-8511-0349dda108e1 nodeName:}" failed. No retries permitted until 2026-09-10 18:41:34.726494698 +0000 UTC m=+620.812832174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5b716014-647a-4159-8511-0349dda108e1-cert") pod "llmisvc-controller-manager-777c6bd7d5-pp5xl" (UID: "5b716014-647a-4159-8511-0349dda108e1") : secret "llmisvc-webhook-server-cert" not found Sep 10 18:41:34.226534 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:41:34.226518 2581 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Sep 10 18:41:34.226753 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:41:34.226556 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2a51711-565b-4fac-8dfd-c1932700f12a-cert podName:d2a51711-565b-4fac-8dfd-c1932700f12a nodeName:}" failed. No retries permitted until 2026-09-10 18:41:34.726545763 +0000 UTC m=+620.812883216 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d2a51711-565b-4fac-8dfd-c1932700f12a-cert") pod "kserve-controller-manager-d6f4b65b8-mbc8q" (UID: "d2a51711-565b-4fac-8dfd-c1932700f12a") : secret "kserve-webhook-server-cert" not found Sep 10 18:41:34.241927 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:34.241907 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8pz7\" (UniqueName: \"kubernetes.io/projected/d2a51711-565b-4fac-8dfd-c1932700f12a-kube-api-access-m8pz7\") pod \"kserve-controller-manager-d6f4b65b8-mbc8q\" (UID: \"d2a51711-565b-4fac-8dfd-c1932700f12a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mbc8q" Sep 10 18:41:34.245541 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:34.245515 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d29bc\" (UniqueName: \"kubernetes.io/projected/5b716014-647a-4159-8511-0349dda108e1-kube-api-access-d29bc\") pod \"llmisvc-controller-manager-777c6bd7d5-pp5xl\" (UID: \"5b716014-647a-4159-8511-0349dda108e1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pp5xl" Sep 10 18:41:34.578637 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:34.578604 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:34.706343 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:34.706297 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:34.731580 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:34.731554 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b716014-647a-4159-8511-0349dda108e1-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-pp5xl\" (UID: \"5b716014-647a-4159-8511-0349dda108e1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pp5xl" Sep 10 18:41:34.731700 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:34.731591 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2a51711-565b-4fac-8dfd-c1932700f12a-cert\") pod \"kserve-controller-manager-d6f4b65b8-mbc8q\" (UID: \"d2a51711-565b-4fac-8dfd-c1932700f12a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mbc8q" Sep 10 18:41:34.734172 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:34.734144 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2a51711-565b-4fac-8dfd-c1932700f12a-cert\") pod \"kserve-controller-manager-d6f4b65b8-mbc8q\" (UID: \"d2a51711-565b-4fac-8dfd-c1932700f12a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mbc8q" Sep 10 18:41:34.734275 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:34.734144 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b716014-647a-4159-8511-0349dda108e1-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-pp5xl\" (UID: \"5b716014-647a-4159-8511-0349dda108e1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pp5xl" Sep 10 18:41:34.921545 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:34.921478 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mbc8q" Sep 10 18:41:34.935736 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:34.935715 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pp5xl" Sep 10 18:41:35.201860 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:35.201750 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-mbc8q"] Sep 10 18:41:35.204164 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:41:35.204143 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a51711_565b_4fac_8dfd_c1932700f12a.slice/crio-07e286ca9bb632cc371c2fa719276a24268e4567a41a5b4584b33828b846d617 WatchSource:0}: Error finding container 07e286ca9bb632cc371c2fa719276a24268e4567a41a5b4584b33828b846d617: Status 404 returned error can't find the container with id 07e286ca9bb632cc371c2fa719276a24268e4567a41a5b4584b33828b846d617 Sep 10 18:41:35.221137 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:35.221116 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-pp5xl"] Sep 10 18:41:35.223091 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:41:35.223070 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5b716014_647a_4159_8511_0349dda108e1.slice/crio-187f96ced2c1a6b46167b57d58a7fb695fbc55715e66bd5e0e59d5c7d94fecb8 WatchSource:0}: Error finding container 187f96ced2c1a6b46167b57d58a7fb695fbc55715e66bd5e0e59d5c7d94fecb8: Status 404 returned error can't find the container with id 187f96ced2c1a6b46167b57d58a7fb695fbc55715e66bd5e0e59d5c7d94fecb8 Sep 10 18:41:35.284898 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:35.284873 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:35.351483 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:35.351460 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:35.396163 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:35.396135 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mbc8q" event={"ID":"d2a51711-565b-4fac-8dfd-c1932700f12a","Type":"ContainerStarted","Data":"07e286ca9bb632cc371c2fa719276a24268e4567a41a5b4584b33828b846d617"} Sep 10 18:41:35.397468 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:35.397447 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pp5xl" event={"ID":"5b716014-647a-4159-8511-0349dda108e1","Type":"ContainerStarted","Data":"187f96ced2c1a6b46167b57d58a7fb695fbc55715e66bd5e0e59d5c7d94fecb8"} Sep 10 18:41:36.386960 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:36.386929 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml"] Sep 10 18:41:36.387632 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:36.387163 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml" podUID="5ea8ee98-b277-45e0-b74c-038215891e3c" containerName="discovery" containerID="cri-o://b8abfcdbfcd0890632dba76523126d32a58c351aac76a93faf9d11e6ab960247" gracePeriod=30 Sep 10 18:41:36.673464 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:36.673434 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml" Sep 10 18:41:36.748724 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:36.748688 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6d4p5\" (UniqueName: \"kubernetes.io/projected/5ea8ee98-b277-45e0-b74c-038215891e3c-kube-api-access-6d4p5\") pod \"5ea8ee98-b277-45e0-b74c-038215891e3c\" (UID: \"5ea8ee98-b277-45e0-b74c-038215891e3c\") " Sep 10 18:41:36.748901 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:36.748769 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ea8ee98-b277-45e0-b74c-038215891e3c-local-certs\") pod \"5ea8ee98-b277-45e0-b74c-038215891e3c\" (UID: \"5ea8ee98-b277-45e0-b74c-038215891e3c\") " Sep 10 18:41:36.748901 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:36.748807 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ea8ee98-b277-45e0-b74c-038215891e3c-istio-token\") pod \"5ea8ee98-b277-45e0-b74c-038215891e3c\" (UID: \"5ea8ee98-b277-45e0-b74c-038215891e3c\") " Sep 10 18:41:36.748901 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:36.748835 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5ea8ee98-b277-45e0-b74c-038215891e3c-cacerts\") pod \"5ea8ee98-b277-45e0-b74c-038215891e3c\" (UID: \"5ea8ee98-b277-45e0-b74c-038215891e3c\") " Sep 10 18:41:36.748901 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:36.748873 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5ea8ee98-b277-45e0-b74c-038215891e3c-istio-csr-ca-configmap\") pod \"5ea8ee98-b277-45e0-b74c-038215891e3c\" (UID: \"5ea8ee98-b277-45e0-b74c-038215891e3c\") " Sep 10 18:41:36.749119 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:36.748908 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5ea8ee98-b277-45e0-b74c-038215891e3c-istio-csr-dns-cert\") pod \"5ea8ee98-b277-45e0-b74c-038215891e3c\" (UID: \"5ea8ee98-b277-45e0-b74c-038215891e3c\") " Sep 10 18:41:36.749119 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:36.748940 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5ea8ee98-b277-45e0-b74c-038215891e3c-istio-kubeconfig\") pod \"5ea8ee98-b277-45e0-b74c-038215891e3c\" (UID: \"5ea8ee98-b277-45e0-b74c-038215891e3c\") " Sep 10 18:41:36.749647 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:36.749610 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ea8ee98-b277-45e0-b74c-038215891e3c-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "5ea8ee98-b277-45e0-b74c-038215891e3c" (UID: "5ea8ee98-b277-45e0-b74c-038215891e3c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 18:41:36.751974 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:36.751942 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ea8ee98-b277-45e0-b74c-038215891e3c-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "5ea8ee98-b277-45e0-b74c-038215891e3c" (UID: "5ea8ee98-b277-45e0-b74c-038215891e3c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 18:41:36.752261 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:36.752212 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ea8ee98-b277-45e0-b74c-038215891e3c-local-certs" (OuterVolumeSpecName: "local-certs") pod "5ea8ee98-b277-45e0-b74c-038215891e3c" (UID: "5ea8ee98-b277-45e0-b74c-038215891e3c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 18:41:36.752388 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:36.752298 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ea8ee98-b277-45e0-b74c-038215891e3c-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "5ea8ee98-b277-45e0-b74c-038215891e3c" (UID: "5ea8ee98-b277-45e0-b74c-038215891e3c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 18:41:36.752388 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:36.752315 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ea8ee98-b277-45e0-b74c-038215891e3c-cacerts" (OuterVolumeSpecName: "cacerts") pod "5ea8ee98-b277-45e0-b74c-038215891e3c" (UID: "5ea8ee98-b277-45e0-b74c-038215891e3c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 18:41:36.752751 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:36.752718 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ea8ee98-b277-45e0-b74c-038215891e3c-kube-api-access-6d4p5" (OuterVolumeSpecName: "kube-api-access-6d4p5") pod "5ea8ee98-b277-45e0-b74c-038215891e3c" (UID: "5ea8ee98-b277-45e0-b74c-038215891e3c"). InnerVolumeSpecName "kube-api-access-6d4p5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 18:41:36.752840 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:36.752802 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ea8ee98-b277-45e0-b74c-038215891e3c-istio-token" (OuterVolumeSpecName: "istio-token") pod "5ea8ee98-b277-45e0-b74c-038215891e3c" (UID: "5ea8ee98-b277-45e0-b74c-038215891e3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 18:41:36.827965 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:36.827933 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:36.849793 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:36.849763 2581 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5ea8ee98-b277-45e0-b74c-038215891e3c-istio-csr-dns-cert\") on node \"ip-10-0-134-166.ec2.internal\" DevicePath \"\"" Sep 10 18:41:36.849793 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:36.849791 2581 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5ea8ee98-b277-45e0-b74c-038215891e3c-istio-kubeconfig\") on node \"ip-10-0-134-166.ec2.internal\" DevicePath \"\"" Sep 10 18:41:36.849983 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:36.849805 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d4p5\" (UniqueName: \"kubernetes.io/projected/5ea8ee98-b277-45e0-b74c-038215891e3c-kube-api-access-6d4p5\") on node \"ip-10-0-134-166.ec2.internal\" DevicePath \"\"" Sep 10 18:41:36.849983 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:36.849813 2581 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ea8ee98-b277-45e0-b74c-038215891e3c-local-certs\") on node \"ip-10-0-134-166.ec2.internal\" DevicePath \"\"" Sep 10 18:41:36.849983 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:36.849821 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ea8ee98-b277-45e0-b74c-038215891e3c-istio-token\") on node \"ip-10-0-134-166.ec2.internal\" DevicePath \"\"" Sep 10 18:41:36.849983 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:36.849829 2581 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5ea8ee98-b277-45e0-b74c-038215891e3c-cacerts\") on node \"ip-10-0-134-166.ec2.internal\" DevicePath \"\"" Sep 10 18:41:36.849983 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:36.849836 2581 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5ea8ee98-b277-45e0-b74c-038215891e3c-istio-csr-ca-configmap\") on node \"ip-10-0-134-166.ec2.internal\" DevicePath \"\"" Sep 10 18:41:36.933811 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:36.933725 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:41:37.410749 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:37.410720 2581 generic.go:356] "Generic (PLEG): container finished" podID="5ea8ee98-b277-45e0-b74c-038215891e3c" containerID="b8abfcdbfcd0890632dba76523126d32a58c351aac76a93faf9d11e6ab960247" exitCode=0 Sep 10 18:41:37.411190 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:37.410779 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml" Sep 10 18:41:37.411190 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:37.410784 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml" event={"ID":"5ea8ee98-b277-45e0-b74c-038215891e3c","Type":"ContainerDied","Data":"b8abfcdbfcd0890632dba76523126d32a58c351aac76a93faf9d11e6ab960247"} Sep 10 18:41:37.411190 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:37.410832 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml" event={"ID":"5ea8ee98-b277-45e0-b74c-038215891e3c","Type":"ContainerDied","Data":"f4e3b0715fd9806ec4d4746e0941216b29aeab462d78d4abdc0160de3951c62c"} Sep 10 18:41:37.411190 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:37.410862 2581 scope.go:117] "RemoveContainer" containerID="b8abfcdbfcd0890632dba76523126d32a58c351aac76a93faf9d11e6ab960247" Sep 10 18:41:37.432495 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:37.432472 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml"] Sep 10 18:41:37.439059 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:37.439035 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bnrml"] Sep 10 18:41:37.903802 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:37.903773 2581 scope.go:117] "RemoveContainer" containerID="b8abfcdbfcd0890632dba76523126d32a58c351aac76a93faf9d11e6ab960247" Sep 10 18:41:37.904198 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:41:37.904173 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8abfcdbfcd0890632dba76523126d32a58c351aac76a93faf9d11e6ab960247\": container with ID starting with b8abfcdbfcd0890632dba76523126d32a58c351aac76a93faf9d11e6ab960247 not found: ID does not exist" containerID="b8abfcdbfcd0890632dba76523126d32a58c351aac76a93faf9d11e6ab960247" Sep 10 18:41:37.904286 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:37.904208 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8abfcdbfcd0890632dba76523126d32a58c351aac76a93faf9d11e6ab960247"} err="failed to get container status \"b8abfcdbfcd0890632dba76523126d32a58c351aac76a93faf9d11e6ab960247\": rpc error: code = NotFound desc = could not find container \"b8abfcdbfcd0890632dba76523126d32a58c351aac76a93faf9d11e6ab960247\": container with ID starting with b8abfcdbfcd0890632dba76523126d32a58c351aac76a93faf9d11e6ab960247 not found: ID does not exist" Sep 10 18:41:38.380798 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:38.380743 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ea8ee98-b277-45e0-b74c-038215891e3c" path="/var/lib/kubelet/pods/5ea8ee98-b277-45e0-b74c-038215891e3c/volumes" Sep 10 18:41:38.416978 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:38.416945 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mbc8q" event={"ID":"d2a51711-565b-4fac-8dfd-c1932700f12a","Type":"ContainerStarted","Data":"d35def531ed896fb825267004e2d8a22ede5431cb273ba8d6d867782b3cb1f72"} Sep 10 18:41:38.417465 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:38.417051 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mbc8q" Sep 10 18:41:38.438347 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:38.438279 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mbc8q" podStartSLOduration=2.695178048 podStartE2EDuration="5.438258194s" totalImagesPullingTime="2.743080146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:41:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:41:38.437278102 +0000 UTC m=+624.523615577" watchObservedRunningTime="2026-09-10 18:41:38.438258194 +0000 UTC m=+624.524595672" Sep 10 18:41:39.423931 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:39.423897 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pp5xl" event={"ID":"5b716014-647a-4159-8511-0349dda108e1","Type":"ContainerStarted","Data":"0d967c60a9c2312e43ad0e3cd8880fabb4c3647f6d1ff7fce77cdfa0fd99483c"} Sep 10 18:41:39.424387 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:39.424189 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pp5xl" Sep 10 18:41:39.438195 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:39.438148 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pp5xl" podStartSLOduration=1.996795332 podStartE2EDuration="5.438134876s" totalImagesPullingTime="3.441339544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:41:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:41:39.437353346 +0000 UTC m=+625.523690822" watchObservedRunningTime="2026-09-10 18:41:39.438134876 +0000 UTC m=+625.524472350" Sep 10 18:41:41.383260 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:41.383223 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-9rgkh" Sep 10 18:41:41.384930 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:41:41.384908 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-958jt" Sep 10 18:42:09.429535 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:09.429500 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mbc8q" Sep 10 18:42:10.429874 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:10.429843 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pp5xl" Sep 10 18:42:25.778842 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:25.778805 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:42:25.876999 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:25.876966 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:42:37.979399 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:37.979362 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:42:38.775760 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:38.775733 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:42:39.578521 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:39.578489 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:42:42.318628 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:42.318598 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gzjvf"] Sep 10 18:42:42.319066 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:42.318871 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ea8ee98-b277-45e0-b74c-038215891e3c" containerName="discovery" Sep 10 18:42:42.319066 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:42.318883 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ea8ee98-b277-45e0-b74c-038215891e3c" containerName="discovery" Sep 10 18:42:42.319066 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:42.318928 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ea8ee98-b277-45e0-b74c-038215891e3c" containerName="discovery" Sep 10 18:42:42.321768 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:42.321753 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gzjvf" Sep 10 18:42:42.324070 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:42.324047 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 10 18:42:42.324208 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:42.324109 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 10 18:42:42.324208 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:42.324181 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-n4z5k\"" Sep 10 18:42:42.334488 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:42.334468 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gzjvf"] Sep 10 18:42:42.416990 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:42.416962 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1933a925-2316-41ae-9639-0f74c43e39a6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gzjvf\" (UID: \"1933a925-2316-41ae-9639-0f74c43e39a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gzjvf" Sep 10 18:42:42.417113 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:42.417005 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klpxl\" (UniqueName: \"kubernetes.io/projected/1933a925-2316-41ae-9639-0f74c43e39a6-kube-api-access-klpxl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gzjvf\" (UID: \"1933a925-2316-41ae-9639-0f74c43e39a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gzjvf" Sep 10 18:42:42.517621 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:42.517596 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klpxl\" (UniqueName: \"kubernetes.io/projected/1933a925-2316-41ae-9639-0f74c43e39a6-kube-api-access-klpxl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gzjvf\" (UID: \"1933a925-2316-41ae-9639-0f74c43e39a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gzjvf" Sep 10 18:42:42.517719 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:42.517681 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1933a925-2316-41ae-9639-0f74c43e39a6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gzjvf\" (UID: \"1933a925-2316-41ae-9639-0f74c43e39a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gzjvf" Sep 10 18:42:42.518089 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:42.518069 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1933a925-2316-41ae-9639-0f74c43e39a6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gzjvf\" (UID: \"1933a925-2316-41ae-9639-0f74c43e39a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gzjvf" Sep 10 18:42:42.526274 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:42.526255 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klpxl\" (UniqueName: \"kubernetes.io/projected/1933a925-2316-41ae-9639-0f74c43e39a6-kube-api-access-klpxl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gzjvf\" (UID: \"1933a925-2316-41ae-9639-0f74c43e39a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gzjvf" Sep 10 18:42:42.579300 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:42.579207 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:42:42.631457 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:42.631432 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gzjvf" Sep 10 18:42:42.861474 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:42.861387 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:42:42.926404 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:42.926375 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gzjvf"] Sep 10 18:42:42.929527 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:42:42.929504 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1933a925_2316_41ae_9639_0f74c43e39a6.slice/crio-0aef097ac51c9647ca926df1fb5b945236d4fe83df59d8303e3c04e81f744452 WatchSource:0}: Error finding container 0aef097ac51c9647ca926df1fb5b945236d4fe83df59d8303e3c04e81f744452: Status 404 returned error can't find the container with id 0aef097ac51c9647ca926df1fb5b945236d4fe83df59d8303e3c04e81f744452 Sep 10 18:42:42.932084 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:42.932068 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 18:42:43.003261 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:43.003203 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:42:43.102283 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:43.102248 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:42:43.665517 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:43.665473 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gzjvf" event={"ID":"1933a925-2316-41ae-9639-0f74c43e39a6","Type":"ContainerStarted","Data":"0aef097ac51c9647ca926df1fb5b945236d4fe83df59d8303e3c04e81f744452"} Sep 10 18:42:44.423560 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:44.423524 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:42:44.524759 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:44.524724 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:42:44.749012 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:44.748930 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-6hl2d"] Sep 10 18:42:44.758780 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:44.758413 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-6hl2d" Sep 10 18:42:44.761544 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:44.761477 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-6hl2d"] Sep 10 18:42:44.761833 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:44.761811 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-cgwj6\"" Sep 10 18:42:44.839858 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:44.839818 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w4cm\" (UniqueName: \"kubernetes.io/projected/df9e3b01-9857-4103-96ce-39a3da42e7d0-kube-api-access-9w4cm\") pod \"authorino-operator-76b478bd78-6hl2d\" (UID: \"df9e3b01-9857-4103-96ce-39a3da42e7d0\") " pod="kuadrant-system/authorino-operator-76b478bd78-6hl2d" Sep 10 18:42:44.941302 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:44.941265 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w4cm\" (UniqueName: \"kubernetes.io/projected/df9e3b01-9857-4103-96ce-39a3da42e7d0-kube-api-access-9w4cm\") pod \"authorino-operator-76b478bd78-6hl2d\" (UID: \"df9e3b01-9857-4103-96ce-39a3da42e7d0\") " pod="kuadrant-system/authorino-operator-76b478bd78-6hl2d" Sep 10 18:42:44.961681 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:44.961646 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w4cm\" (UniqueName: \"kubernetes.io/projected/df9e3b01-9857-4103-96ce-39a3da42e7d0-kube-api-access-9w4cm\") pod \"authorino-operator-76b478bd78-6hl2d\" (UID: \"df9e3b01-9857-4103-96ce-39a3da42e7d0\") " pod="kuadrant-system/authorino-operator-76b478bd78-6hl2d" Sep 10 18:42:45.072741 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:45.072703 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-6hl2d" Sep 10 18:42:45.554694 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:45.554658 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-6hl2d"] Sep 10 18:42:45.559490 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:42:45.559454 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf9e3b01_9857_4103_96ce_39a3da42e7d0.slice/crio-9d84bc5ad631e70150a8b13dcef48ddab53dae38187d08d83907af451658c112 WatchSource:0}: Error finding container 9d84bc5ad631e70150a8b13dcef48ddab53dae38187d08d83907af451658c112: Status 404 returned error can't find the container with id 9d84bc5ad631e70150a8b13dcef48ddab53dae38187d08d83907af451658c112 Sep 10 18:42:45.609034 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:45.608944 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:42:45.680969 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:45.680931 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-6hl2d" event={"ID":"df9e3b01-9857-4103-96ce-39a3da42e7d0","Type":"ContainerStarted","Data":"9d84bc5ad631e70150a8b13dcef48ddab53dae38187d08d83907af451658c112"} Sep 10 18:42:45.828894 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:45.828845 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:42:47.168475 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:47.168442 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:42:47.198783 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:47.198744 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9bf85"] Sep 10 18:42:47.202012 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:47.201985 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9bf85" Sep 10 18:42:47.204072 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:47.204048 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 10 18:42:47.204175 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:47.204068 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-nh7nv\"" Sep 10 18:42:47.211772 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:47.211750 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9bf85"] Sep 10 18:42:47.261951 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:47.261867 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l8x8\" (UniqueName: \"kubernetes.io/projected/674fa0a2-179a-47c9-b8d8-c7e1e947e93b-kube-api-access-5l8x8\") pod \"dns-operator-controller-manager-7b84cd7c9c-9bf85\" (UID: \"674fa0a2-179a-47c9-b8d8-c7e1e947e93b\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9bf85" Sep 10 18:42:47.362918 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:47.362886 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l8x8\" (UniqueName: \"kubernetes.io/projected/674fa0a2-179a-47c9-b8d8-c7e1e947e93b-kube-api-access-5l8x8\") pod \"dns-operator-controller-manager-7b84cd7c9c-9bf85\" (UID: \"674fa0a2-179a-47c9-b8d8-c7e1e947e93b\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9bf85" Sep 10 18:42:47.373618 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:47.373597 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l8x8\" (UniqueName: \"kubernetes.io/projected/674fa0a2-179a-47c9-b8d8-c7e1e947e93b-kube-api-access-5l8x8\") pod \"dns-operator-controller-manager-7b84cd7c9c-9bf85\" (UID: \"674fa0a2-179a-47c9-b8d8-c7e1e947e93b\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9bf85" Sep 10 18:42:47.514650 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:47.514570 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9bf85" Sep 10 18:42:47.691971 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:47.691942 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gzjvf" event={"ID":"1933a925-2316-41ae-9639-0f74c43e39a6","Type":"ContainerStarted","Data":"77ecef1c898fc8e73956e1e02664fdae6c5b9b6f5ad94a35a9b3fa6ce1888e09"} Sep 10 18:42:47.692138 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:47.692081 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gzjvf" Sep 10 18:42:47.712820 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:47.712768 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gzjvf" podStartSLOduration=1.647766861 podStartE2EDuration="5.712753486s" totalImagesPullingTime="4.064986625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:42:47.712096705 +0000 UTC m=+693.798434172" watchObservedRunningTime="2026-09-10 18:42:47.712753486 +0000 UTC m=+693.799090960" Sep 10 18:42:47.784869 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:47.784723 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9bf85"] Sep 10 18:42:47.785228 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:47.785204 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:42:47.787521 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:42:47.787494 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod674fa0a2_179a_47c9_b8d8_c7e1e947e93b.slice/crio-e37ef375658ca56808f262adfdb2a3da7860b3510644696222dac54778b37df3 WatchSource:0}: Error finding container e37ef375658ca56808f262adfdb2a3da7860b3510644696222dac54778b37df3: Status 404 returned error can't find the container with id e37ef375658ca56808f262adfdb2a3da7860b3510644696222dac54778b37df3 Sep 10 18:42:48.699140 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:48.699101 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9bf85" event={"ID":"674fa0a2-179a-47c9-b8d8-c7e1e947e93b","Type":"ContainerStarted","Data":"e37ef375658ca56808f262adfdb2a3da7860b3510644696222dac54778b37df3"} Sep 10 18:42:49.199828 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:49.199801 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5v6zs"] Sep 10 18:42:49.203097 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:49.203079 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5v6zs" Sep 10 18:42:49.205202 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:49.205183 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-csw6d\"" Sep 10 18:42:49.212644 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:49.212454 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5v6zs"] Sep 10 18:42:49.280224 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:49.280182 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwtxf\" (UniqueName: \"kubernetes.io/projected/82ee372f-bd75-483d-8474-5126dcada0aa-kube-api-access-rwtxf\") pod \"limitador-operator-controller-manager-74495c69f8-5v6zs\" (UID: \"82ee372f-bd75-483d-8474-5126dcada0aa\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5v6zs" Sep 10 18:42:49.380626 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:49.380588 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwtxf\" (UniqueName: \"kubernetes.io/projected/82ee372f-bd75-483d-8474-5126dcada0aa-kube-api-access-rwtxf\") pod \"limitador-operator-controller-manager-74495c69f8-5v6zs\" (UID: \"82ee372f-bd75-483d-8474-5126dcada0aa\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5v6zs" Sep 10 18:42:49.391249 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:49.391219 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwtxf\" (UniqueName: \"kubernetes.io/projected/82ee372f-bd75-483d-8474-5126dcada0aa-kube-api-access-rwtxf\") pod \"limitador-operator-controller-manager-74495c69f8-5v6zs\" (UID: \"82ee372f-bd75-483d-8474-5126dcada0aa\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5v6zs" Sep 10 18:42:49.522681 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:49.522597 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5v6zs" Sep 10 18:42:49.616699 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:49.616672 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:42:49.704849 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:49.704810 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9bf85" event={"ID":"674fa0a2-179a-47c9-b8d8-c7e1e947e93b","Type":"ContainerStarted","Data":"b42361f074a24b3868487ca223fc2f7312324625a79900f285ece68b2bec956c"} Sep 10 18:42:49.705226 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:49.704898 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9bf85" Sep 10 18:42:49.718987 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:49.718938 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9bf85" podStartSLOduration=1.314412444 podStartE2EDuration="2.718923975s" totalImagesPullingTime="1.404511531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:42:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:42:49.71811552 +0000 UTC m=+695.804452986" watchObservedRunningTime="2026-09-10 18:42:49.718923975 +0000 UTC m=+695.805261450" Sep 10 18:42:49.797170 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:49.797145 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5v6zs"] Sep 10 18:42:49.800234 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:42:49.800209 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ee372f_bd75_483d_8474_5126dcada0aa.slice/crio-1dbc5add676449c935f9610711bbf62882ef3b73f32fa9c0129e188c0d97ed1d WatchSource:0}: Error finding container 1dbc5add676449c935f9610711bbf62882ef3b73f32fa9c0129e188c0d97ed1d: Status 404 returned error can't find the container with id 1dbc5add676449c935f9610711bbf62882ef3b73f32fa9c0129e188c0d97ed1d Sep 10 18:42:49.974884 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:49.974855 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-8sp4k"] Sep 10 18:42:49.979385 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:49.979362 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8sp4k" Sep 10 18:42:49.981509 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:49.981470 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Sep 10 18:42:49.981606 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:49.981481 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Sep 10 18:42:49.981669 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:49.981603 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4zxlc\"" Sep 10 18:42:49.989284 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:49.989260 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-8sp4k"] Sep 10 18:42:49.990524 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:49.990502 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:42:50.087654 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:50.087578 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/57122aa5-9906-4e29-a8b6-0424da9fceb7-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-8sp4k\" (UID: \"57122aa5-9906-4e29-a8b6-0424da9fceb7\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8sp4k" Sep 10 18:42:50.087654 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:50.087615 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/57122aa5-9906-4e29-a8b6-0424da9fceb7-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-8sp4k\" (UID: \"57122aa5-9906-4e29-a8b6-0424da9fceb7\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8sp4k" Sep 10 18:42:50.087654 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:50.087646 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfcld\" (UniqueName: \"kubernetes.io/projected/57122aa5-9906-4e29-a8b6-0424da9fceb7-kube-api-access-sfcld\") pod \"kuadrant-console-plugin-fbdd4f445-8sp4k\" (UID: \"57122aa5-9906-4e29-a8b6-0424da9fceb7\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8sp4k" Sep 10 18:42:50.188563 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:50.188532 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/57122aa5-9906-4e29-a8b6-0424da9fceb7-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-8sp4k\" (UID: \"57122aa5-9906-4e29-a8b6-0424da9fceb7\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8sp4k" Sep 10 18:42:50.188723 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:50.188574 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/57122aa5-9906-4e29-a8b6-0424da9fceb7-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-8sp4k\" (UID: \"57122aa5-9906-4e29-a8b6-0424da9fceb7\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8sp4k" Sep 10 18:42:50.188723 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:50.188617 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfcld\" (UniqueName: \"kubernetes.io/projected/57122aa5-9906-4e29-a8b6-0424da9fceb7-kube-api-access-sfcld\") pod \"kuadrant-console-plugin-fbdd4f445-8sp4k\" (UID: \"57122aa5-9906-4e29-a8b6-0424da9fceb7\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8sp4k" Sep 10 18:42:50.188723 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:42:50.188686 2581 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Sep 10 18:42:50.188846 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:42:50.188768 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57122aa5-9906-4e29-a8b6-0424da9fceb7-plugin-serving-cert podName:57122aa5-9906-4e29-a8b6-0424da9fceb7 nodeName:}" failed. No retries permitted until 2026-09-10 18:42:50.688747903 +0000 UTC m=+696.775085359 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/57122aa5-9906-4e29-a8b6-0424da9fceb7-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-8sp4k" (UID: "57122aa5-9906-4e29-a8b6-0424da9fceb7") : secret "plugin-serving-cert" not found Sep 10 18:42:50.189136 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:50.189116 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/57122aa5-9906-4e29-a8b6-0424da9fceb7-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-8sp4k\" (UID: \"57122aa5-9906-4e29-a8b6-0424da9fceb7\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8sp4k" Sep 10 18:42:50.196195 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:50.196170 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfcld\" (UniqueName: \"kubernetes.io/projected/57122aa5-9906-4e29-a8b6-0424da9fceb7-kube-api-access-sfcld\") pod \"kuadrant-console-plugin-fbdd4f445-8sp4k\" (UID: \"57122aa5-9906-4e29-a8b6-0424da9fceb7\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8sp4k" Sep 10 18:42:50.693081 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:50.693045 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/57122aa5-9906-4e29-a8b6-0424da9fceb7-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-8sp4k\" (UID: \"57122aa5-9906-4e29-a8b6-0424da9fceb7\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8sp4k" Sep 10 18:42:50.695532 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:50.695507 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/57122aa5-9906-4e29-a8b6-0424da9fceb7-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-8sp4k\" (UID: \"57122aa5-9906-4e29-a8b6-0424da9fceb7\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8sp4k" Sep 10 18:42:50.711733 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:50.711698 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5v6zs" event={"ID":"82ee372f-bd75-483d-8474-5126dcada0aa","Type":"ContainerStarted","Data":"1dbc5add676449c935f9610711bbf62882ef3b73f32fa9c0129e188c0d97ed1d"} Sep 10 18:42:50.736822 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:50.736789 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:42:50.890267 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:50.890235 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8sp4k" Sep 10 18:42:51.254707 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:51.254624 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-8sp4k"] Sep 10 18:42:51.257626 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:42:51.257601 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57122aa5_9906_4e29_a8b6_0424da9fceb7.slice/crio-a69a2e58780ed35df86c5503e02fe6ae318f479529f3dcae7d9cfdc335edd541 WatchSource:0}: Error finding container a69a2e58780ed35df86c5503e02fe6ae318f479529f3dcae7d9cfdc335edd541: Status 404 returned error can't find the container with id a69a2e58780ed35df86c5503e02fe6ae318f479529f3dcae7d9cfdc335edd541 Sep 10 18:42:51.716871 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:51.716837 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8sp4k" event={"ID":"57122aa5-9906-4e29-a8b6-0424da9fceb7","Type":"ContainerStarted","Data":"a69a2e58780ed35df86c5503e02fe6ae318f479529f3dcae7d9cfdc335edd541"} Sep 10 18:42:51.718317 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:51.718295 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5v6zs" event={"ID":"82ee372f-bd75-483d-8474-5126dcada0aa","Type":"ContainerStarted","Data":"483d53a9975b41028318c38b9d2144c09e66d14341c4455d06424d47733af04c"} Sep 10 18:42:51.718483 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:51.718443 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5v6zs" Sep 10 18:42:51.733194 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:51.733149 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5v6zs" podStartSLOduration=1.482087431 podStartE2EDuration="2.733130981s" totalImagesPullingTime="1.25104355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:42:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:42:51.73205038 +0000 UTC m=+697.818387855" watchObservedRunningTime="2026-09-10 18:42:51.733130981 +0000 UTC m=+697.819468457" Sep 10 18:42:52.984646 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:52.984610 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:42:53.084776 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:53.084705 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:42:53.239856 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:53.239781 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:42:53.382225 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:53.382192 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:42:53.483399 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:53.483361 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:42:53.787334 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:53.787299 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:42:53.888041 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:53.887993 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:42:56.744061 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:56.744024 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8sp4k" event={"ID":"57122aa5-9906-4e29-a8b6-0424da9fceb7","Type":"ContainerStarted","Data":"0980a7d2513ebb6b13108cbf9fcfb8ac758640caffc848c8212a39f370b4a001"} Sep 10 18:42:56.762310 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:56.762230 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8sp4k" podStartSLOduration=2.518035512 podStartE2EDuration="7.762214468s" totalImagesPullingTime="5.244178956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:42:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:42:56.760417103 +0000 UTC m=+702.846754578" watchObservedRunningTime="2026-09-10 18:42:56.762214468 +0000 UTC m=+702.848551942" Sep 10 18:42:57.750504 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:57.750403 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-6hl2d" event={"ID":"df9e3b01-9857-4103-96ce-39a3da42e7d0","Type":"ContainerStarted","Data":"9cb71767d00bb2af02599789641e4dad14571df528b44ed0406155bfc471556d"} Sep 10 18:42:57.750843 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:57.750641 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-6hl2d" Sep 10 18:42:57.769659 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:57.769613 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-6hl2d" podStartSLOduration=1.961402731 podStartE2EDuration="13.769598029s" totalImagesPullingTime="11.808195298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:42:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:42:57.767877912 +0000 UTC m=+703.854215388" watchObservedRunningTime="2026-09-10 18:42:57.769598029 +0000 UTC m=+703.855935504" Sep 10 18:42:58.701746 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:42:58.701717 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gzjvf" Sep 10 18:43:00.714814 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:00.714787 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9bf85" Sep 10 18:43:00.777429 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:00.777402 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gzjvf"] Sep 10 18:43:00.777600 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:00.777575 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gzjvf" podUID="1933a925-2316-41ae-9639-0f74c43e39a6" containerName="manager" containerID="cri-o://77ecef1c898fc8e73956e1e02664fdae6c5b9b6f5ad94a35a9b3fa6ce1888e09" gracePeriod=2 Sep 10 18:43:00.781949 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:00.781919 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gzjvf"] Sep 10 18:43:00.790673 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:00.790648 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5v6zs"] Sep 10 18:43:00.790900 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:00.790873 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5v6zs" podUID="82ee372f-bd75-483d-8474-5126dcada0aa" containerName="manager" containerID="cri-o://483d53a9975b41028318c38b9d2144c09e66d14341c4455d06424d47733af04c" gracePeriod=2 Sep 10 18:43:00.792990 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:43:00.792903 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-gzjvf\" is forbidden: User \"system:node:ip-10-0-134-166.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-166.ec2.internal' and this object" podUID="1933a925-2316-41ae-9639-0f74c43e39a6" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gzjvf" Sep 10 18:43:00.793971 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:00.793917 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5v6zs" Sep 10 18:43:00.798089 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:00.798064 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5v6zs"] Sep 10 18:43:00.802532 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:43:00.802497 2581 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"82ee372f-bd75-483d-8474-5126dcada0aa\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-74495c69f8-5v6zs\": pods \"limitador-operator-controller-manager-74495c69f8-5v6zs\" not found" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5v6zs" Sep 10 18:43:00.804668 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:43:00.804632 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-gzjvf\" is forbidden: User \"system:node:ip-10-0-134-166.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-166.ec2.internal' and this object" podUID="1933a925-2316-41ae-9639-0f74c43e39a6" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gzjvf" Sep 10 18:43:00.805095 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:00.805075 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g6c6n"] Sep 10 18:43:00.805390 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:00.805370 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1933a925-2316-41ae-9639-0f74c43e39a6" containerName="manager" Sep 10 18:43:00.805390 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:00.805391 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1933a925-2316-41ae-9639-0f74c43e39a6" containerName="manager" Sep 10 18:43:00.805508 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:00.805419 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82ee372f-bd75-483d-8474-5126dcada0aa" containerName="manager" Sep 10 18:43:00.805508 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:00.805428 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="82ee372f-bd75-483d-8474-5126dcada0aa" containerName="manager" Sep 10 18:43:00.805508 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:00.805501 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="82ee372f-bd75-483d-8474-5126dcada0aa" containerName="manager" Sep 10 18:43:00.805648 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:00.805515 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1933a925-2316-41ae-9639-0f74c43e39a6" containerName="manager" Sep 10 18:43:00.806053 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:43:00.806018 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5v6zs\" is forbidden: User \"system:node:ip-10-0-134-166.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-166.ec2.internal' and this object" podUID="82ee372f-bd75-483d-8474-5126dcada0aa" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5v6zs" Sep 10 18:43:00.808347 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:00.808315 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g6c6n" Sep 10 18:43:00.809848 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:43:00.809821 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-gzjvf\" is forbidden: User \"system:node:ip-10-0-134-166.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-166.ec2.internal' and this object" podUID="1933a925-2316-41ae-9639-0f74c43e39a6" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gzjvf" Sep 10 18:43:00.811214 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:43:00.811191 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5v6zs\" is forbidden: User \"system:node:ip-10-0-134-166.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-166.ec2.internal' and this object" podUID="82ee372f-bd75-483d-8474-5126dcada0aa" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5v6zs" Sep 10 18:43:00.812818 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:00.812796 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-ptlsk"] Sep 10 18:43:00.816171 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:00.816155 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ptlsk" Sep 10 18:43:00.822593 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:00.822575 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g6c6n"] Sep 10 18:43:00.828658 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:00.828602 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-ptlsk"] Sep 10 18:43:00.842391 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:43:00.842363 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-gzjvf\" is forbidden: User \"system:node:ip-10-0-134-166.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-166.ec2.internal' and this object" podUID="1933a925-2316-41ae-9639-0f74c43e39a6" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gzjvf" Sep 10 18:43:00.844004 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:43:00.843975 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5v6zs\" is forbidden: User \"system:node:ip-10-0-134-166.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-166.ec2.internal' and this object" podUID="82ee372f-bd75-483d-8474-5126dcada0aa" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5v6zs" Sep 10 18:43:00.988984 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:00.988905 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6de0670d-8028-43bb-9407-4d4096ff97d2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-g6c6n\" (UID: \"6de0670d-8028-43bb-9407-4d4096ff97d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g6c6n" Sep 10 18:43:00.988984 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:00.988971 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrf6q\" (UniqueName: \"kubernetes.io/projected/6de0670d-8028-43bb-9407-4d4096ff97d2-kube-api-access-lrf6q\") pod \"kuadrant-operator-controller-manager-76f8f686bf-g6c6n\" (UID: \"6de0670d-8028-43bb-9407-4d4096ff97d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g6c6n" Sep 10 18:43:00.989137 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:00.989011 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj4hq\" (UniqueName: \"kubernetes.io/projected/3d15dbde-9c7a-4f40-871f-33179a38f87b-kube-api-access-zj4hq\") pod \"limitador-operator-controller-manager-74495c69f8-ptlsk\" (UID: \"3d15dbde-9c7a-4f40-871f-33179a38f87b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ptlsk" Sep 10 18:43:01.029936 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.029908 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gzjvf" Sep 10 18:43:01.031696 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:43:01.031670 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-gzjvf\" is forbidden: User \"system:node:ip-10-0-134-166.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-166.ec2.internal' and this object" podUID="1933a925-2316-41ae-9639-0f74c43e39a6" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gzjvf" Sep 10 18:43:01.032567 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.032550 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5v6zs" Sep 10 18:43:01.033260 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:43:01.033239 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5v6zs\" is forbidden: User \"system:node:ip-10-0-134-166.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-166.ec2.internal' and this object" podUID="82ee372f-bd75-483d-8474-5126dcada0aa" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5v6zs" Sep 10 18:43:01.036444 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:43:01.036423 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-gzjvf\" is forbidden: User \"system:node:ip-10-0-134-166.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-166.ec2.internal' and this object" podUID="1933a925-2316-41ae-9639-0f74c43e39a6" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gzjvf" Sep 10 18:43:01.037945 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:43:01.037928 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5v6zs\" is forbidden: User \"system:node:ip-10-0-134-166.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-166.ec2.internal' and this object" podUID="82ee372f-bd75-483d-8474-5126dcada0aa" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5v6zs" Sep 10 18:43:01.089624 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.089604 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj4hq\" (UniqueName: \"kubernetes.io/projected/3d15dbde-9c7a-4f40-871f-33179a38f87b-kube-api-access-zj4hq\") pod \"limitador-operator-controller-manager-74495c69f8-ptlsk\" (UID: \"3d15dbde-9c7a-4f40-871f-33179a38f87b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ptlsk" Sep 10 18:43:01.089703 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.089663 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6de0670d-8028-43bb-9407-4d4096ff97d2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-g6c6n\" (UID: \"6de0670d-8028-43bb-9407-4d4096ff97d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g6c6n" Sep 10 18:43:01.089748 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.089704 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrf6q\" (UniqueName: \"kubernetes.io/projected/6de0670d-8028-43bb-9407-4d4096ff97d2-kube-api-access-lrf6q\") pod \"kuadrant-operator-controller-manager-76f8f686bf-g6c6n\" (UID: \"6de0670d-8028-43bb-9407-4d4096ff97d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g6c6n" Sep 10 18:43:01.090048 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.090030 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6de0670d-8028-43bb-9407-4d4096ff97d2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-g6c6n\" (UID: \"6de0670d-8028-43bb-9407-4d4096ff97d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g6c6n" Sep 10 18:43:01.099831 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.099813 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrf6q\" (UniqueName: \"kubernetes.io/projected/6de0670d-8028-43bb-9407-4d4096ff97d2-kube-api-access-lrf6q\") pod \"kuadrant-operator-controller-manager-76f8f686bf-g6c6n\" (UID: \"6de0670d-8028-43bb-9407-4d4096ff97d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g6c6n" Sep 10 18:43:01.099931 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.099914 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj4hq\" (UniqueName: \"kubernetes.io/projected/3d15dbde-9c7a-4f40-871f-33179a38f87b-kube-api-access-zj4hq\") pod \"limitador-operator-controller-manager-74495c69f8-ptlsk\" (UID: \"3d15dbde-9c7a-4f40-871f-33179a38f87b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ptlsk" Sep 10 18:43:01.190153 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.190134 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klpxl\" (UniqueName: \"kubernetes.io/projected/1933a925-2316-41ae-9639-0f74c43e39a6-kube-api-access-klpxl\") pod \"1933a925-2316-41ae-9639-0f74c43e39a6\" (UID: \"1933a925-2316-41ae-9639-0f74c43e39a6\") " Sep 10 18:43:01.190249 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.190180 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwtxf\" (UniqueName: \"kubernetes.io/projected/82ee372f-bd75-483d-8474-5126dcada0aa-kube-api-access-rwtxf\") pod \"82ee372f-bd75-483d-8474-5126dcada0aa\" (UID: \"82ee372f-bd75-483d-8474-5126dcada0aa\") " Sep 10 18:43:01.190249 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.190215 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1933a925-2316-41ae-9639-0f74c43e39a6-extensions-socket-volume\") pod \"1933a925-2316-41ae-9639-0f74c43e39a6\" (UID: \"1933a925-2316-41ae-9639-0f74c43e39a6\") " Sep 10 18:43:01.190716 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.190695 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1933a925-2316-41ae-9639-0f74c43e39a6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1933a925-2316-41ae-9639-0f74c43e39a6" (UID: "1933a925-2316-41ae-9639-0f74c43e39a6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 18:43:01.192210 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.192188 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1933a925-2316-41ae-9639-0f74c43e39a6-kube-api-access-klpxl" (OuterVolumeSpecName: "kube-api-access-klpxl") pod "1933a925-2316-41ae-9639-0f74c43e39a6" (UID: "1933a925-2316-41ae-9639-0f74c43e39a6"). InnerVolumeSpecName "kube-api-access-klpxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 18:43:01.192342 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.192308 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ee372f-bd75-483d-8474-5126dcada0aa-kube-api-access-rwtxf" (OuterVolumeSpecName: "kube-api-access-rwtxf") pod "82ee372f-bd75-483d-8474-5126dcada0aa" (UID: "82ee372f-bd75-483d-8474-5126dcada0aa"). InnerVolumeSpecName "kube-api-access-rwtxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 18:43:01.216700 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.216676 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g6c6n" Sep 10 18:43:01.223409 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.223392 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ptlsk" Sep 10 18:43:01.291203 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.291150 2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1933a925-2316-41ae-9639-0f74c43e39a6-extensions-socket-volume\") on node \"ip-10-0-134-166.ec2.internal\" DevicePath \"\"" Sep 10 18:43:01.291203 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.291173 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klpxl\" (UniqueName: \"kubernetes.io/projected/1933a925-2316-41ae-9639-0f74c43e39a6-kube-api-access-klpxl\") on node \"ip-10-0-134-166.ec2.internal\" DevicePath \"\"" Sep 10 18:43:01.291203 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.291183 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwtxf\" (UniqueName: \"kubernetes.io/projected/82ee372f-bd75-483d-8474-5126dcada0aa-kube-api-access-rwtxf\") on node \"ip-10-0-134-166.ec2.internal\" DevicePath \"\"" Sep 10 18:43:01.497157 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.497135 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g6c6n"] Sep 10 18:43:01.500014 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:43:01.499991 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6de0670d_8028_43bb_9407_4d4096ff97d2.slice/crio-9968c0c6cf25a3a5b90f6102deaebdad9d2f46e1312dc93d4bb731528c152f49 WatchSource:0}: Error finding container 9968c0c6cf25a3a5b90f6102deaebdad9d2f46e1312dc93d4bb731528c152f49: Status 404 returned error can't find the container with id 9968c0c6cf25a3a5b90f6102deaebdad9d2f46e1312dc93d4bb731528c152f49 Sep 10 18:43:01.505683 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.505662 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-ptlsk"] Sep 10 18:43:01.508256 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:43:01.508236 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d15dbde_9c7a_4f40_871f_33179a38f87b.slice/crio-8321baba0a11285f61ba1d8c7ff4b5cdb53b412fa7aa5ae5f4e48f4c05f11c92 WatchSource:0}: Error finding container 8321baba0a11285f61ba1d8c7ff4b5cdb53b412fa7aa5ae5f4e48f4c05f11c92: Status 404 returned error can't find the container with id 8321baba0a11285f61ba1d8c7ff4b5cdb53b412fa7aa5ae5f4e48f4c05f11c92 Sep 10 18:43:01.770051 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.770014 2581 generic.go:356] "Generic (PLEG): container finished" podID="1933a925-2316-41ae-9639-0f74c43e39a6" containerID="77ecef1c898fc8e73956e1e02664fdae6c5b9b6f5ad94a35a9b3fa6ce1888e09" exitCode=0 Sep 10 18:43:01.770503 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.770065 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gzjvf" Sep 10 18:43:01.770503 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.770114 2581 scope.go:117] "RemoveContainer" containerID="77ecef1c898fc8e73956e1e02664fdae6c5b9b6f5ad94a35a9b3fa6ce1888e09" Sep 10 18:43:01.771779 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.771755 2581 generic.go:356] "Generic (PLEG): container finished" podID="82ee372f-bd75-483d-8474-5126dcada0aa" containerID="483d53a9975b41028318c38b9d2144c09e66d14341c4455d06424d47733af04c" exitCode=0 Sep 10 18:43:01.772120 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.771803 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5v6zs" Sep 10 18:43:01.772408 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:43:01.772377 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-gzjvf\" is forbidden: User \"system:node:ip-10-0-134-166.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-166.ec2.internal' and this object" podUID="1933a925-2316-41ae-9639-0f74c43e39a6" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gzjvf" Sep 10 18:43:01.773875 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.773850 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ptlsk" event={"ID":"3d15dbde-9c7a-4f40-871f-33179a38f87b","Type":"ContainerStarted","Data":"4446508264ef7c07d83bb8f7314623e1b60b13d1b9e2be417692931e4f76f70e"} Sep 10 18:43:01.773963 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.773900 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ptlsk" event={"ID":"3d15dbde-9c7a-4f40-871f-33179a38f87b","Type":"ContainerStarted","Data":"8321baba0a11285f61ba1d8c7ff4b5cdb53b412fa7aa5ae5f4e48f4c05f11c92"} Sep 10 18:43:01.773963 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.773929 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ptlsk" Sep 10 18:43:01.774051 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:43:01.774018 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5v6zs\" is forbidden: User \"system:node:ip-10-0-134-166.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-166.ec2.internal' and this object" podUID="82ee372f-bd75-483d-8474-5126dcada0aa" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5v6zs" Sep 10 18:43:01.775794 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:43:01.775681 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-gzjvf\" is forbidden: User \"system:node:ip-10-0-134-166.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-166.ec2.internal' and this object" podUID="1933a925-2316-41ae-9639-0f74c43e39a6" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gzjvf" Sep 10 18:43:01.775794 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.775771 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g6c6n" event={"ID":"6de0670d-8028-43bb-9407-4d4096ff97d2","Type":"ContainerStarted","Data":"04d2a73fd63d3924378e2e3c2e5ed5d86cdc9f79a01e603095545126b937ff41"} Sep 10 18:43:01.775912 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.775815 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g6c6n" event={"ID":"6de0670d-8028-43bb-9407-4d4096ff97d2","Type":"ContainerStarted","Data":"9968c0c6cf25a3a5b90f6102deaebdad9d2f46e1312dc93d4bb731528c152f49"} Sep 10 18:43:01.775961 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.775936 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g6c6n" Sep 10 18:43:01.777084 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:43:01.777062 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5v6zs\" is forbidden: User \"system:node:ip-10-0-134-166.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-166.ec2.internal' and this object" podUID="82ee372f-bd75-483d-8474-5126dcada0aa" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5v6zs" Sep 10 18:43:01.779627 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.779487 2581 scope.go:117] "RemoveContainer" containerID="77ecef1c898fc8e73956e1e02664fdae6c5b9b6f5ad94a35a9b3fa6ce1888e09" Sep 10 18:43:01.779845 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:43:01.779828 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77ecef1c898fc8e73956e1e02664fdae6c5b9b6f5ad94a35a9b3fa6ce1888e09\": container with ID starting with 77ecef1c898fc8e73956e1e02664fdae6c5b9b6f5ad94a35a9b3fa6ce1888e09 not found: ID does not exist" containerID="77ecef1c898fc8e73956e1e02664fdae6c5b9b6f5ad94a35a9b3fa6ce1888e09" Sep 10 18:43:01.779901 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.779852 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77ecef1c898fc8e73956e1e02664fdae6c5b9b6f5ad94a35a9b3fa6ce1888e09"} err="failed to get container status \"77ecef1c898fc8e73956e1e02664fdae6c5b9b6f5ad94a35a9b3fa6ce1888e09\": rpc error: code = NotFound desc = could not find container \"77ecef1c898fc8e73956e1e02664fdae6c5b9b6f5ad94a35a9b3fa6ce1888e09\": container with ID starting with 77ecef1c898fc8e73956e1e02664fdae6c5b9b6f5ad94a35a9b3fa6ce1888e09 not found: ID does not exist" Sep 10 18:43:01.779901 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.779870 2581 scope.go:117] "RemoveContainer" containerID="483d53a9975b41028318c38b9d2144c09e66d14341c4455d06424d47733af04c" Sep 10 18:43:01.787515 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.787501 2581 scope.go:117] "RemoveContainer" containerID="483d53a9975b41028318c38b9d2144c09e66d14341c4455d06424d47733af04c" Sep 10 18:43:01.787756 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:43:01.787740 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"483d53a9975b41028318c38b9d2144c09e66d14341c4455d06424d47733af04c\": container with ID starting with 483d53a9975b41028318c38b9d2144c09e66d14341c4455d06424d47733af04c not found: ID does not exist" containerID="483d53a9975b41028318c38b9d2144c09e66d14341c4455d06424d47733af04c" Sep 10 18:43:01.787798 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.787762 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"483d53a9975b41028318c38b9d2144c09e66d14341c4455d06424d47733af04c"} err="failed to get container status \"483d53a9975b41028318c38b9d2144c09e66d14341c4455d06424d47733af04c\": rpc error: code = NotFound desc = could not find container \"483d53a9975b41028318c38b9d2144c09e66d14341c4455d06424d47733af04c\": container with ID starting with 483d53a9975b41028318c38b9d2144c09e66d14341c4455d06424d47733af04c not found: ID does not exist" Sep 10 18:43:01.794200 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.794147 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ptlsk" podStartSLOduration=1.7941311039999999 podStartE2EDuration="1.794131104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:43:01.793644083 +0000 UTC m=+707.879981557" watchObservedRunningTime="2026-09-10 18:43:01.794131104 +0000 UTC m=+707.880468580" Sep 10 18:43:01.795576 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:43:01.795551 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-gzjvf\" is forbidden: User \"system:node:ip-10-0-134-166.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-166.ec2.internal' and this object" podUID="1933a925-2316-41ae-9639-0f74c43e39a6" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gzjvf" Sep 10 18:43:01.797014 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:43:01.796991 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5v6zs\" is forbidden: User \"system:node:ip-10-0-134-166.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-166.ec2.internal' and this object" podUID="82ee372f-bd75-483d-8474-5126dcada0aa" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5v6zs" Sep 10 18:43:01.825576 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:01.822372 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g6c6n" podStartSLOduration=1.822351014 podStartE2EDuration="1.822351014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:43:01.819275988 +0000 UTC m=+707.905613454" watchObservedRunningTime="2026-09-10 18:43:01.822351014 +0000 UTC m=+707.908688496" Sep 10 18:43:02.018137 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:02.018102 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:02.116923 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:02.116891 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:02.380836 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:02.380761 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1933a925-2316-41ae-9639-0f74c43e39a6" path="/var/lib/kubelet/pods/1933a925-2316-41ae-9639-0f74c43e39a6/volumes" Sep 10 18:43:02.381069 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:02.381057 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82ee372f-bd75-483d-8474-5126dcada0aa" path="/var/lib/kubelet/pods/82ee372f-bd75-483d-8474-5126dcada0aa/volumes" Sep 10 18:43:04.449910 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:04.449886 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:04.567925 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:04.567897 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:08.757128 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:08.757057 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-6hl2d" Sep 10 18:43:09.218289 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:09.218259 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:09.389278 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:09.389241 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:12.784312 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:12.784279 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ptlsk" Sep 10 18:43:12.784776 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:12.784426 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g6c6n" Sep 10 18:43:12.815927 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:12.815893 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:14.065553 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:14.065522 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:14.166601 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:14.166566 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:14.267278 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:14.267250 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:17.668840 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:17.668804 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:17.766058 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:17.766026 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:18.353184 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:18.353146 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g6c6n"] Sep 10 18:43:18.353383 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:18.353358 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g6c6n" podUID="6de0670d-8028-43bb-9407-4d4096ff97d2" containerName="manager" containerID="cri-o://04d2a73fd63d3924378e2e3c2e5ed5d86cdc9f79a01e603095545126b937ff41" gracePeriod=10 Sep 10 18:43:18.584720 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:18.584697 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g6c6n" Sep 10 18:43:18.619953 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:18.619892 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6de0670d-8028-43bb-9407-4d4096ff97d2-extensions-socket-volume\") pod \"6de0670d-8028-43bb-9407-4d4096ff97d2\" (UID: \"6de0670d-8028-43bb-9407-4d4096ff97d2\") " Sep 10 18:43:18.619953 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:18.619937 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrf6q\" (UniqueName: \"kubernetes.io/projected/6de0670d-8028-43bb-9407-4d4096ff97d2-kube-api-access-lrf6q\") pod \"6de0670d-8028-43bb-9407-4d4096ff97d2\" (UID: \"6de0670d-8028-43bb-9407-4d4096ff97d2\") " Sep 10 18:43:18.620260 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:18.620236 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de0670d-8028-43bb-9407-4d4096ff97d2-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6de0670d-8028-43bb-9407-4d4096ff97d2" (UID: "6de0670d-8028-43bb-9407-4d4096ff97d2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 18:43:18.622049 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:18.622028 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6de0670d-8028-43bb-9407-4d4096ff97d2-kube-api-access-lrf6q" (OuterVolumeSpecName: "kube-api-access-lrf6q") pod "6de0670d-8028-43bb-9407-4d4096ff97d2" (UID: "6de0670d-8028-43bb-9407-4d4096ff97d2"). InnerVolumeSpecName "kube-api-access-lrf6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 18:43:18.720593 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:18.720556 2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6de0670d-8028-43bb-9407-4d4096ff97d2-extensions-socket-volume\") on node \"ip-10-0-134-166.ec2.internal\" DevicePath \"\"" Sep 10 18:43:18.720593 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:18.720589 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrf6q\" (UniqueName: \"kubernetes.io/projected/6de0670d-8028-43bb-9407-4d4096ff97d2-kube-api-access-lrf6q\") on node \"ip-10-0-134-166.ec2.internal\" DevicePath \"\"" Sep 10 18:43:18.852803 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:18.852769 2581 generic.go:356] "Generic (PLEG): container finished" podID="6de0670d-8028-43bb-9407-4d4096ff97d2" containerID="04d2a73fd63d3924378e2e3c2e5ed5d86cdc9f79a01e603095545126b937ff41" exitCode=0 Sep 10 18:43:18.852962 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:18.852810 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g6c6n" event={"ID":"6de0670d-8028-43bb-9407-4d4096ff97d2","Type":"ContainerDied","Data":"04d2a73fd63d3924378e2e3c2e5ed5d86cdc9f79a01e603095545126b937ff41"} Sep 10 18:43:18.852962 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:18.852813 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g6c6n" Sep 10 18:43:18.852962 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:18.852846 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g6c6n" event={"ID":"6de0670d-8028-43bb-9407-4d4096ff97d2","Type":"ContainerDied","Data":"9968c0c6cf25a3a5b90f6102deaebdad9d2f46e1312dc93d4bb731528c152f49"} Sep 10 18:43:18.852962 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:18.852868 2581 scope.go:117] "RemoveContainer" containerID="04d2a73fd63d3924378e2e3c2e5ed5d86cdc9f79a01e603095545126b937ff41" Sep 10 18:43:18.861105 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:18.861089 2581 scope.go:117] "RemoveContainer" containerID="04d2a73fd63d3924378e2e3c2e5ed5d86cdc9f79a01e603095545126b937ff41" Sep 10 18:43:18.861381 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:43:18.861362 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04d2a73fd63d3924378e2e3c2e5ed5d86cdc9f79a01e603095545126b937ff41\": container with ID starting with 04d2a73fd63d3924378e2e3c2e5ed5d86cdc9f79a01e603095545126b937ff41 not found: ID does not exist" containerID="04d2a73fd63d3924378e2e3c2e5ed5d86cdc9f79a01e603095545126b937ff41" Sep 10 18:43:18.861440 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:18.861390 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d2a73fd63d3924378e2e3c2e5ed5d86cdc9f79a01e603095545126b937ff41"} err="failed to get container status \"04d2a73fd63d3924378e2e3c2e5ed5d86cdc9f79a01e603095545126b937ff41\": rpc error: code = NotFound desc = could not find container \"04d2a73fd63d3924378e2e3c2e5ed5d86cdc9f79a01e603095545126b937ff41\": container with ID starting with 04d2a73fd63d3924378e2e3c2e5ed5d86cdc9f79a01e603095545126b937ff41 not found: ID does not exist" Sep 10 18:43:18.874377 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:18.874301 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g6c6n"] Sep 10 18:43:18.878429 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:18.878402 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g6c6n"] Sep 10 18:43:20.380552 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:20.380519 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6de0670d-8028-43bb-9407-4d4096ff97d2" path="/var/lib/kubelet/pods/6de0670d-8028-43bb-9407-4d4096ff97d2/volumes" Sep 10 18:43:22.886177 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:22.886144 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:22.986251 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:22.986215 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:25.342380 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:25.342348 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:25.439539 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:25.439505 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:30.089587 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:30.089548 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:30.190396 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:30.190367 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:34.642113 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:34.642082 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2"] Sep 10 18:43:34.642540 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:34.642525 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6de0670d-8028-43bb-9407-4d4096ff97d2" containerName="manager" Sep 10 18:43:34.642588 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:34.642543 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6de0670d-8028-43bb-9407-4d4096ff97d2" containerName="manager" Sep 10 18:43:34.642653 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:34.642642 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6de0670d-8028-43bb-9407-4d4096ff97d2" containerName="manager" Sep 10 18:43:34.659198 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:34.659173 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2"] Sep 10 18:43:34.659368 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:34.659281 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" Sep 10 18:43:34.661137 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:34.661099 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 10 18:43:34.661274 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:34.661177 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-gtdvb\"" Sep 10 18:43:34.682338 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:34.682303 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:34.735207 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:34.735184 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/952dcc01-8c30-41c0-9d9f-2964ed5c0760-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6mg2\" (UID: \"952dcc01-8c30-41c0-9d9f-2964ed5c0760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" Sep 10 18:43:34.735308 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:34.735212 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/952dcc01-8c30-41c0-9d9f-2964ed5c0760-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6mg2\" (UID: \"952dcc01-8c30-41c0-9d9f-2964ed5c0760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" Sep 10 18:43:34.735308 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:34.735249 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/952dcc01-8c30-41c0-9d9f-2964ed5c0760-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6mg2\" (UID: \"952dcc01-8c30-41c0-9d9f-2964ed5c0760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" Sep 10 18:43:34.735308 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:34.735270 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/952dcc01-8c30-41c0-9d9f-2964ed5c0760-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6mg2\" (UID: \"952dcc01-8c30-41c0-9d9f-2964ed5c0760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" Sep 10 18:43:34.735308 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:34.735295 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/952dcc01-8c30-41c0-9d9f-2964ed5c0760-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6mg2\" (UID: \"952dcc01-8c30-41c0-9d9f-2964ed5c0760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" Sep 10 18:43:34.735537 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:34.735450 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/952dcc01-8c30-41c0-9d9f-2964ed5c0760-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6mg2\" (UID: \"952dcc01-8c30-41c0-9d9f-2964ed5c0760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" Sep 10 18:43:34.735537 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:34.735483 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/952dcc01-8c30-41c0-9d9f-2964ed5c0760-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6mg2\" (UID: \"952dcc01-8c30-41c0-9d9f-2964ed5c0760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" Sep 10 18:43:34.735537 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:34.735509 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hsrk\" (UniqueName: \"kubernetes.io/projected/952dcc01-8c30-41c0-9d9f-2964ed5c0760-kube-api-access-4hsrk\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6mg2\" (UID: \"952dcc01-8c30-41c0-9d9f-2964ed5c0760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" Sep 10 18:43:34.735635 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:34.735544 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/952dcc01-8c30-41c0-9d9f-2964ed5c0760-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6mg2\" (UID: \"952dcc01-8c30-41c0-9d9f-2964ed5c0760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" Sep 10 18:43:34.836495 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:34.836471 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/952dcc01-8c30-41c0-9d9f-2964ed5c0760-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6mg2\" (UID: \"952dcc01-8c30-41c0-9d9f-2964ed5c0760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" Sep 10 18:43:34.836620 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:34.836497 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/952dcc01-8c30-41c0-9d9f-2964ed5c0760-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6mg2\" (UID: \"952dcc01-8c30-41c0-9d9f-2964ed5c0760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" Sep 10 18:43:34.836620 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:34.836516 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hsrk\" (UniqueName: \"kubernetes.io/projected/952dcc01-8c30-41c0-9d9f-2964ed5c0760-kube-api-access-4hsrk\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6mg2\" (UID: \"952dcc01-8c30-41c0-9d9f-2964ed5c0760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" Sep 10 18:43:34.836620 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:34.836535 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/952dcc01-8c30-41c0-9d9f-2964ed5c0760-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6mg2\" (UID: \"952dcc01-8c30-41c0-9d9f-2964ed5c0760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" Sep 10 18:43:34.836620 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:34.836552 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/952dcc01-8c30-41c0-9d9f-2964ed5c0760-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6mg2\" (UID: \"952dcc01-8c30-41c0-9d9f-2964ed5c0760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" Sep 10 18:43:34.836620 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:34.836575 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/952dcc01-8c30-41c0-9d9f-2964ed5c0760-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6mg2\" (UID: \"952dcc01-8c30-41c0-9d9f-2964ed5c0760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" Sep 10 18:43:34.836620 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:34.836614 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/952dcc01-8c30-41c0-9d9f-2964ed5c0760-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6mg2\" (UID: \"952dcc01-8c30-41c0-9d9f-2964ed5c0760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" Sep 10 18:43:34.836926 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:34.836653 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/952dcc01-8c30-41c0-9d9f-2964ed5c0760-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6mg2\" (UID: \"952dcc01-8c30-41c0-9d9f-2964ed5c0760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" Sep 10 18:43:34.836926 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:34.836750 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/952dcc01-8c30-41c0-9d9f-2964ed5c0760-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6mg2\" (UID: \"952dcc01-8c30-41c0-9d9f-2964ed5c0760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" Sep 10 18:43:34.837013 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:34.836991 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/952dcc01-8c30-41c0-9d9f-2964ed5c0760-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6mg2\" (UID: \"952dcc01-8c30-41c0-9d9f-2964ed5c0760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" Sep 10 18:43:34.837089 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:34.837059 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/952dcc01-8c30-41c0-9d9f-2964ed5c0760-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6mg2\" (UID: \"952dcc01-8c30-41c0-9d9f-2964ed5c0760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" Sep 10 18:43:34.837390 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:34.837125 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/952dcc01-8c30-41c0-9d9f-2964ed5c0760-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6mg2\" (UID: \"952dcc01-8c30-41c0-9d9f-2964ed5c0760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" Sep 10 18:43:34.837390 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:34.837148 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/952dcc01-8c30-41c0-9d9f-2964ed5c0760-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6mg2\" (UID: \"952dcc01-8c30-41c0-9d9f-2964ed5c0760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" Sep 10 18:43:34.837520 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:34.837496 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/952dcc01-8c30-41c0-9d9f-2964ed5c0760-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6mg2\" (UID: \"952dcc01-8c30-41c0-9d9f-2964ed5c0760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" Sep 10 18:43:34.838922 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:34.838903 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/952dcc01-8c30-41c0-9d9f-2964ed5c0760-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6mg2\" (UID: \"952dcc01-8c30-41c0-9d9f-2964ed5c0760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" Sep 10 18:43:34.839191 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:34.839177 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/952dcc01-8c30-41c0-9d9f-2964ed5c0760-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6mg2\" (UID: \"952dcc01-8c30-41c0-9d9f-2964ed5c0760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" Sep 10 18:43:34.845870 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:34.845850 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hsrk\" (UniqueName: \"kubernetes.io/projected/952dcc01-8c30-41c0-9d9f-2964ed5c0760-kube-api-access-4hsrk\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6mg2\" (UID: \"952dcc01-8c30-41c0-9d9f-2964ed5c0760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" Sep 10 18:43:34.845963 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:34.845913 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/952dcc01-8c30-41c0-9d9f-2964ed5c0760-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6mg2\" (UID: \"952dcc01-8c30-41c0-9d9f-2964ed5c0760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" Sep 10 18:43:34.969764 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:34.969697 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" Sep 10 18:43:35.053361 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:35.053318 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:35.147975 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:35.147943 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:35.207346 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:35.207300 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:35.244139 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:35.244119 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2"] Sep 10 18:43:35.246186 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:43:35.246164 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod952dcc01_8c30_41c0_9d9f_2964ed5c0760.slice/crio-0b6c1fa8b1c0ad6a4f3a0ddbeda43eb6e6e76673ed757ed6c7fdb1560672bb1e WatchSource:0}: Error finding container 0b6c1fa8b1c0ad6a4f3a0ddbeda43eb6e6e76673ed757ed6c7fdb1560672bb1e: Status 404 returned error can't find the container with id 0b6c1fa8b1c0ad6a4f3a0ddbeda43eb6e6e76673ed757ed6c7fdb1560672bb1e Sep 10 18:43:35.498844 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:35.498760 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:35.599219 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:35.599188 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:35.900026 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:35.900000 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:35.930347 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:35.930298 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" event={"ID":"952dcc01-8c30-41c0-9d9f-2964ed5c0760","Type":"ContainerStarted","Data":"0b6c1fa8b1c0ad6a4f3a0ddbeda43eb6e6e76673ed757ed6c7fdb1560672bb1e"} Sep 10 18:43:35.999406 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:35.999379 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:38.237725 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:38.237687 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 18:43:38.238012 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:38.237755 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 18:43:38.238012 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:38.237783 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 18:43:38.946740 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:38.946701 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" event={"ID":"952dcc01-8c30-41c0-9d9f-2964ed5c0760","Type":"ContainerStarted","Data":"04964628345d7d80865897ec542ba3edc7be0143aed1a38e3c32f52299cddbaa"} Sep 10 18:43:38.965308 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:38.965266 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" podStartSLOduration=1.976395171 podStartE2EDuration="4.965251638s" totalImagesPullingTime="2.988856467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:43:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:43:38.963540037 +0000 UTC m=+745.049877511" watchObservedRunningTime="2026-09-10 18:43:38.965251638 +0000 UTC m=+745.051589113" Sep 10 18:43:38.970642 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:38.970619 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" Sep 10 18:43:39.896923 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:39.896882 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-ppp9v"] Sep 10 18:43:39.897394 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:39.897182 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:39.900086 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:39.900068 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-ppp9v" Sep 10 18:43:39.902002 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:39.901986 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 10 18:43:39.907212 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:39.907189 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-ppp9v"] Sep 10 18:43:39.974699 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:39.974662 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" Sep 10 18:43:39.979030 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:39.979003 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77skg\" (UniqueName: \"kubernetes.io/projected/8ad9685a-5cd2-44bc-a7c4-a0519182a1d6-kube-api-access-77skg\") pod \"limitador-limitador-8588749977-ppp9v\" (UID: \"8ad9685a-5cd2-44bc-a7c4-a0519182a1d6\") " pod="kuadrant-system/limitador-limitador-8588749977-ppp9v" Sep 10 18:43:39.979194 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:39.979060 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8ad9685a-5cd2-44bc-a7c4-a0519182a1d6-config-file\") pod \"limitador-limitador-8588749977-ppp9v\" (UID: \"8ad9685a-5cd2-44bc-a7c4-a0519182a1d6\") " pod="kuadrant-system/limitador-limitador-8588749977-ppp9v" Sep 10 18:43:39.990859 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:39.990830 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-ppp9v"] Sep 10 18:43:39.999226 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:39.999202 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:40.056839 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:40.056811 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:40.079847 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:40.079822 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77skg\" (UniqueName: \"kubernetes.io/projected/8ad9685a-5cd2-44bc-a7c4-a0519182a1d6-kube-api-access-77skg\") pod \"limitador-limitador-8588749977-ppp9v\" (UID: \"8ad9685a-5cd2-44bc-a7c4-a0519182a1d6\") " pod="kuadrant-system/limitador-limitador-8588749977-ppp9v" Sep 10 18:43:40.079938 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:40.079877 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8ad9685a-5cd2-44bc-a7c4-a0519182a1d6-config-file\") pod \"limitador-limitador-8588749977-ppp9v\" (UID: \"8ad9685a-5cd2-44bc-a7c4-a0519182a1d6\") " pod="kuadrant-system/limitador-limitador-8588749977-ppp9v" Sep 10 18:43:40.080584 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:40.080562 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8ad9685a-5cd2-44bc-a7c4-a0519182a1d6-config-file\") pod \"limitador-limitador-8588749977-ppp9v\" (UID: \"8ad9685a-5cd2-44bc-a7c4-a0519182a1d6\") " pod="kuadrant-system/limitador-limitador-8588749977-ppp9v" Sep 10 18:43:40.088101 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:40.088076 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77skg\" (UniqueName: \"kubernetes.io/projected/8ad9685a-5cd2-44bc-a7c4-a0519182a1d6-kube-api-access-77skg\") pod \"limitador-limitador-8588749977-ppp9v\" (UID: \"8ad9685a-5cd2-44bc-a7c4-a0519182a1d6\") " pod="kuadrant-system/limitador-limitador-8588749977-ppp9v" Sep 10 18:43:40.210403 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:40.210309 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-ppp9v" Sep 10 18:43:40.484047 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:40.484022 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-ppp9v"] Sep 10 18:43:40.486639 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:43:40.486611 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ad9685a_5cd2_44bc_a7c4_a0519182a1d6.slice/crio-f1a4f66af40db78653543d36b97c7ce0bba5c2fb0e40fe249295cb1d26084bdc WatchSource:0}: Error finding container f1a4f66af40db78653543d36b97c7ce0bba5c2fb0e40fe249295cb1d26084bdc: Status 404 returned error can't find the container with id f1a4f66af40db78653543d36b97c7ce0bba5c2fb0e40fe249295cb1d26084bdc Sep 10 18:43:40.670069 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:40.670041 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-lrwgf"] Sep 10 18:43:40.674365 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:40.674350 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-lrwgf" Sep 10 18:43:40.676151 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:40.676136 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-v49zl\"" Sep 10 18:43:40.680014 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:40.679996 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-lrwgf"] Sep 10 18:43:40.702577 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:40.702537 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:40.779178 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:40.779111 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:40.785212 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:40.785193 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2x6h\" (UniqueName: \"kubernetes.io/projected/a07c0ad1-eeb0-4b33-995b-04e109cc6c78-kube-api-access-x2x6h\") pod \"authorino-7bcfd8f9cd-lrwgf\" (UID: \"a07c0ad1-eeb0-4b33-995b-04e109cc6c78\") " pod="kuadrant-system/authorino-7bcfd8f9cd-lrwgf" Sep 10 18:43:40.886547 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:40.886522 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2x6h\" (UniqueName: \"kubernetes.io/projected/a07c0ad1-eeb0-4b33-995b-04e109cc6c78-kube-api-access-x2x6h\") pod \"authorino-7bcfd8f9cd-lrwgf\" (UID: \"a07c0ad1-eeb0-4b33-995b-04e109cc6c78\") " pod="kuadrant-system/authorino-7bcfd8f9cd-lrwgf" Sep 10 18:43:40.894427 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:40.894400 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2x6h\" (UniqueName: \"kubernetes.io/projected/a07c0ad1-eeb0-4b33-995b-04e109cc6c78-kube-api-access-x2x6h\") pod \"authorino-7bcfd8f9cd-lrwgf\" (UID: \"a07c0ad1-eeb0-4b33-995b-04e109cc6c78\") " pod="kuadrant-system/authorino-7bcfd8f9cd-lrwgf" Sep 10 18:43:40.957202 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:40.957178 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-ppp9v" event={"ID":"8ad9685a-5cd2-44bc-a7c4-a0519182a1d6","Type":"ContainerStarted","Data":"f1a4f66af40db78653543d36b97c7ce0bba5c2fb0e40fe249295cb1d26084bdc"} Sep 10 18:43:40.957508 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:40.957358 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" Sep 10 18:43:40.958275 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:40.958255 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" Sep 10 18:43:40.985429 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:40.985399 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-lrwgf" Sep 10 18:43:41.258132 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:41.258093 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-lrwgf"] Sep 10 18:43:41.260214 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:43:41.260179 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda07c0ad1_eeb0_4b33_995b_04e109cc6c78.slice/crio-397609f6b1397e79c63f79be5150ba5fd9c4af77c6e60484dc8122d19c29168e WatchSource:0}: Error finding container 397609f6b1397e79c63f79be5150ba5fd9c4af77c6e60484dc8122d19c29168e: Status 404 returned error can't find the container with id 397609f6b1397e79c63f79be5150ba5fd9c4af77c6e60484dc8122d19c29168e Sep 10 18:43:41.962624 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:41.962593 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-lrwgf" event={"ID":"a07c0ad1-eeb0-4b33-995b-04e109cc6c78","Type":"ContainerStarted","Data":"397609f6b1397e79c63f79be5150ba5fd9c4af77c6e60484dc8122d19c29168e"} Sep 10 18:43:43.514695 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:43.514671 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-lrwgf"] Sep 10 18:43:43.975502 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:43.975469 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-lrwgf" event={"ID":"a07c0ad1-eeb0-4b33-995b-04e109cc6c78","Type":"ContainerStarted","Data":"c129147b7140221358fd73c71c7b6829e7e82430c1fdb843a53e36275b98c8d3"} Sep 10 18:43:43.976956 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:43.976938 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-ppp9v" event={"ID":"8ad9685a-5cd2-44bc-a7c4-a0519182a1d6","Type":"ContainerStarted","Data":"6c124a2b220fea97469f0094ab26269f5c06b983140fc2ff75d6c0619015e33a"} Sep 10 18:43:43.977027 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:43.976982 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-ppp9v" Sep 10 18:43:43.989373 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:43.989304 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-lrwgf" podStartSLOduration=2.148900086 podStartE2EDuration="3.989290029s" totalImagesPullingTime="1.840389943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:43:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:43:43.988126212 +0000 UTC m=+750.074463686" watchObservedRunningTime="2026-09-10 18:43:43.989290029 +0000 UTC m=+750.075627505" Sep 10 18:43:44.001707 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:44.001670 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-ppp9v" podStartSLOduration=2.005156523 podStartE2EDuration="5.001658864s" totalImagesPullingTime="2.996502341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:43:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:43:44.000936384 +0000 UTC m=+750.087273872" watchObservedRunningTime="2026-09-10 18:43:44.001658864 +0000 UTC m=+750.087996340" Sep 10 18:43:44.310748 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:44.310718 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:44.408469 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:44.408436 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:44.981607 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:44.981572 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-lrwgf" podUID="a07c0ad1-eeb0-4b33-995b-04e109cc6c78" containerName="authorino" containerID="cri-o://c129147b7140221358fd73c71c7b6829e7e82430c1fdb843a53e36275b98c8d3" gracePeriod=30 Sep 10 18:43:45.234780 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:45.234731 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-lrwgf" Sep 10 18:43:45.322204 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:45.322179 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2x6h\" (UniqueName: \"kubernetes.io/projected/a07c0ad1-eeb0-4b33-995b-04e109cc6c78-kube-api-access-x2x6h\") pod \"a07c0ad1-eeb0-4b33-995b-04e109cc6c78\" (UID: \"a07c0ad1-eeb0-4b33-995b-04e109cc6c78\") " Sep 10 18:43:45.324263 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:45.324236 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a07c0ad1-eeb0-4b33-995b-04e109cc6c78-kube-api-access-x2x6h" (OuterVolumeSpecName: "kube-api-access-x2x6h") pod "a07c0ad1-eeb0-4b33-995b-04e109cc6c78" (UID: "a07c0ad1-eeb0-4b33-995b-04e109cc6c78"). InnerVolumeSpecName "kube-api-access-x2x6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 18:43:45.423042 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:45.423013 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2x6h\" (UniqueName: \"kubernetes.io/projected/a07c0ad1-eeb0-4b33-995b-04e109cc6c78-kube-api-access-x2x6h\") on node \"ip-10-0-134-166.ec2.internal\" DevicePath \"\"" Sep 10 18:43:45.987341 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:45.987279 2581 generic.go:356] "Generic (PLEG): container finished" podID="a07c0ad1-eeb0-4b33-995b-04e109cc6c78" containerID="c129147b7140221358fd73c71c7b6829e7e82430c1fdb843a53e36275b98c8d3" exitCode=0 Sep 10 18:43:45.987757 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:45.987357 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-lrwgf" Sep 10 18:43:45.987757 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:45.987367 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-lrwgf" event={"ID":"a07c0ad1-eeb0-4b33-995b-04e109cc6c78","Type":"ContainerDied","Data":"c129147b7140221358fd73c71c7b6829e7e82430c1fdb843a53e36275b98c8d3"} Sep 10 18:43:45.987757 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:45.987418 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-lrwgf" event={"ID":"a07c0ad1-eeb0-4b33-995b-04e109cc6c78","Type":"ContainerDied","Data":"397609f6b1397e79c63f79be5150ba5fd9c4af77c6e60484dc8122d19c29168e"} Sep 10 18:43:45.987757 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:45.987438 2581 scope.go:117] "RemoveContainer" containerID="c129147b7140221358fd73c71c7b6829e7e82430c1fdb843a53e36275b98c8d3" Sep 10 18:43:45.996083 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:45.996068 2581 scope.go:117] "RemoveContainer" containerID="c129147b7140221358fd73c71c7b6829e7e82430c1fdb843a53e36275b98c8d3" Sep 10 18:43:45.996370 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:43:45.996348 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c129147b7140221358fd73c71c7b6829e7e82430c1fdb843a53e36275b98c8d3\": container with ID starting with c129147b7140221358fd73c71c7b6829e7e82430c1fdb843a53e36275b98c8d3 not found: ID does not exist" containerID="c129147b7140221358fd73c71c7b6829e7e82430c1fdb843a53e36275b98c8d3" Sep 10 18:43:45.996470 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:45.996376 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c129147b7140221358fd73c71c7b6829e7e82430c1fdb843a53e36275b98c8d3"} err="failed to get container status \"c129147b7140221358fd73c71c7b6829e7e82430c1fdb843a53e36275b98c8d3\": rpc error: code = NotFound desc = could not find container \"c129147b7140221358fd73c71c7b6829e7e82430c1fdb843a53e36275b98c8d3\": container with ID starting with c129147b7140221358fd73c71c7b6829e7e82430c1fdb843a53e36275b98c8d3 not found: ID does not exist" Sep 10 18:43:46.005545 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:46.005523 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-lrwgf"] Sep 10 18:43:46.009255 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:46.009234 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-lrwgf"] Sep 10 18:43:46.381210 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:46.381175 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a07c0ad1-eeb0-4b33-995b-04e109cc6c78" path="/var/lib/kubelet/pods/a07c0ad1-eeb0-4b33-995b-04e109cc6c78/volumes" Sep 10 18:43:46.748354 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:46.748247 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:46.798965 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:46.798938 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:51.500524 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:51.500489 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:51.606550 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:51.606513 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:53.517363 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:53.517316 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-ppp9v"] Sep 10 18:43:53.517714 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:53.517545 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-ppp9v" podUID="8ad9685a-5cd2-44bc-a7c4-a0519182a1d6" containerName="limitador" containerID="cri-o://6c124a2b220fea97469f0094ab26269f5c06b983140fc2ff75d6c0619015e33a" gracePeriod=30 Sep 10 18:43:53.518460 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:53.518095 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-ppp9v" Sep 10 18:43:54.024912 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:54.024878 2581 generic.go:356] "Generic (PLEG): container finished" podID="8ad9685a-5cd2-44bc-a7c4-a0519182a1d6" containerID="6c124a2b220fea97469f0094ab26269f5c06b983140fc2ff75d6c0619015e33a" exitCode=0 Sep 10 18:43:54.025067 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:54.024954 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-ppp9v" event={"ID":"8ad9685a-5cd2-44bc-a7c4-a0519182a1d6","Type":"ContainerDied","Data":"6c124a2b220fea97469f0094ab26269f5c06b983140fc2ff75d6c0619015e33a"} Sep 10 18:43:54.463295 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:54.463272 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-ppp9v" Sep 10 18:43:54.595251 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:54.595171 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8ad9685a-5cd2-44bc-a7c4-a0519182a1d6-config-file\") pod \"8ad9685a-5cd2-44bc-a7c4-a0519182a1d6\" (UID: \"8ad9685a-5cd2-44bc-a7c4-a0519182a1d6\") " Sep 10 18:43:54.595251 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:54.595213 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77skg\" (UniqueName: \"kubernetes.io/projected/8ad9685a-5cd2-44bc-a7c4-a0519182a1d6-kube-api-access-77skg\") pod \"8ad9685a-5cd2-44bc-a7c4-a0519182a1d6\" (UID: \"8ad9685a-5cd2-44bc-a7c4-a0519182a1d6\") " Sep 10 18:43:54.595640 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:54.595487 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ad9685a-5cd2-44bc-a7c4-a0519182a1d6-config-file" (OuterVolumeSpecName: "config-file") pod "8ad9685a-5cd2-44bc-a7c4-a0519182a1d6" (UID: "8ad9685a-5cd2-44bc-a7c4-a0519182a1d6"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 18:43:54.597275 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:54.597256 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ad9685a-5cd2-44bc-a7c4-a0519182a1d6-kube-api-access-77skg" (OuterVolumeSpecName: "kube-api-access-77skg") pod "8ad9685a-5cd2-44bc-a7c4-a0519182a1d6" (UID: "8ad9685a-5cd2-44bc-a7c4-a0519182a1d6"). InnerVolumeSpecName "kube-api-access-77skg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 18:43:54.696692 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:54.696658 2581 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8ad9685a-5cd2-44bc-a7c4-a0519182a1d6-config-file\") on node \"ip-10-0-134-166.ec2.internal\" DevicePath \"\"" Sep 10 18:43:54.696692 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:54.696689 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77skg\" (UniqueName: \"kubernetes.io/projected/8ad9685a-5cd2-44bc-a7c4-a0519182a1d6-kube-api-access-77skg\") on node \"ip-10-0-134-166.ec2.internal\" DevicePath \"\"" Sep 10 18:43:55.030519 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:55.030442 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-ppp9v" Sep 10 18:43:55.030519 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:55.030462 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-ppp9v" event={"ID":"8ad9685a-5cd2-44bc-a7c4-a0519182a1d6","Type":"ContainerDied","Data":"f1a4f66af40db78653543d36b97c7ce0bba5c2fb0e40fe249295cb1d26084bdc"} Sep 10 18:43:55.030519 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:55.030519 2581 scope.go:117] "RemoveContainer" containerID="6c124a2b220fea97469f0094ab26269f5c06b983140fc2ff75d6c0619015e33a" Sep 10 18:43:55.048021 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:55.047997 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-ppp9v"] Sep 10 18:43:55.051088 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:55.051068 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-ppp9v"] Sep 10 18:43:56.087658 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:56.087626 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:56.187525 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:56.187471 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:56.379553 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:56.379482 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ad9685a-5cd2-44bc-a7c4-a0519182a1d6" path="/var/lib/kubelet/pods/8ad9685a-5cd2-44bc-a7c4-a0519182a1d6/volumes" Sep 10 18:43:56.455481 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:56.455451 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-7cvlb"] Sep 10 18:43:56.455900 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:56.455884 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a07c0ad1-eeb0-4b33-995b-04e109cc6c78" containerName="authorino" Sep 10 18:43:56.455962 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:56.455906 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a07c0ad1-eeb0-4b33-995b-04e109cc6c78" containerName="authorino" Sep 10 18:43:56.455962 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:56.455920 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ad9685a-5cd2-44bc-a7c4-a0519182a1d6" containerName="limitador" Sep 10 18:43:56.455962 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:56.455927 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ad9685a-5cd2-44bc-a7c4-a0519182a1d6" containerName="limitador" Sep 10 18:43:56.456053 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:56.456023 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ad9685a-5cd2-44bc-a7c4-a0519182a1d6" containerName="limitador" Sep 10 18:43:56.456053 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:56.456037 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a07c0ad1-eeb0-4b33-995b-04e109cc6c78" containerName="authorino" Sep 10 18:43:56.460514 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:56.460496 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7cvlb" Sep 10 18:43:56.462453 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:56.462423 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 10 18:43:56.462453 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:56.462437 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 18:43:56.462645 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:56.462464 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 10 18:43:56.462818 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:56.462804 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-slfg2\"" Sep 10 18:43:56.465402 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:56.465383 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-7cvlb"] Sep 10 18:43:56.488660 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:56.488620 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:43:56.610862 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:56.610835 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq8gw\" (UniqueName: \"kubernetes.io/projected/9de637bc-a885-4dcb-ba47-b6c8be00b70c-kube-api-access-pq8gw\") pod \"postgres-69fb4594cf-7cvlb\" (UID: \"9de637bc-a885-4dcb-ba47-b6c8be00b70c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-7cvlb" Sep 10 18:43:56.610980 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:56.610894 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9de637bc-a885-4dcb-ba47-b6c8be00b70c-data\") pod \"postgres-69fb4594cf-7cvlb\" (UID: \"9de637bc-a885-4dcb-ba47-b6c8be00b70c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-7cvlb" Sep 10 18:43:56.711711 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:56.711652 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq8gw\" (UniqueName: \"kubernetes.io/projected/9de637bc-a885-4dcb-ba47-b6c8be00b70c-kube-api-access-pq8gw\") pod \"postgres-69fb4594cf-7cvlb\" (UID: \"9de637bc-a885-4dcb-ba47-b6c8be00b70c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-7cvlb" Sep 10 18:43:56.711790 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:56.711723 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9de637bc-a885-4dcb-ba47-b6c8be00b70c-data\") pod \"postgres-69fb4594cf-7cvlb\" (UID: \"9de637bc-a885-4dcb-ba47-b6c8be00b70c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-7cvlb" Sep 10 18:43:56.712079 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:56.712063 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9de637bc-a885-4dcb-ba47-b6c8be00b70c-data\") pod \"postgres-69fb4594cf-7cvlb\" (UID: \"9de637bc-a885-4dcb-ba47-b6c8be00b70c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-7cvlb" Sep 10 18:43:56.719029 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:56.719006 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq8gw\" (UniqueName: \"kubernetes.io/projected/9de637bc-a885-4dcb-ba47-b6c8be00b70c-kube-api-access-pq8gw\") pod \"postgres-69fb4594cf-7cvlb\" (UID: \"9de637bc-a885-4dcb-ba47-b6c8be00b70c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-7cvlb" Sep 10 18:43:56.771206 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:56.771186 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7cvlb" Sep 10 18:43:57.041084 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:57.041057 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-7cvlb"] Sep 10 18:43:57.044019 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:43:57.043994 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de637bc_a885_4dcb_ba47_b6c8be00b70c.slice/crio-eb4bf769c798ee0037b018bcf05b4641da38f006c8c9103e0cef9e2a663873ed WatchSource:0}: Error finding container eb4bf769c798ee0037b018bcf05b4641da38f006c8c9103e0cef9e2a663873ed: Status 404 returned error can't find the container with id eb4bf769c798ee0037b018bcf05b4641da38f006c8c9103e0cef9e2a663873ed Sep 10 18:43:58.046560 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:43:58.046521 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7cvlb" event={"ID":"9de637bc-a885-4dcb-ba47-b6c8be00b70c","Type":"ContainerStarted","Data":"eb4bf769c798ee0037b018bcf05b4641da38f006c8c9103e0cef9e2a663873ed"} Sep 10 18:44:00.061163 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:00.061130 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7cvlb" event={"ID":"9de637bc-a885-4dcb-ba47-b6c8be00b70c","Type":"ContainerStarted","Data":"1987256d4d2c27fb657062e38b03fe06e41060db65f9418816485010ff512851"} Sep 10 18:44:00.061545 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:00.061215 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7cvlb" Sep 10 18:44:00.073836 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:00.073791 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7cvlb" podStartSLOduration=2.125603852 podStartE2EDuration="4.073772698s" totalImagesPullingTime="1.948168846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:43:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:44:00.073130452 +0000 UTC m=+766.159467963" watchObservedRunningTime="2026-09-10 18:44:00.073772698 +0000 UTC m=+766.160110173" Sep 10 18:44:06.093384 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:06.093349 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7cvlb" Sep 10 18:44:06.378753 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:06.378664 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:44:06.602714 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:06.602681 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-wj2mz"] Sep 10 18:44:06.607444 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:06.607422 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-wj2mz" Sep 10 18:44:06.609073 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:06.609057 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-v49zl\"" Sep 10 18:44:06.611728 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:06.611707 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-wj2mz"] Sep 10 18:44:06.630912 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:06.630851 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:44:06.678249 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:06.678224 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-wj2mz"] Sep 10 18:44:06.678501 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:44:06.678482 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-9ckhz], unattached volumes=[], failed to process volumes=[kube-api-access-9ckhz]: context canceled" pod="kuadrant-system/authorino-557f876f69-wj2mz" podUID="6752d966-ea1d-4749-81c7-2d549f03a456" Sep 10 18:44:06.702742 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:06.702717 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-xlhm5"] Sep 10 18:44:06.709710 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:06.709686 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-xlhm5" Sep 10 18:44:06.711434 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:06.711415 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 10 18:44:06.712204 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:06.712182 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-xlhm5"] Sep 10 18:44:06.789088 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:06.789058 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ckhz\" (UniqueName: \"kubernetes.io/projected/6752d966-ea1d-4749-81c7-2d549f03a456-kube-api-access-9ckhz\") pod \"authorino-557f876f69-wj2mz\" (UID: \"6752d966-ea1d-4749-81c7-2d549f03a456\") " pod="kuadrant-system/authorino-557f876f69-wj2mz" Sep 10 18:44:06.832613 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:06.832586 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-xlhm5"] Sep 10 18:44:06.832807 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:44:06.832786 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-pccm7 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-xlhm5" podUID="5abb7d39-f0ae-4244-b6fb-aa4c6f500ebf" Sep 10 18:44:06.864434 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:06.864404 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-f4bb7cdb7-4tqbj"] Sep 10 18:44:06.867556 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:06.867542 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f4bb7cdb7-4tqbj" Sep 10 18:44:06.874342 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:06.874304 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f4bb7cdb7-4tqbj"] Sep 10 18:44:06.890266 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:06.890209 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pccm7\" (UniqueName: \"kubernetes.io/projected/5abb7d39-f0ae-4244-b6fb-aa4c6f500ebf-kube-api-access-pccm7\") pod \"authorino-6d4dfc649f-xlhm5\" (UID: \"5abb7d39-f0ae-4244-b6fb-aa4c6f500ebf\") " pod="kuadrant-system/authorino-6d4dfc649f-xlhm5" Sep 10 18:44:06.890401 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:06.890268 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5abb7d39-f0ae-4244-b6fb-aa4c6f500ebf-tls-cert\") pod \"authorino-6d4dfc649f-xlhm5\" (UID: \"5abb7d39-f0ae-4244-b6fb-aa4c6f500ebf\") " pod="kuadrant-system/authorino-6d4dfc649f-xlhm5" Sep 10 18:44:06.890401 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:06.890364 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ckhz\" (UniqueName: \"kubernetes.io/projected/6752d966-ea1d-4749-81c7-2d549f03a456-kube-api-access-9ckhz\") pod \"authorino-557f876f69-wj2mz\" (UID: \"6752d966-ea1d-4749-81c7-2d549f03a456\") " pod="kuadrant-system/authorino-557f876f69-wj2mz" Sep 10 18:44:06.897278 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:06.897256 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ckhz\" (UniqueName: \"kubernetes.io/projected/6752d966-ea1d-4749-81c7-2d549f03a456-kube-api-access-9ckhz\") pod \"authorino-557f876f69-wj2mz\" (UID: \"6752d966-ea1d-4749-81c7-2d549f03a456\") " pod="kuadrant-system/authorino-557f876f69-wj2mz" Sep 10 18:44:06.991495 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:06.991462 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pccm7\" (UniqueName: \"kubernetes.io/projected/5abb7d39-f0ae-4244-b6fb-aa4c6f500ebf-kube-api-access-pccm7\") pod \"authorino-6d4dfc649f-xlhm5\" (UID: \"5abb7d39-f0ae-4244-b6fb-aa4c6f500ebf\") " pod="kuadrant-system/authorino-6d4dfc649f-xlhm5" Sep 10 18:44:06.991630 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:06.991578 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/adefb8c0-d030-424a-ac8c-a5723f66d6e3-tls-cert\") pod \"authorino-f4bb7cdb7-4tqbj\" (UID: \"adefb8c0-d030-424a-ac8c-a5723f66d6e3\") " pod="kuadrant-system/authorino-f4bb7cdb7-4tqbj" Sep 10 18:44:06.991704 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:06.991687 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5abb7d39-f0ae-4244-b6fb-aa4c6f500ebf-tls-cert\") pod \"authorino-6d4dfc649f-xlhm5\" (UID: \"5abb7d39-f0ae-4244-b6fb-aa4c6f500ebf\") " pod="kuadrant-system/authorino-6d4dfc649f-xlhm5" Sep 10 18:44:06.991746 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:06.991731 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4mfq\" (UniqueName: \"kubernetes.io/projected/adefb8c0-d030-424a-ac8c-a5723f66d6e3-kube-api-access-w4mfq\") pod \"authorino-f4bb7cdb7-4tqbj\" (UID: \"adefb8c0-d030-424a-ac8c-a5723f66d6e3\") " pod="kuadrant-system/authorino-f4bb7cdb7-4tqbj" Sep 10 18:44:06.993994 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:06.993970 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5abb7d39-f0ae-4244-b6fb-aa4c6f500ebf-tls-cert\") pod \"authorino-6d4dfc649f-xlhm5\" (UID: \"5abb7d39-f0ae-4244-b6fb-aa4c6f500ebf\") " pod="kuadrant-system/authorino-6d4dfc649f-xlhm5" Sep 10 18:44:06.998487 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:06.998461 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pccm7\" (UniqueName: \"kubernetes.io/projected/5abb7d39-f0ae-4244-b6fb-aa4c6f500ebf-kube-api-access-pccm7\") pod \"authorino-6d4dfc649f-xlhm5\" (UID: \"5abb7d39-f0ae-4244-b6fb-aa4c6f500ebf\") " pod="kuadrant-system/authorino-6d4dfc649f-xlhm5" Sep 10 18:44:07.090138 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:07.090107 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-xlhm5" Sep 10 18:44:07.090351 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:07.090121 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-wj2mz" Sep 10 18:44:07.092435 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:07.092413 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/adefb8c0-d030-424a-ac8c-a5723f66d6e3-tls-cert\") pod \"authorino-f4bb7cdb7-4tqbj\" (UID: \"adefb8c0-d030-424a-ac8c-a5723f66d6e3\") " pod="kuadrant-system/authorino-f4bb7cdb7-4tqbj" Sep 10 18:44:07.092576 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:07.092512 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4mfq\" (UniqueName: \"kubernetes.io/projected/adefb8c0-d030-424a-ac8c-a5723f66d6e3-kube-api-access-w4mfq\") pod \"authorino-f4bb7cdb7-4tqbj\" (UID: \"adefb8c0-d030-424a-ac8c-a5723f66d6e3\") " pod="kuadrant-system/authorino-f4bb7cdb7-4tqbj" Sep 10 18:44:07.095171 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:07.095152 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/adefb8c0-d030-424a-ac8c-a5723f66d6e3-tls-cert\") pod \"authorino-f4bb7cdb7-4tqbj\" (UID: \"adefb8c0-d030-424a-ac8c-a5723f66d6e3\") " pod="kuadrant-system/authorino-f4bb7cdb7-4tqbj" Sep 10 18:44:07.097501 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:07.097481 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-xlhm5" Sep 10 18:44:07.099470 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:07.099450 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4mfq\" (UniqueName: \"kubernetes.io/projected/adefb8c0-d030-424a-ac8c-a5723f66d6e3-kube-api-access-w4mfq\") pod \"authorino-f4bb7cdb7-4tqbj\" (UID: \"adefb8c0-d030-424a-ac8c-a5723f66d6e3\") " pod="kuadrant-system/authorino-f4bb7cdb7-4tqbj" Sep 10 18:44:07.113496 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:07.113472 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-wj2mz" Sep 10 18:44:07.176505 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:07.176417 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f4bb7cdb7-4tqbj" Sep 10 18:44:07.193317 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:07.193293 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5abb7d39-f0ae-4244-b6fb-aa4c6f500ebf-tls-cert\") pod \"5abb7d39-f0ae-4244-b6fb-aa4c6f500ebf\" (UID: \"5abb7d39-f0ae-4244-b6fb-aa4c6f500ebf\") " Sep 10 18:44:07.193450 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:07.193362 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pccm7\" (UniqueName: \"kubernetes.io/projected/5abb7d39-f0ae-4244-b6fb-aa4c6f500ebf-kube-api-access-pccm7\") pod \"5abb7d39-f0ae-4244-b6fb-aa4c6f500ebf\" (UID: \"5abb7d39-f0ae-4244-b6fb-aa4c6f500ebf\") " Sep 10 18:44:07.195420 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:07.195371 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5abb7d39-f0ae-4244-b6fb-aa4c6f500ebf-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "5abb7d39-f0ae-4244-b6fb-aa4c6f500ebf" (UID: "5abb7d39-f0ae-4244-b6fb-aa4c6f500ebf"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 18:44:07.195533 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:07.195509 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5abb7d39-f0ae-4244-b6fb-aa4c6f500ebf-kube-api-access-pccm7" (OuterVolumeSpecName: "kube-api-access-pccm7") pod "5abb7d39-f0ae-4244-b6fb-aa4c6f500ebf" (UID: "5abb7d39-f0ae-4244-b6fb-aa4c6f500ebf"). InnerVolumeSpecName "kube-api-access-pccm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 18:44:07.294200 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:07.294161 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ckhz\" (UniqueName: \"kubernetes.io/projected/6752d966-ea1d-4749-81c7-2d549f03a456-kube-api-access-9ckhz\") pod \"6752d966-ea1d-4749-81c7-2d549f03a456\" (UID: \"6752d966-ea1d-4749-81c7-2d549f03a456\") " Sep 10 18:44:07.294440 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:07.294419 2581 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5abb7d39-f0ae-4244-b6fb-aa4c6f500ebf-tls-cert\") on node \"ip-10-0-134-166.ec2.internal\" DevicePath \"\"" Sep 10 18:44:07.294440 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:07.294438 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pccm7\" (UniqueName: \"kubernetes.io/projected/5abb7d39-f0ae-4244-b6fb-aa4c6f500ebf-kube-api-access-pccm7\") on node \"ip-10-0-134-166.ec2.internal\" DevicePath \"\"" Sep 10 18:44:07.296105 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:07.296084 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6752d966-ea1d-4749-81c7-2d549f03a456-kube-api-access-9ckhz" (OuterVolumeSpecName: "kube-api-access-9ckhz") pod "6752d966-ea1d-4749-81c7-2d549f03a456" (UID: "6752d966-ea1d-4749-81c7-2d549f03a456"). InnerVolumeSpecName "kube-api-access-9ckhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 18:44:07.395051 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:07.395020 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ckhz\" (UniqueName: \"kubernetes.io/projected/6752d966-ea1d-4749-81c7-2d549f03a456-kube-api-access-9ckhz\") on node \"ip-10-0-134-166.ec2.internal\" DevicePath \"\"" Sep 10 18:44:07.440591 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:07.440570 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f4bb7cdb7-4tqbj"] Sep 10 18:44:07.442903 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:44:07.442881 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podadefb8c0_d030_424a_ac8c_a5723f66d6e3.slice/crio-f7d85db9f31233c177586f3c12f37ebc0f5cb9414085999c641586fea589c14a WatchSource:0}: Error finding container f7d85db9f31233c177586f3c12f37ebc0f5cb9414085999c641586fea589c14a: Status 404 returned error can't find the container with id f7d85db9f31233c177586f3c12f37ebc0f5cb9414085999c641586fea589c14a Sep 10 18:44:08.096309 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:08.096281 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f4bb7cdb7-4tqbj" event={"ID":"adefb8c0-d030-424a-ac8c-a5723f66d6e3","Type":"ContainerStarted","Data":"f7d85db9f31233c177586f3c12f37ebc0f5cb9414085999c641586fea589c14a"} Sep 10 18:44:08.096640 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:08.096308 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-xlhm5" Sep 10 18:44:08.096640 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:08.096308 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-wj2mz" Sep 10 18:44:08.123708 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:08.123687 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-wj2mz"] Sep 10 18:44:08.126793 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:08.126770 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-wj2mz"] Sep 10 18:44:08.144525 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:08.144499 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-xlhm5"] Sep 10 18:44:08.147638 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:08.147617 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-xlhm5"] Sep 10 18:44:08.379816 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:08.379785 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5abb7d39-f0ae-4244-b6fb-aa4c6f500ebf" path="/var/lib/kubelet/pods/5abb7d39-f0ae-4244-b6fb-aa4c6f500ebf/volumes" Sep 10 18:44:08.379994 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:08.379982 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6752d966-ea1d-4749-81c7-2d549f03a456" path="/var/lib/kubelet/pods/6752d966-ea1d-4749-81c7-2d549f03a456/volumes" Sep 10 18:44:09.102427 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:09.102382 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f4bb7cdb7-4tqbj" event={"ID":"adefb8c0-d030-424a-ac8c-a5723f66d6e3","Type":"ContainerStarted","Data":"15762e023473af274b2c863630c8ca9c733ff65a849779e928fabd8e65ae52be"} Sep 10 18:44:09.115782 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:09.115735 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-f4bb7cdb7-4tqbj" podStartSLOduration=2.506953149 podStartE2EDuration="3.115720245s" totalImagesPullingTime="608.767096ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:44:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:44:09.114897254 +0000 UTC m=+775.201234727" watchObservedRunningTime="2026-09-10 18:44:09.115720245 +0000 UTC m=+775.202057718" Sep 10 18:44:13.786154 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:13.786122 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:44:13.881441 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:13.881410 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:44:13.936121 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:13.936096 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:44:14.361255 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:14.361232 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:44:17.984756 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:17.984717 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:44:18.100122 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:18.100088 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:44:51.250096 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:51.250018 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:44:51.349282 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:51.349251 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:44:52.716199 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:52.716162 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:44:52.817449 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:52.817417 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:44:52.981203 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:52.981126 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-79bbc6b8-xl5l8"] Sep 10 18:44:52.983411 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:52.983395 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79bbc6b8-xl5l8" Sep 10 18:44:52.985264 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:52.985248 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 10 18:44:52.985453 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:52.985438 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 10 18:44:52.985544 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:52.985528 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-g48qc\"" Sep 10 18:44:52.994007 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:52.993985 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79bbc6b8-xl5l8"] Sep 10 18:44:53.014771 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:53.014745 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6987549-3bf4-4234-bb10-8b92064a64d7-openshift-service-ca\") pod \"maas-api-79bbc6b8-xl5l8\" (UID: \"c6987549-3bf4-4234-bb10-8b92064a64d7\") " pod="odh-ai-gateway-infra/maas-api-79bbc6b8-xl5l8" Sep 10 18:44:53.014933 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:53.014783 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6987549-3bf4-4234-bb10-8b92064a64d7-maas-api-tls\") pod \"maas-api-79bbc6b8-xl5l8\" (UID: \"c6987549-3bf4-4234-bb10-8b92064a64d7\") " pod="odh-ai-gateway-infra/maas-api-79bbc6b8-xl5l8" Sep 10 18:44:53.014933 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:53.014820 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6987549-3bf4-4234-bb10-8b92064a64d7-metrics-certs\") pod \"maas-api-79bbc6b8-xl5l8\" (UID: \"c6987549-3bf4-4234-bb10-8b92064a64d7\") " pod="odh-ai-gateway-infra/maas-api-79bbc6b8-xl5l8" Sep 10 18:44:53.014933 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:53.014886 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gnml\" (UniqueName: \"kubernetes.io/projected/c6987549-3bf4-4234-bb10-8b92064a64d7-kube-api-access-7gnml\") pod \"maas-api-79bbc6b8-xl5l8\" (UID: \"c6987549-3bf4-4234-bb10-8b92064a64d7\") " pod="odh-ai-gateway-infra/maas-api-79bbc6b8-xl5l8" Sep 10 18:44:53.085666 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:53.085631 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:44:53.116079 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:53.116047 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6987549-3bf4-4234-bb10-8b92064a64d7-maas-api-tls\") pod \"maas-api-79bbc6b8-xl5l8\" (UID: \"c6987549-3bf4-4234-bb10-8b92064a64d7\") " pod="odh-ai-gateway-infra/maas-api-79bbc6b8-xl5l8" Sep 10 18:44:53.116263 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:53.116102 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6987549-3bf4-4234-bb10-8b92064a64d7-metrics-certs\") pod \"maas-api-79bbc6b8-xl5l8\" (UID: \"c6987549-3bf4-4234-bb10-8b92064a64d7\") " pod="odh-ai-gateway-infra/maas-api-79bbc6b8-xl5l8" Sep 10 18:44:53.116263 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:53.116127 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gnml\" (UniqueName: \"kubernetes.io/projected/c6987549-3bf4-4234-bb10-8b92064a64d7-kube-api-access-7gnml\") pod \"maas-api-79bbc6b8-xl5l8\" (UID: \"c6987549-3bf4-4234-bb10-8b92064a64d7\") " pod="odh-ai-gateway-infra/maas-api-79bbc6b8-xl5l8" Sep 10 18:44:53.116263 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:44:53.116200 2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found Sep 10 18:44:53.116263 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:44:53.116264 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6987549-3bf4-4234-bb10-8b92064a64d7-metrics-certs podName:c6987549-3bf4-4234-bb10-8b92064a64d7 nodeName:}" failed. No retries permitted until 2026-09-10 18:44:53.616241896 +0000 UTC m=+819.702579349 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6987549-3bf4-4234-bb10-8b92064a64d7-metrics-certs") pod "maas-api-79bbc6b8-xl5l8" (UID: "c6987549-3bf4-4234-bb10-8b92064a64d7") : secret "maas-api-metrics-tls" not found Sep 10 18:44:53.116531 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:53.116286 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6987549-3bf4-4234-bb10-8b92064a64d7-openshift-service-ca\") pod \"maas-api-79bbc6b8-xl5l8\" (UID: \"c6987549-3bf4-4234-bb10-8b92064a64d7\") " pod="odh-ai-gateway-infra/maas-api-79bbc6b8-xl5l8" Sep 10 18:44:53.116913 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:53.116894 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6987549-3bf4-4234-bb10-8b92064a64d7-openshift-service-ca\") pod \"maas-api-79bbc6b8-xl5l8\" (UID: \"c6987549-3bf4-4234-bb10-8b92064a64d7\") " pod="odh-ai-gateway-infra/maas-api-79bbc6b8-xl5l8" Sep 10 18:44:53.118626 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:53.118604 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6987549-3bf4-4234-bb10-8b92064a64d7-maas-api-tls\") pod \"maas-api-79bbc6b8-xl5l8\" (UID: \"c6987549-3bf4-4234-bb10-8b92064a64d7\") " pod="odh-ai-gateway-infra/maas-api-79bbc6b8-xl5l8" Sep 10 18:44:53.123692 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:53.123672 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gnml\" (UniqueName: \"kubernetes.io/projected/c6987549-3bf4-4234-bb10-8b92064a64d7-kube-api-access-7gnml\") pod \"maas-api-79bbc6b8-xl5l8\" (UID: \"c6987549-3bf4-4234-bb10-8b92064a64d7\") " pod="odh-ai-gateway-infra/maas-api-79bbc6b8-xl5l8" Sep 10 18:44:53.388591 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:53.388561 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:44:53.484586 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:53.484547 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:44:53.559523 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:53.559493 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:44:53.621789 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:53.621755 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6987549-3bf4-4234-bb10-8b92064a64d7-metrics-certs\") pod \"maas-api-79bbc6b8-xl5l8\" (UID: \"c6987549-3bf4-4234-bb10-8b92064a64d7\") " pod="odh-ai-gateway-infra/maas-api-79bbc6b8-xl5l8" Sep 10 18:44:53.624470 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:53.624446 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6987549-3bf4-4234-bb10-8b92064a64d7-metrics-certs\") pod \"maas-api-79bbc6b8-xl5l8\" (UID: \"c6987549-3bf4-4234-bb10-8b92064a64d7\") " pod="odh-ai-gateway-infra/maas-api-79bbc6b8-xl5l8" Sep 10 18:44:53.894729 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:53.894688 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79bbc6b8-xl5l8" Sep 10 18:44:54.161355 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:54.161284 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79bbc6b8-xl5l8"] Sep 10 18:44:54.163561 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:44:54.163530 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6987549_3bf4_4234_bb10_8b92064a64d7.slice/crio-5357021953d62fd5ea5474ffda1a7b7d2c6f50850fd71240a45d8e6fd442e642 WatchSource:0}: Error finding container 5357021953d62fd5ea5474ffda1a7b7d2c6f50850fd71240a45d8e6fd442e642: Status 404 returned error can't find the container with id 5357021953d62fd5ea5474ffda1a7b7d2c6f50850fd71240a45d8e6fd442e642 Sep 10 18:44:54.295484 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:54.295449 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79bbc6b8-xl5l8" event={"ID":"c6987549-3bf4-4234-bb10-8b92064a64d7","Type":"ContainerStarted","Data":"5357021953d62fd5ea5474ffda1a7b7d2c6f50850fd71240a45d8e6fd442e642"} Sep 10 18:44:57.333720 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:57.333688 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79bbc6b8-xl5l8" event={"ID":"c6987549-3bf4-4234-bb10-8b92064a64d7","Type":"ContainerStarted","Data":"37bd0d58bd13a1618341b638ab23c0b449e5ed39a62a8e489f5172f2e704bc08"} Sep 10 18:44:57.334077 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:57.333793 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-79bbc6b8-xl5l8" Sep 10 18:44:57.349399 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:44:57.349347 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-79bbc6b8-xl5l8" podStartSLOduration=2.358568538 podStartE2EDuration="5.3493115s" totalImagesPullingTime="2.990742962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:44:57.349049921 +0000 UTC m=+823.435387400" watchObservedRunningTime="2026-09-10 18:44:57.3493115 +0000 UTC m=+823.435648975" Sep 10 18:45:03.343626 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:03.343593 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-79bbc6b8-xl5l8" Sep 10 18:45:03.365576 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:03.365542 2581 generic.go:356] "Generic (PLEG): container finished" podID="952dcc01-8c30-41c0-9d9f-2964ed5c0760" containerID="04964628345d7d80865897ec542ba3edc7be0143aed1a38e3c32f52299cddbaa" exitCode=0 Sep 10 18:45:03.365878 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:03.365600 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" event={"ID":"952dcc01-8c30-41c0-9d9f-2964ed5c0760","Type":"ContainerDied","Data":"04964628345d7d80865897ec542ba3edc7be0143aed1a38e3c32f52299cddbaa"} Sep 10 18:45:03.366247 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:03.366222 2581 scope.go:117] "RemoveContainer" containerID="04964628345d7d80865897ec542ba3edc7be0143aed1a38e3c32f52299cddbaa" Sep 10 18:45:03.367456 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:03.367424 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 18:45:03.367590 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:03.367491 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 18:45:03.367590 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:03.367526 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 18:45:04.306018 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:04.305974 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:45:04.372873 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:04.372841 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" event={"ID":"952dcc01-8c30-41c0-9d9f-2964ed5c0760","Type":"ContainerStarted","Data":"2f1782dd45061e961f78f1bdb63cf8b08317dd52cbd29025f209aebb4e70fd84"} Sep 10 18:45:04.399742 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:04.399711 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:45:04.970343 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:04.970295 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" Sep 10 18:45:04.970563 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:04.970538 2581 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-h6mg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Sep 10 18:45:04.970643 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:04.970598 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" podUID="952dcc01-8c30-41c0-9d9f-2964ed5c0760" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Sep 10 18:45:05.970102 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:05.970070 2581 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-h6mg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Sep 10 18:45:05.970505 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:05.970125 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" podUID="952dcc01-8c30-41c0-9d9f-2964ed5c0760" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Sep 10 18:45:06.970689 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:06.970644 2581 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-h6mg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Sep 10 18:45:06.971074 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:06.970711 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" podUID="952dcc01-8c30-41c0-9d9f-2964ed5c0760" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Sep 10 18:45:07.970969 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:07.970937 2581 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-h6mg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Sep 10 18:45:07.971409 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:07.970993 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" podUID="952dcc01-8c30-41c0-9d9f-2964ed5c0760" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Sep 10 18:45:08.970044 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:08.970007 2581 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-h6mg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Sep 10 18:45:08.970212 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:08.970064 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" podUID="952dcc01-8c30-41c0-9d9f-2964ed5c0760" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Sep 10 18:45:09.974713 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:09.974683 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" Sep 10 18:45:09.975131 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:09.974964 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" Sep 10 18:45:09.975565 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:09.975548 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6mg2" Sep 10 18:45:15.039247 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:15.039210 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:45:16.061580 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:16.061550 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76"] Sep 10 18:45:16.064782 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:16.064765 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:45:16.066757 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:16.066725 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 10 18:45:16.066895 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:16.066727 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 10 18:45:16.067448 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:16.067429 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-9zgnh\"" Sep 10 18:45:16.067557 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:16.067432 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 10 18:45:16.073774 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:16.073753 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76"] Sep 10 18:45:16.081651 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:16.081628 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:45:16.113931 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:16.113904 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1f4af20-061e-4e66-b2c1-d14d0c501e01-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-859856b874-jrt76\" (UID: \"e1f4af20-061e-4e66-b2c1-d14d0c501e01\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:45:16.114060 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:16.113959 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1f4af20-061e-4e66-b2c1-d14d0c501e01-home\") pod \"e2e-distinct-2-simulated-kserve-859856b874-jrt76\" (UID: \"e1f4af20-061e-4e66-b2c1-d14d0c501e01\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:45:16.114060 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:16.114000 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1f4af20-061e-4e66-b2c1-d14d0c501e01-model-cache\") pod \"e2e-distinct-2-simulated-kserve-859856b874-jrt76\" (UID: \"e1f4af20-061e-4e66-b2c1-d14d0c501e01\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:45:16.114060 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:16.114042 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1f4af20-061e-4e66-b2c1-d14d0c501e01-dshm\") pod \"e2e-distinct-2-simulated-kserve-859856b874-jrt76\" (UID: \"e1f4af20-061e-4e66-b2c1-d14d0c501e01\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:45:16.114183 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:16.114068 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1f4af20-061e-4e66-b2c1-d14d0c501e01-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-859856b874-jrt76\" (UID: \"e1f4af20-061e-4e66-b2c1-d14d0c501e01\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:45:16.114183 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:16.114104 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t8q7\" (UniqueName: \"kubernetes.io/projected/e1f4af20-061e-4e66-b2c1-d14d0c501e01-kube-api-access-7t8q7\") pod \"e2e-distinct-2-simulated-kserve-859856b874-jrt76\" (UID: \"e1f4af20-061e-4e66-b2c1-d14d0c501e01\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:45:16.214691 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:16.214659 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1f4af20-061e-4e66-b2c1-d14d0c501e01-home\") pod \"e2e-distinct-2-simulated-kserve-859856b874-jrt76\" (UID: \"e1f4af20-061e-4e66-b2c1-d14d0c501e01\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:45:16.214850 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:16.214698 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1f4af20-061e-4e66-b2c1-d14d0c501e01-model-cache\") pod \"e2e-distinct-2-simulated-kserve-859856b874-jrt76\" (UID: \"e1f4af20-061e-4e66-b2c1-d14d0c501e01\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:45:16.214850 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:16.214721 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1f4af20-061e-4e66-b2c1-d14d0c501e01-dshm\") pod \"e2e-distinct-2-simulated-kserve-859856b874-jrt76\" (UID: \"e1f4af20-061e-4e66-b2c1-d14d0c501e01\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:45:16.214996 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:16.214845 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1f4af20-061e-4e66-b2c1-d14d0c501e01-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-859856b874-jrt76\" (UID: \"e1f4af20-061e-4e66-b2c1-d14d0c501e01\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:45:16.214996 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:16.214916 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t8q7\" (UniqueName: \"kubernetes.io/projected/e1f4af20-061e-4e66-b2c1-d14d0c501e01-kube-api-access-7t8q7\") pod \"e2e-distinct-2-simulated-kserve-859856b874-jrt76\" (UID: \"e1f4af20-061e-4e66-b2c1-d14d0c501e01\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:45:16.214996 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:16.214978 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1f4af20-061e-4e66-b2c1-d14d0c501e01-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-859856b874-jrt76\" (UID: \"e1f4af20-061e-4e66-b2c1-d14d0c501e01\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:45:16.215159 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:16.215084 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1f4af20-061e-4e66-b2c1-d14d0c501e01-model-cache\") pod \"e2e-distinct-2-simulated-kserve-859856b874-jrt76\" (UID: \"e1f4af20-061e-4e66-b2c1-d14d0c501e01\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:45:16.215159 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:16.215127 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1f4af20-061e-4e66-b2c1-d14d0c501e01-home\") pod \"e2e-distinct-2-simulated-kserve-859856b874-jrt76\" (UID: \"e1f4af20-061e-4e66-b2c1-d14d0c501e01\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:45:16.215435 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:16.215411 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1f4af20-061e-4e66-b2c1-d14d0c501e01-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-859856b874-jrt76\" (UID: \"e1f4af20-061e-4e66-b2c1-d14d0c501e01\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:45:16.217604 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:16.217563 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1f4af20-061e-4e66-b2c1-d14d0c501e01-dshm\") pod \"e2e-distinct-2-simulated-kserve-859856b874-jrt76\" (UID: \"e1f4af20-061e-4e66-b2c1-d14d0c501e01\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:45:16.217723 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:16.217650 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1f4af20-061e-4e66-b2c1-d14d0c501e01-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-859856b874-jrt76\" (UID: \"e1f4af20-061e-4e66-b2c1-d14d0c501e01\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:45:16.224430 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:16.224408 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t8q7\" (UniqueName: \"kubernetes.io/projected/e1f4af20-061e-4e66-b2c1-d14d0c501e01-kube-api-access-7t8q7\") pod \"e2e-distinct-2-simulated-kserve-859856b874-jrt76\" (UID: \"e1f4af20-061e-4e66-b2c1-d14d0c501e01\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:45:16.377099 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:16.377004 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:45:16.648642 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:16.648616 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76"] Sep 10 18:45:16.651764 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:45:16.651729 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1f4af20_061e_4e66_b2c1_d14d0c501e01.slice/crio-049ebc6fa77989b259bcd0bf075a6840ea653bc4e6bd9bf7b30e8499658a39f0 WatchSource:0}: Error finding container 049ebc6fa77989b259bcd0bf075a6840ea653bc4e6bd9bf7b30e8499658a39f0: Status 404 returned error can't find the container with id 049ebc6fa77989b259bcd0bf075a6840ea653bc4e6bd9bf7b30e8499658a39f0 Sep 10 18:45:17.435406 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:17.435367 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" event={"ID":"e1f4af20-061e-4e66-b2c1-d14d0c501e01","Type":"ContainerStarted","Data":"049ebc6fa77989b259bcd0bf075a6840ea653bc4e6bd9bf7b30e8499658a39f0"} Sep 10 18:45:18.441630 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:18.441585 2581 generic.go:356] "Generic (PLEG): container finished" podID="e1f4af20-061e-4e66-b2c1-d14d0c501e01" containerID="b911ed9b4789f692c7dfe7e1b329925e2003111b9e104138d4f8c9eb9438829e" exitCode=0 Sep 10 18:45:18.442063 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:18.441644 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" event={"ID":"e1f4af20-061e-4e66-b2c1-d14d0c501e01","Type":"ContainerDied","Data":"b911ed9b4789f692c7dfe7e1b329925e2003111b9e104138d4f8c9eb9438829e"} Sep 10 18:45:18.442063 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:18.441897 2581 scope.go:117] "RemoveContainer" containerID="b911ed9b4789f692c7dfe7e1b329925e2003111b9e104138d4f8c9eb9438829e" Sep 10 18:45:19.448212 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:19.448173 2581 generic.go:356] "Generic (PLEG): container finished" podID="e1f4af20-061e-4e66-b2c1-d14d0c501e01" containerID="2e22a53f37f7ed17f9b5e88403c13bac0660b983e88e9a864b782e00ebb3c1e8" exitCode=0 Sep 10 18:45:19.448631 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:19.448242 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" event={"ID":"e1f4af20-061e-4e66-b2c1-d14d0c501e01","Type":"ContainerDied","Data":"2e22a53f37f7ed17f9b5e88403c13bac0660b983e88e9a864b782e00ebb3c1e8"} Sep 10 18:45:19.448631 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:19.448300 2581 scope.go:117] "RemoveContainer" containerID="b911ed9b4789f692c7dfe7e1b329925e2003111b9e104138d4f8c9eb9438829e" Sep 10 18:45:19.448631 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:19.448534 2581 scope.go:117] "RemoveContainer" containerID="2e22a53f37f7ed17f9b5e88403c13bac0660b983e88e9a864b782e00ebb3c1e8" Sep 10 18:45:19.448744 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:45:19.448721 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-jrt76_llm(e1f4af20-061e-4e66-b2c1-d14d0c501e01)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" podUID="e1f4af20-061e-4e66-b2c1-d14d0c501e01" Sep 10 18:45:20.454637 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:20.454601 2581 scope.go:117] "RemoveContainer" containerID="2e22a53f37f7ed17f9b5e88403c13bac0660b983e88e9a864b782e00ebb3c1e8" Sep 10 18:45:20.455008 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:45:20.454750 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-jrt76_llm(e1f4af20-061e-4e66-b2c1-d14d0c501e01)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" podUID="e1f4af20-061e-4e66-b2c1-d14d0c501e01" Sep 10 18:45:26.380824 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:26.380791 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:45:26.381465 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:26.380848 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:45:26.381465 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:26.380867 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:45:26.381465 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:26.380883 2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:45:26.381465 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:26.381295 2581 scope.go:117] "RemoveContainer" containerID="2e22a53f37f7ed17f9b5e88403c13bac0660b983e88e9a864b782e00ebb3c1e8" Sep 10 18:45:26.381702 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:45:26.381516 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-jrt76_llm(e1f4af20-061e-4e66-b2c1-d14d0c501e01)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" podUID="e1f4af20-061e-4e66-b2c1-d14d0c501e01" Sep 10 18:45:32.864189 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:32.864146 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-6685d86975-lgkff"] Sep 10 18:45:32.867119 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:32.867095 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:45:32.869342 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:32.869292 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 10 18:45:32.878699 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:32.878677 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-6685d86975-lgkff"] Sep 10 18:45:32.895233 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:32.895205 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:45:32.944056 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:32.944028 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/894e00dc-93dd-4245-846f-3529d796f688-model-cache\") pod \"e2e-distinct-simulated-kserve-6685d86975-lgkff\" (UID: \"894e00dc-93dd-4245-846f-3529d796f688\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:45:32.944217 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:32.944063 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/894e00dc-93dd-4245-846f-3529d796f688-tls-certs\") pod \"e2e-distinct-simulated-kserve-6685d86975-lgkff\" (UID: \"894e00dc-93dd-4245-846f-3529d796f688\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:45:32.944217 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:32.944087 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5w26\" (UniqueName: \"kubernetes.io/projected/894e00dc-93dd-4245-846f-3529d796f688-kube-api-access-w5w26\") pod \"e2e-distinct-simulated-kserve-6685d86975-lgkff\" (UID: \"894e00dc-93dd-4245-846f-3529d796f688\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:45:32.944217 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:32.944185 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/894e00dc-93dd-4245-846f-3529d796f688-dshm\") pod \"e2e-distinct-simulated-kserve-6685d86975-lgkff\" (UID: \"894e00dc-93dd-4245-846f-3529d796f688\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:45:32.944412 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:32.944217 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/894e00dc-93dd-4245-846f-3529d796f688-tmp-dir\") pod \"e2e-distinct-simulated-kserve-6685d86975-lgkff\" (UID: \"894e00dc-93dd-4245-846f-3529d796f688\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:45:32.944412 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:32.944369 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/894e00dc-93dd-4245-846f-3529d796f688-home\") pod \"e2e-distinct-simulated-kserve-6685d86975-lgkff\" (UID: \"894e00dc-93dd-4245-846f-3529d796f688\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:45:33.045630 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:33.045598 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/894e00dc-93dd-4245-846f-3529d796f688-tls-certs\") pod \"e2e-distinct-simulated-kserve-6685d86975-lgkff\" (UID: \"894e00dc-93dd-4245-846f-3529d796f688\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:45:33.045812 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:33.045641 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5w26\" (UniqueName: \"kubernetes.io/projected/894e00dc-93dd-4245-846f-3529d796f688-kube-api-access-w5w26\") pod \"e2e-distinct-simulated-kserve-6685d86975-lgkff\" (UID: \"894e00dc-93dd-4245-846f-3529d796f688\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:45:33.045812 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:33.045671 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/894e00dc-93dd-4245-846f-3529d796f688-dshm\") pod \"e2e-distinct-simulated-kserve-6685d86975-lgkff\" (UID: \"894e00dc-93dd-4245-846f-3529d796f688\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:45:33.045812 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:33.045686 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/894e00dc-93dd-4245-846f-3529d796f688-tmp-dir\") pod \"e2e-distinct-simulated-kserve-6685d86975-lgkff\" (UID: \"894e00dc-93dd-4245-846f-3529d796f688\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:45:33.045812 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:33.045734 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/894e00dc-93dd-4245-846f-3529d796f688-home\") pod \"e2e-distinct-simulated-kserve-6685d86975-lgkff\" (UID: \"894e00dc-93dd-4245-846f-3529d796f688\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:45:33.045812 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:33.045753 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/894e00dc-93dd-4245-846f-3529d796f688-model-cache\") pod \"e2e-distinct-simulated-kserve-6685d86975-lgkff\" (UID: \"894e00dc-93dd-4245-846f-3529d796f688\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:45:33.046163 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:33.046139 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/894e00dc-93dd-4245-846f-3529d796f688-model-cache\") pod \"e2e-distinct-simulated-kserve-6685d86975-lgkff\" (UID: \"894e00dc-93dd-4245-846f-3529d796f688\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:45:33.046232 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:33.046215 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/894e00dc-93dd-4245-846f-3529d796f688-home\") pod \"e2e-distinct-simulated-kserve-6685d86975-lgkff\" (UID: \"894e00dc-93dd-4245-846f-3529d796f688\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:45:33.046282 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:33.046213 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/894e00dc-93dd-4245-846f-3529d796f688-tmp-dir\") pod \"e2e-distinct-simulated-kserve-6685d86975-lgkff\" (UID: \"894e00dc-93dd-4245-846f-3529d796f688\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:45:33.048083 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:33.048059 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/894e00dc-93dd-4245-846f-3529d796f688-dshm\") pod \"e2e-distinct-simulated-kserve-6685d86975-lgkff\" (UID: \"894e00dc-93dd-4245-846f-3529d796f688\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:45:33.048314 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:33.048299 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/894e00dc-93dd-4245-846f-3529d796f688-tls-certs\") pod \"e2e-distinct-simulated-kserve-6685d86975-lgkff\" (UID: \"894e00dc-93dd-4245-846f-3529d796f688\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:45:33.052625 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:33.052604 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5w26\" (UniqueName: \"kubernetes.io/projected/894e00dc-93dd-4245-846f-3529d796f688-kube-api-access-w5w26\") pod \"e2e-distinct-simulated-kserve-6685d86975-lgkff\" (UID: \"894e00dc-93dd-4245-846f-3529d796f688\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:45:33.177668 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:33.177601 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:45:33.445229 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:33.445205 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-6685d86975-lgkff"] Sep 10 18:45:33.447140 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:45:33.447114 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod894e00dc_93dd_4245_846f_3529d796f688.slice/crio-a2ae47708ef18c1c98bdd7cf4d65ba9d476305930ffc86b53dcbfe006e106fc9 WatchSource:0}: Error finding container a2ae47708ef18c1c98bdd7cf4d65ba9d476305930ffc86b53dcbfe006e106fc9: Status 404 returned error can't find the container with id a2ae47708ef18c1c98bdd7cf4d65ba9d476305930ffc86b53dcbfe006e106fc9 Sep 10 18:45:33.512395 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:33.512361 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" event={"ID":"894e00dc-93dd-4245-846f-3529d796f688","Type":"ContainerStarted","Data":"a2ae47708ef18c1c98bdd7cf4d65ba9d476305930ffc86b53dcbfe006e106fc9"} Sep 10 18:45:34.518086 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:34.518058 2581 generic.go:356] "Generic (PLEG): container finished" podID="894e00dc-93dd-4245-846f-3529d796f688" containerID="5db954dde69f878dcbff5e4a3e0c70f541626cd9c1dbd6eb0c007790d701300f" exitCode=0 Sep 10 18:45:34.518519 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:34.518146 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" event={"ID":"894e00dc-93dd-4245-846f-3529d796f688","Type":"ContainerDied","Data":"5db954dde69f878dcbff5e4a3e0c70f541626cd9c1dbd6eb0c007790d701300f"} Sep 10 18:45:34.518519 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:34.518386 2581 scope.go:117] "RemoveContainer" containerID="5db954dde69f878dcbff5e4a3e0c70f541626cd9c1dbd6eb0c007790d701300f" Sep 10 18:45:35.524064 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:35.524035 2581 generic.go:356] "Generic (PLEG): container finished" podID="894e00dc-93dd-4245-846f-3529d796f688" containerID="e2a268f88cf6d1a682cf1ee05ef94cd160f835d6651fa711d0cfbcde3807f107" exitCode=0 Sep 10 18:45:35.524465 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:35.524102 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" event={"ID":"894e00dc-93dd-4245-846f-3529d796f688","Type":"ContainerDied","Data":"e2a268f88cf6d1a682cf1ee05ef94cd160f835d6651fa711d0cfbcde3807f107"} Sep 10 18:45:35.524465 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:35.524149 2581 scope.go:117] "RemoveContainer" containerID="5db954dde69f878dcbff5e4a3e0c70f541626cd9c1dbd6eb0c007790d701300f" Sep 10 18:45:35.524465 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:35.524354 2581 scope.go:117] "RemoveContainer" containerID="e2a268f88cf6d1a682cf1ee05ef94cd160f835d6651fa711d0cfbcde3807f107" Sep 10 18:45:35.524625 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:45:35.524551 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-lgkff_llm(894e00dc-93dd-4245-846f-3529d796f688)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" podUID="894e00dc-93dd-4245-846f-3529d796f688" Sep 10 18:45:36.530344 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:36.530299 2581 scope.go:117] "RemoveContainer" containerID="e2a268f88cf6d1a682cf1ee05ef94cd160f835d6651fa711d0cfbcde3807f107" Sep 10 18:45:36.530830 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:45:36.530544 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-lgkff_llm(894e00dc-93dd-4245-846f-3529d796f688)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" podUID="894e00dc-93dd-4245-846f-3529d796f688" Sep 10 18:45:40.374820 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:40.374787 2581 scope.go:117] "RemoveContainer" containerID="2e22a53f37f7ed17f9b5e88403c13bac0660b983e88e9a864b782e00ebb3c1e8" Sep 10 18:45:41.551753 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:41.551719 2581 generic.go:356] "Generic (PLEG): container finished" podID="e1f4af20-061e-4e66-b2c1-d14d0c501e01" containerID="6e9f567e2ecd7362e2ff7c57616058a0713f17e676b776e5ae50fbff17d3d4d1" exitCode=0 Sep 10 18:45:41.552182 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:41.551793 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" event={"ID":"e1f4af20-061e-4e66-b2c1-d14d0c501e01","Type":"ContainerDied","Data":"6e9f567e2ecd7362e2ff7c57616058a0713f17e676b776e5ae50fbff17d3d4d1"} Sep 10 18:45:41.552182 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:41.551850 2581 scope.go:117] "RemoveContainer" containerID="2e22a53f37f7ed17f9b5e88403c13bac0660b983e88e9a864b782e00ebb3c1e8" Sep 10 18:45:41.552182 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:41.552133 2581 scope.go:117] "RemoveContainer" containerID="6e9f567e2ecd7362e2ff7c57616058a0713f17e676b776e5ae50fbff17d3d4d1" Sep 10 18:45:41.552315 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:45:41.552298 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-jrt76_llm(e1f4af20-061e-4e66-b2c1-d14d0c501e01)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" podUID="e1f4af20-061e-4e66-b2c1-d14d0c501e01" Sep 10 18:45:43.177869 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:43.177826 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:45:43.178224 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:43.177918 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:45:43.178224 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:43.177936 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:45:43.178224 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:43.177948 2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:45:43.178381 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:43.178262 2581 scope.go:117] "RemoveContainer" containerID="e2a268f88cf6d1a682cf1ee05ef94cd160f835d6651fa711d0cfbcde3807f107" Sep 10 18:45:43.178457 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:45:43.178437 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-lgkff_llm(894e00dc-93dd-4245-846f-3529d796f688)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" podUID="894e00dc-93dd-4245-846f-3529d796f688" Sep 10 18:45:46.253255 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:46.253210 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:45:46.380226 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:46.380196 2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:45:46.380390 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:46.380246 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:45:46.380390 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:46.380261 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:45:46.380390 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:46.380275 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:45:46.380607 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:46.380595 2581 scope.go:117] "RemoveContainer" containerID="6e9f567e2ecd7362e2ff7c57616058a0713f17e676b776e5ae50fbff17d3d4d1" Sep 10 18:45:46.380779 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:45:46.380758 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-jrt76_llm(e1f4af20-061e-4e66-b2c1-d14d0c501e01)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" podUID="e1f4af20-061e-4e66-b2c1-d14d0c501e01" Sep 10 18:45:54.377638 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:54.377600 2581 scope.go:117] "RemoveContainer" containerID="e2a268f88cf6d1a682cf1ee05ef94cd160f835d6651fa711d0cfbcde3807f107" Sep 10 18:45:54.619666 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:54.619640 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" event={"ID":"894e00dc-93dd-4245-846f-3529d796f688","Type":"ContainerStarted","Data":"9f9e902b857119769d5fd9d77756ded4783cf6d220ee29df5dacac9cdba7030e"} Sep 10 18:45:54.636267 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:54.636223 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" podStartSLOduration=22.127936809 podStartE2EDuration="22.636207395s" totalImagesPullingTime="508.270586ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:45:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:45:54.633972176 +0000 UTC m=+880.720309651" watchObservedRunningTime="2026-09-10 18:45:54.636207395 +0000 UTC m=+880.722544870" Sep 10 18:45:55.626422 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:55.626376 2581 generic.go:356] "Generic (PLEG): container finished" podID="894e00dc-93dd-4245-846f-3529d796f688" containerID="9f9e902b857119769d5fd9d77756ded4783cf6d220ee29df5dacac9cdba7030e" exitCode=0 Sep 10 18:45:55.626832 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:55.626446 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" event={"ID":"894e00dc-93dd-4245-846f-3529d796f688","Type":"ContainerDied","Data":"9f9e902b857119769d5fd9d77756ded4783cf6d220ee29df5dacac9cdba7030e"} Sep 10 18:45:55.626832 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:55.626502 2581 scope.go:117] "RemoveContainer" containerID="e2a268f88cf6d1a682cf1ee05ef94cd160f835d6651fa711d0cfbcde3807f107" Sep 10 18:45:55.626832 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:55.626763 2581 scope.go:117] "RemoveContainer" containerID="9f9e902b857119769d5fd9d77756ded4783cf6d220ee29df5dacac9cdba7030e" Sep 10 18:45:55.626966 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:45:55.626928 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-lgkff_llm(894e00dc-93dd-4245-846f-3529d796f688)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" podUID="894e00dc-93dd-4245-846f-3529d796f688" Sep 10 18:45:55.886354 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:55.886272 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:45:57.374734 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:45:57.374705 2581 scope.go:117] "RemoveContainer" containerID="6e9f567e2ecd7362e2ff7c57616058a0713f17e676b776e5ae50fbff17d3d4d1" Sep 10 18:45:57.375139 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:45:57.374896 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-jrt76_llm(e1f4af20-061e-4e66-b2c1-d14d0c501e01)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" podUID="e1f4af20-061e-4e66-b2c1-d14d0c501e01" Sep 10 18:46:03.178556 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:03.178510 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:46:03.179044 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:03.178581 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:46:03.179044 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:03.178600 2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:46:03.179044 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:03.178614 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:46:03.179044 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:03.178995 2581 scope.go:117] "RemoveContainer" containerID="9f9e902b857119769d5fd9d77756ded4783cf6d220ee29df5dacac9cdba7030e" Sep 10 18:46:03.179199 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:46:03.179180 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-lgkff_llm(894e00dc-93dd-4245-846f-3529d796f688)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" podUID="894e00dc-93dd-4245-846f-3529d796f688" Sep 10 18:46:04.377353 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:04.377297 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:46:12.374795 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:12.374771 2581 scope.go:117] "RemoveContainer" containerID="6e9f567e2ecd7362e2ff7c57616058a0713f17e676b776e5ae50fbff17d3d4d1" Sep 10 18:46:12.708503 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:12.708472 2581 generic.go:356] "Generic (PLEG): container finished" podID="e1f4af20-061e-4e66-b2c1-d14d0c501e01" containerID="08cf25d87635140e0fbff416321dd5fbadacba9be0e2926484192e8e06b7bdf3" exitCode=0 Sep 10 18:46:12.708623 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:12.708530 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" event={"ID":"e1f4af20-061e-4e66-b2c1-d14d0c501e01","Type":"ContainerDied","Data":"08cf25d87635140e0fbff416321dd5fbadacba9be0e2926484192e8e06b7bdf3"} Sep 10 18:46:12.708623 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:12.708575 2581 scope.go:117] "RemoveContainer" containerID="6e9f567e2ecd7362e2ff7c57616058a0713f17e676b776e5ae50fbff17d3d4d1" Sep 10 18:46:12.708911 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:12.708893 2581 scope.go:117] "RemoveContainer" containerID="08cf25d87635140e0fbff416321dd5fbadacba9be0e2926484192e8e06b7bdf3" Sep 10 18:46:12.709104 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:46:12.709081 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-jrt76_llm(e1f4af20-061e-4e66-b2c1-d14d0c501e01)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" podUID="e1f4af20-061e-4e66-b2c1-d14d0c501e01" Sep 10 18:46:15.374838 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:15.374810 2581 scope.go:117] "RemoveContainer" containerID="9f9e902b857119769d5fd9d77756ded4783cf6d220ee29df5dacac9cdba7030e" Sep 10 18:46:15.725032 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:15.724997 2581 generic.go:356] "Generic (PLEG): container finished" podID="894e00dc-93dd-4245-846f-3529d796f688" containerID="8f760f1bc3bbf57fb1914c7d301feb36fa195e3454c5d3715497774fac0a5d80" exitCode=0 Sep 10 18:46:15.725185 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:15.725067 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" event={"ID":"894e00dc-93dd-4245-846f-3529d796f688","Type":"ContainerDied","Data":"8f760f1bc3bbf57fb1914c7d301feb36fa195e3454c5d3715497774fac0a5d80"} Sep 10 18:46:15.725185 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:15.725128 2581 scope.go:117] "RemoveContainer" containerID="9f9e902b857119769d5fd9d77756ded4783cf6d220ee29df5dacac9cdba7030e" Sep 10 18:46:15.725566 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:15.725544 2581 scope.go:117] "RemoveContainer" containerID="8f760f1bc3bbf57fb1914c7d301feb36fa195e3454c5d3715497774fac0a5d80" Sep 10 18:46:15.725764 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:46:15.725747 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-lgkff_llm(894e00dc-93dd-4245-846f-3529d796f688)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" podUID="894e00dc-93dd-4245-846f-3529d796f688" Sep 10 18:46:15.851935 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:15.851907 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5"] Sep 10 18:46:15.856361 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:15.856344 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:46:15.858102 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:15.858078 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 10 18:46:15.865237 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:15.865215 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5"] Sep 10 18:46:15.962499 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:15.962410 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:46:15.989762 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:15.989733 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b404b14f-e5c2-4cdc-97bc-e0a9e5964d06-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5\" (UID: \"b404b14f-e5c2-4cdc-97bc-e0a9e5964d06\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:46:15.989909 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:15.989795 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b404b14f-e5c2-4cdc-97bc-e0a9e5964d06-home\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5\" (UID: \"b404b14f-e5c2-4cdc-97bc-e0a9e5964d06\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:46:15.989909 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:15.989896 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b404b14f-e5c2-4cdc-97bc-e0a9e5964d06-dshm\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5\" (UID: \"b404b14f-e5c2-4cdc-97bc-e0a9e5964d06\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:46:15.989993 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:15.989926 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b404b14f-e5c2-4cdc-97bc-e0a9e5964d06-model-cache\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5\" (UID: \"b404b14f-e5c2-4cdc-97bc-e0a9e5964d06\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:46:15.989993 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:15.989944 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhf9g\" (UniqueName: \"kubernetes.io/projected/b404b14f-e5c2-4cdc-97bc-e0a9e5964d06-kube-api-access-mhf9g\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5\" (UID: \"b404b14f-e5c2-4cdc-97bc-e0a9e5964d06\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:46:15.990060 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:15.989992 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b404b14f-e5c2-4cdc-97bc-e0a9e5964d06-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5\" (UID: \"b404b14f-e5c2-4cdc-97bc-e0a9e5964d06\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:46:16.090959 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:16.090926 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b404b14f-e5c2-4cdc-97bc-e0a9e5964d06-home\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5\" (UID: \"b404b14f-e5c2-4cdc-97bc-e0a9e5964d06\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:46:16.091121 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:16.090988 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b404b14f-e5c2-4cdc-97bc-e0a9e5964d06-dshm\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5\" (UID: \"b404b14f-e5c2-4cdc-97bc-e0a9e5964d06\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:46:16.091121 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:16.091005 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b404b14f-e5c2-4cdc-97bc-e0a9e5964d06-model-cache\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5\" (UID: \"b404b14f-e5c2-4cdc-97bc-e0a9e5964d06\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:46:16.091121 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:16.091021 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhf9g\" (UniqueName: \"kubernetes.io/projected/b404b14f-e5c2-4cdc-97bc-e0a9e5964d06-kube-api-access-mhf9g\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5\" (UID: \"b404b14f-e5c2-4cdc-97bc-e0a9e5964d06\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:46:16.091121 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:16.091047 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b404b14f-e5c2-4cdc-97bc-e0a9e5964d06-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5\" (UID: \"b404b14f-e5c2-4cdc-97bc-e0a9e5964d06\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:46:16.091121 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:16.091085 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b404b14f-e5c2-4cdc-97bc-e0a9e5964d06-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5\" (UID: \"b404b14f-e5c2-4cdc-97bc-e0a9e5964d06\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:46:16.091410 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:16.091299 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b404b14f-e5c2-4cdc-97bc-e0a9e5964d06-home\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5\" (UID: \"b404b14f-e5c2-4cdc-97bc-e0a9e5964d06\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:46:16.091410 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:16.091375 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b404b14f-e5c2-4cdc-97bc-e0a9e5964d06-model-cache\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5\" (UID: \"b404b14f-e5c2-4cdc-97bc-e0a9e5964d06\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:46:16.091514 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:16.091497 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b404b14f-e5c2-4cdc-97bc-e0a9e5964d06-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5\" (UID: \"b404b14f-e5c2-4cdc-97bc-e0a9e5964d06\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:46:16.093254 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:16.093234 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b404b14f-e5c2-4cdc-97bc-e0a9e5964d06-dshm\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5\" (UID: \"b404b14f-e5c2-4cdc-97bc-e0a9e5964d06\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:46:16.093583 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:16.093564 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b404b14f-e5c2-4cdc-97bc-e0a9e5964d06-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5\" (UID: \"b404b14f-e5c2-4cdc-97bc-e0a9e5964d06\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:46:16.097670 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:16.097645 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhf9g\" (UniqueName: \"kubernetes.io/projected/b404b14f-e5c2-4cdc-97bc-e0a9e5964d06-kube-api-access-mhf9g\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5\" (UID: \"b404b14f-e5c2-4cdc-97bc-e0a9e5964d06\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:46:16.166820 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:16.166792 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:46:16.379975 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:16.379945 2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:46:16.380370 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:16.379996 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:46:16.380370 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:16.380011 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:46:16.380370 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:16.380023 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:46:16.380370 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:16.380319 2581 scope.go:117] "RemoveContainer" containerID="08cf25d87635140e0fbff416321dd5fbadacba9be0e2926484192e8e06b7bdf3" Sep 10 18:46:16.380537 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:46:16.380519 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-jrt76_llm(e1f4af20-061e-4e66-b2c1-d14d0c501e01)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" podUID="e1f4af20-061e-4e66-b2c1-d14d0c501e01" Sep 10 18:46:16.437955 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:16.437905 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5"] Sep 10 18:46:16.440140 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:46:16.440115 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb404b14f_e5c2_4cdc_97bc_e0a9e5964d06.slice/crio-745484f23de688e0827718324c9a22739a7607ea27d53d12f81e0d736aaec412 WatchSource:0}: Error finding container 745484f23de688e0827718324c9a22739a7607ea27d53d12f81e0d736aaec412: Status 404 returned error can't find the container with id 745484f23de688e0827718324c9a22739a7607ea27d53d12f81e0d736aaec412 Sep 10 18:46:16.734709 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:16.734686 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" event={"ID":"b404b14f-e5c2-4cdc-97bc-e0a9e5964d06","Type":"ContainerStarted","Data":"ff2cb8d6660f5f52a07dd270dad8ec69c29499fb522c6d12152438bb9ddbd84e"} Sep 10 18:46:16.734818 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:16.734736 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" event={"ID":"b404b14f-e5c2-4cdc-97bc-e0a9e5964d06","Type":"ContainerStarted","Data":"745484f23de688e0827718324c9a22739a7607ea27d53d12f81e0d736aaec412"} Sep 10 18:46:16.734959 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:16.734947 2581 scope.go:117] "RemoveContainer" containerID="ff2cb8d6660f5f52a07dd270dad8ec69c29499fb522c6d12152438bb9ddbd84e" Sep 10 18:46:17.742492 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:17.742461 2581 generic.go:356] "Generic (PLEG): container finished" podID="b404b14f-e5c2-4cdc-97bc-e0a9e5964d06" containerID="ff2cb8d6660f5f52a07dd270dad8ec69c29499fb522c6d12152438bb9ddbd84e" exitCode=0 Sep 10 18:46:17.742492 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:17.742486 2581 generic.go:356] "Generic (PLEG): container finished" podID="b404b14f-e5c2-4cdc-97bc-e0a9e5964d06" containerID="36c8dc9528050cd4d94616f9e6af096be53102eb762fcfdc5487426492623fc1" exitCode=0 Sep 10 18:46:17.742993 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:17.742546 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" event={"ID":"b404b14f-e5c2-4cdc-97bc-e0a9e5964d06","Type":"ContainerDied","Data":"ff2cb8d6660f5f52a07dd270dad8ec69c29499fb522c6d12152438bb9ddbd84e"} Sep 10 18:46:17.742993 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:17.742607 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" event={"ID":"b404b14f-e5c2-4cdc-97bc-e0a9e5964d06","Type":"ContainerDied","Data":"36c8dc9528050cd4d94616f9e6af096be53102eb762fcfdc5487426492623fc1"} Sep 10 18:46:17.742993 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:17.742635 2581 scope.go:117] "RemoveContainer" containerID="ff2cb8d6660f5f52a07dd270dad8ec69c29499fb522c6d12152438bb9ddbd84e" Sep 10 18:46:17.742993 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:17.742942 2581 scope.go:117] "RemoveContainer" containerID="36c8dc9528050cd4d94616f9e6af096be53102eb762fcfdc5487426492623fc1" Sep 10 18:46:17.743182 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:46:17.743161 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5_llm(b404b14f-e5c2-4cdc-97bc-e0a9e5964d06)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" podUID="b404b14f-e5c2-4cdc-97bc-e0a9e5964d06" Sep 10 18:46:17.754636 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:17.754619 2581 scope.go:117] "RemoveContainer" containerID="ff2cb8d6660f5f52a07dd270dad8ec69c29499fb522c6d12152438bb9ddbd84e" Sep 10 18:46:17.754930 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:46:17.754911 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff2cb8d6660f5f52a07dd270dad8ec69c29499fb522c6d12152438bb9ddbd84e\": container with ID starting with ff2cb8d6660f5f52a07dd270dad8ec69c29499fb522c6d12152438bb9ddbd84e not found: ID does not exist" containerID="ff2cb8d6660f5f52a07dd270dad8ec69c29499fb522c6d12152438bb9ddbd84e" Sep 10 18:46:17.754994 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:17.754939 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff2cb8d6660f5f52a07dd270dad8ec69c29499fb522c6d12152438bb9ddbd84e"} err="failed to get container status \"ff2cb8d6660f5f52a07dd270dad8ec69c29499fb522c6d12152438bb9ddbd84e\": rpc error: code = NotFound desc = could not find container \"ff2cb8d6660f5f52a07dd270dad8ec69c29499fb522c6d12152438bb9ddbd84e\": container with ID starting with ff2cb8d6660f5f52a07dd270dad8ec69c29499fb522c6d12152438bb9ddbd84e not found: ID does not exist" Sep 10 18:46:18.749254 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:18.749222 2581 scope.go:117] "RemoveContainer" containerID="36c8dc9528050cd4d94616f9e6af096be53102eb762fcfdc5487426492623fc1" Sep 10 18:46:18.749631 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:46:18.749404 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5_llm(b404b14f-e5c2-4cdc-97bc-e0a9e5964d06)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" podUID="b404b14f-e5c2-4cdc-97bc-e0a9e5964d06" Sep 10 18:46:23.178249 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:23.178213 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:46:23.178677 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:23.178299 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:46:23.178677 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:23.178315 2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:46:23.178677 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:23.178354 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:46:23.178836 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:23.178697 2581 scope.go:117] "RemoveContainer" containerID="8f760f1bc3bbf57fb1914c7d301feb36fa195e3454c5d3715497774fac0a5d80" Sep 10 18:46:23.178901 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:46:23.178879 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-lgkff_llm(894e00dc-93dd-4245-846f-3529d796f688)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" podUID="894e00dc-93dd-4245-846f-3529d796f688" Sep 10 18:46:26.166903 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:26.166865 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:46:26.167229 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:26.166923 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:46:26.167229 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:26.166941 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:46:26.167229 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:26.167032 2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:46:26.167409 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:26.167312 2581 scope.go:117] "RemoveContainer" containerID="36c8dc9528050cd4d94616f9e6af096be53102eb762fcfdc5487426492623fc1" Sep 10 18:46:26.167515 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:46:26.167496 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5_llm(b404b14f-e5c2-4cdc-97bc-e0a9e5964d06)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" podUID="b404b14f-e5c2-4cdc-97bc-e0a9e5964d06" Sep 10 18:46:26.786818 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:26.786788 2581 scope.go:117] "RemoveContainer" containerID="36c8dc9528050cd4d94616f9e6af096be53102eb762fcfdc5487426492623fc1" Sep 10 18:46:26.786988 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:46:26.786941 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5_llm(b404b14f-e5c2-4cdc-97bc-e0a9e5964d06)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" podUID="b404b14f-e5c2-4cdc-97bc-e0a9e5964d06" Sep 10 18:46:30.374970 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:30.374938 2581 scope.go:117] "RemoveContainer" containerID="08cf25d87635140e0fbff416321dd5fbadacba9be0e2926484192e8e06b7bdf3" Sep 10 18:46:30.375376 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:46:30.375167 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-jrt76_llm(e1f4af20-061e-4e66-b2c1-d14d0c501e01)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" podUID="e1f4af20-061e-4e66-b2c1-d14d0c501e01" Sep 10 18:46:35.374980 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:35.374947 2581 scope.go:117] "RemoveContainer" containerID="8f760f1bc3bbf57fb1914c7d301feb36fa195e3454c5d3715497774fac0a5d80" Sep 10 18:46:35.375467 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:46:35.375127 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-lgkff_llm(894e00dc-93dd-4245-846f-3529d796f688)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" podUID="894e00dc-93dd-4245-846f-3529d796f688" Sep 10 18:46:40.995265 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:40.995228 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:46:41.374530 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:41.374505 2581 scope.go:117] "RemoveContainer" containerID="36c8dc9528050cd4d94616f9e6af096be53102eb762fcfdc5487426492623fc1" Sep 10 18:46:41.856015 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:41.855985 2581 generic.go:356] "Generic (PLEG): container finished" podID="b404b14f-e5c2-4cdc-97bc-e0a9e5964d06" containerID="8c15148ed0db9891a2d550a4279229892f43bbcabd1c3ad306a35f74125e958a" exitCode=0 Sep 10 18:46:41.856163 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:41.856063 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" event={"ID":"b404b14f-e5c2-4cdc-97bc-e0a9e5964d06","Type":"ContainerDied","Data":"8c15148ed0db9891a2d550a4279229892f43bbcabd1c3ad306a35f74125e958a"} Sep 10 18:46:41.856163 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:41.856144 2581 scope.go:117] "RemoveContainer" containerID="36c8dc9528050cd4d94616f9e6af096be53102eb762fcfdc5487426492623fc1" Sep 10 18:46:41.856487 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:41.856467 2581 scope.go:117] "RemoveContainer" containerID="8c15148ed0db9891a2d550a4279229892f43bbcabd1c3ad306a35f74125e958a" Sep 10 18:46:41.856674 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:46:41.856655 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5_llm(b404b14f-e5c2-4cdc-97bc-e0a9e5964d06)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" podUID="b404b14f-e5c2-4cdc-97bc-e0a9e5964d06" Sep 10 18:46:45.374893 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:45.374861 2581 scope.go:117] "RemoveContainer" containerID="08cf25d87635140e0fbff416321dd5fbadacba9be0e2926484192e8e06b7bdf3" Sep 10 18:46:45.375282 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:46:45.375022 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-jrt76_llm(e1f4af20-061e-4e66-b2c1-d14d0c501e01)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" podUID="e1f4af20-061e-4e66-b2c1-d14d0c501e01" Sep 10 18:46:46.167140 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:46.167114 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:46:46.167343 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:46.167198 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:46:46.167343 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:46.167223 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:46:46.167343 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:46.167244 2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:46:46.167637 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:46.167621 2581 scope.go:117] "RemoveContainer" containerID="8c15148ed0db9891a2d550a4279229892f43bbcabd1c3ad306a35f74125e958a" Sep 10 18:46:46.167825 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:46:46.167803 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5_llm(b404b14f-e5c2-4cdc-97bc-e0a9e5964d06)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" podUID="b404b14f-e5c2-4cdc-97bc-e0a9e5964d06" Sep 10 18:46:46.374879 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:46.374857 2581 scope.go:117] "RemoveContainer" containerID="8f760f1bc3bbf57fb1914c7d301feb36fa195e3454c5d3715497774fac0a5d80" Sep 10 18:46:46.375022 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:46:46.375004 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-lgkff_llm(894e00dc-93dd-4245-846f-3529d796f688)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" podUID="894e00dc-93dd-4245-846f-3529d796f688" Sep 10 18:46:46.883403 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:46.883377 2581 scope.go:117] "RemoveContainer" containerID="8c15148ed0db9891a2d550a4279229892f43bbcabd1c3ad306a35f74125e958a" Sep 10 18:46:46.883566 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:46:46.883544 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5_llm(b404b14f-e5c2-4cdc-97bc-e0a9e5964d06)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" podUID="b404b14f-e5c2-4cdc-97bc-e0a9e5964d06" Sep 10 18:46:59.374588 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:59.374555 2581 scope.go:117] "RemoveContainer" containerID="8f760f1bc3bbf57fb1914c7d301feb36fa195e3454c5d3715497774fac0a5d80" Sep 10 18:46:59.375108 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:59.374691 2581 scope.go:117] "RemoveContainer" containerID="8c15148ed0db9891a2d550a4279229892f43bbcabd1c3ad306a35f74125e958a" Sep 10 18:46:59.375108 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:46:59.374854 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5_llm(b404b14f-e5c2-4cdc-97bc-e0a9e5964d06)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" podUID="b404b14f-e5c2-4cdc-97bc-e0a9e5964d06" Sep 10 18:46:59.942441 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:59.942363 2581 generic.go:356] "Generic (PLEG): container finished" podID="894e00dc-93dd-4245-846f-3529d796f688" containerID="5d5d7731b0e71315ac76d0ff19f38232e785dfb5307d2bf8c5c4f5bc3ccf2b36" exitCode=0 Sep 10 18:46:59.942441 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:59.942419 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" event={"ID":"894e00dc-93dd-4245-846f-3529d796f688","Type":"ContainerDied","Data":"5d5d7731b0e71315ac76d0ff19f38232e785dfb5307d2bf8c5c4f5bc3ccf2b36"} Sep 10 18:46:59.942601 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:59.942491 2581 scope.go:117] "RemoveContainer" containerID="8f760f1bc3bbf57fb1914c7d301feb36fa195e3454c5d3715497774fac0a5d80" Sep 10 18:46:59.942775 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:46:59.942756 2581 scope.go:117] "RemoveContainer" containerID="5d5d7731b0e71315ac76d0ff19f38232e785dfb5307d2bf8c5c4f5bc3ccf2b36" Sep 10 18:46:59.942980 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:46:59.942959 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-lgkff_llm(894e00dc-93dd-4245-846f-3529d796f688)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" podUID="894e00dc-93dd-4245-846f-3529d796f688" Sep 10 18:47:00.375419 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:00.375378 2581 scope.go:117] "RemoveContainer" containerID="08cf25d87635140e0fbff416321dd5fbadacba9be0e2926484192e8e06b7bdf3" Sep 10 18:47:00.949975 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:00.949887 2581 generic.go:356] "Generic (PLEG): container finished" podID="e1f4af20-061e-4e66-b2c1-d14d0c501e01" containerID="71fc486a8ce5cabb5a34c27799a43dc10ce476a7aefcdeaf80dcc44a2f476ad3" exitCode=0 Sep 10 18:47:00.949975 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:00.949955 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" event={"ID":"e1f4af20-061e-4e66-b2c1-d14d0c501e01","Type":"ContainerDied","Data":"71fc486a8ce5cabb5a34c27799a43dc10ce476a7aefcdeaf80dcc44a2f476ad3"} Sep 10 18:47:00.950165 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:00.950016 2581 scope.go:117] "RemoveContainer" containerID="08cf25d87635140e0fbff416321dd5fbadacba9be0e2926484192e8e06b7bdf3" Sep 10 18:47:00.950373 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:00.950354 2581 scope.go:117] "RemoveContainer" containerID="71fc486a8ce5cabb5a34c27799a43dc10ce476a7aefcdeaf80dcc44a2f476ad3" Sep 10 18:47:00.950553 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:47:00.950535 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-jrt76_llm(e1f4af20-061e-4e66-b2c1-d14d0c501e01)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" podUID="e1f4af20-061e-4e66-b2c1-d14d0c501e01" Sep 10 18:47:03.178411 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:03.178375 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:47:03.178812 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:03.178474 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:47:03.178812 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:03.178527 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:47:03.178812 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:03.178545 2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:47:03.178985 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:03.178967 2581 scope.go:117] "RemoveContainer" containerID="5d5d7731b0e71315ac76d0ff19f38232e785dfb5307d2bf8c5c4f5bc3ccf2b36" Sep 10 18:47:03.179215 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:47:03.179194 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-lgkff_llm(894e00dc-93dd-4245-846f-3529d796f688)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" podUID="894e00dc-93dd-4245-846f-3529d796f688" Sep 10 18:47:03.741276 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:03.741243 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:47:03.800631 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:03.800600 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:47:03.965736 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:03.965710 2581 scope.go:117] "RemoveContainer" containerID="5d5d7731b0e71315ac76d0ff19f38232e785dfb5307d2bf8c5c4f5bc3ccf2b36" Sep 10 18:47:03.965951 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:47:03.965895 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-lgkff_llm(894e00dc-93dd-4245-846f-3529d796f688)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" podUID="894e00dc-93dd-4245-846f-3529d796f688" Sep 10 18:47:06.380227 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:06.380193 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:47:06.380696 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:06.380267 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:47:06.380696 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:06.380292 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:47:06.380696 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:06.380313 2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:47:06.380696 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:06.380694 2581 scope.go:117] "RemoveContainer" containerID="71fc486a8ce5cabb5a34c27799a43dc10ce476a7aefcdeaf80dcc44a2f476ad3" Sep 10 18:47:06.380913 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:47:06.380890 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-jrt76_llm(e1f4af20-061e-4e66-b2c1-d14d0c501e01)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" podUID="e1f4af20-061e-4e66-b2c1-d14d0c501e01" Sep 10 18:47:12.374853 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:12.374821 2581 scope.go:117] "RemoveContainer" containerID="8c15148ed0db9891a2d550a4279229892f43bbcabd1c3ad306a35f74125e958a" Sep 10 18:47:13.008816 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:13.008722 2581 generic.go:356] "Generic (PLEG): container finished" podID="b404b14f-e5c2-4cdc-97bc-e0a9e5964d06" containerID="c061cce360b35a71fc7fadca33dbf9120e1341ea3d6c0d37e6684e06525c3ce8" exitCode=0 Sep 10 18:47:13.008816 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:13.008795 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" event={"ID":"b404b14f-e5c2-4cdc-97bc-e0a9e5964d06","Type":"ContainerDied","Data":"c061cce360b35a71fc7fadca33dbf9120e1341ea3d6c0d37e6684e06525c3ce8"} Sep 10 18:47:13.009011 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:13.008857 2581 scope.go:117] "RemoveContainer" containerID="8c15148ed0db9891a2d550a4279229892f43bbcabd1c3ad306a35f74125e958a" Sep 10 18:47:13.009203 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:13.009182 2581 scope.go:117] "RemoveContainer" containerID="c061cce360b35a71fc7fadca33dbf9120e1341ea3d6c0d37e6684e06525c3ce8" Sep 10 18:47:13.009449 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:47:13.009427 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5_llm(b404b14f-e5c2-4cdc-97bc-e0a9e5964d06)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" podUID="b404b14f-e5c2-4cdc-97bc-e0a9e5964d06" Sep 10 18:47:16.167890 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:16.167844 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:47:16.168245 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:16.167925 2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:47:16.168245 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:16.167943 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:47:16.168245 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:16.167957 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:47:16.168373 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:16.168292 2581 scope.go:117] "RemoveContainer" containerID="c061cce360b35a71fc7fadca33dbf9120e1341ea3d6c0d37e6684e06525c3ce8" Sep 10 18:47:16.168522 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:47:16.168498 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5_llm(b404b14f-e5c2-4cdc-97bc-e0a9e5964d06)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" podUID="b404b14f-e5c2-4cdc-97bc-e0a9e5964d06" Sep 10 18:47:17.375080 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:17.375045 2581 scope.go:117] "RemoveContainer" containerID="71fc486a8ce5cabb5a34c27799a43dc10ce476a7aefcdeaf80dcc44a2f476ad3" Sep 10 18:47:17.375678 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:47:17.375255 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-jrt76_llm(e1f4af20-061e-4e66-b2c1-d14d0c501e01)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" podUID="e1f4af20-061e-4e66-b2c1-d14d0c501e01" Sep 10 18:47:18.375245 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:18.375217 2581 scope.go:117] "RemoveContainer" containerID="5d5d7731b0e71315ac76d0ff19f38232e785dfb5307d2bf8c5c4f5bc3ccf2b36" Sep 10 18:47:18.375656 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:47:18.375389 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-lgkff_llm(894e00dc-93dd-4245-846f-3529d796f688)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" podUID="894e00dc-93dd-4245-846f-3529d796f688" Sep 10 18:47:27.374658 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:27.374626 2581 scope.go:117] "RemoveContainer" containerID="c061cce360b35a71fc7fadca33dbf9120e1341ea3d6c0d37e6684e06525c3ce8" Sep 10 18:47:27.375053 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:47:27.374793 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5_llm(b404b14f-e5c2-4cdc-97bc-e0a9e5964d06)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" podUID="b404b14f-e5c2-4cdc-97bc-e0a9e5964d06" Sep 10 18:47:32.374945 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:32.374915 2581 scope.go:117] "RemoveContainer" containerID="71fc486a8ce5cabb5a34c27799a43dc10ce476a7aefcdeaf80dcc44a2f476ad3" Sep 10 18:47:32.375388 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:47:32.375100 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-jrt76_llm(e1f4af20-061e-4e66-b2c1-d14d0c501e01)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" podUID="e1f4af20-061e-4e66-b2c1-d14d0c501e01" Sep 10 18:47:33.374562 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:33.374533 2581 scope.go:117] "RemoveContainer" containerID="5d5d7731b0e71315ac76d0ff19f38232e785dfb5307d2bf8c5c4f5bc3ccf2b36" Sep 10 18:47:33.374739 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:47:33.374708 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-lgkff_llm(894e00dc-93dd-4245-846f-3529d796f688)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" podUID="894e00dc-93dd-4245-846f-3529d796f688" Sep 10 18:47:42.375029 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:42.374947 2581 scope.go:117] "RemoveContainer" containerID="c061cce360b35a71fc7fadca33dbf9120e1341ea3d6c0d37e6684e06525c3ce8" Sep 10 18:47:42.375522 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:47:42.375132 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5_llm(b404b14f-e5c2-4cdc-97bc-e0a9e5964d06)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" podUID="b404b14f-e5c2-4cdc-97bc-e0a9e5964d06" Sep 10 18:47:44.808366 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:44.808312 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:47:44.906417 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:44.906387 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:47:47.375187 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:47.375152 2581 scope.go:117] "RemoveContainer" containerID="71fc486a8ce5cabb5a34c27799a43dc10ce476a7aefcdeaf80dcc44a2f476ad3" Sep 10 18:47:47.375605 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:47:47.375305 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-jrt76_llm(e1f4af20-061e-4e66-b2c1-d14d0c501e01)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" podUID="e1f4af20-061e-4e66-b2c1-d14d0c501e01" Sep 10 18:47:48.374741 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:48.374708 2581 scope.go:117] "RemoveContainer" containerID="5d5d7731b0e71315ac76d0ff19f38232e785dfb5307d2bf8c5c4f5bc3ccf2b36" Sep 10 18:47:48.375042 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:47:48.374867 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-lgkff_llm(894e00dc-93dd-4245-846f-3529d796f688)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" podUID="894e00dc-93dd-4245-846f-3529d796f688" Sep 10 18:47:57.374890 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:57.374855 2581 scope.go:117] "RemoveContainer" containerID="c061cce360b35a71fc7fadca33dbf9120e1341ea3d6c0d37e6684e06525c3ce8" Sep 10 18:47:57.375283 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:57.375132 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 18:47:58.223684 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:58.223647 2581 generic.go:356] "Generic (PLEG): container finished" podID="b404b14f-e5c2-4cdc-97bc-e0a9e5964d06" containerID="8d62224ed171bd17740b888b6b0c36b5971330d64d8fd4b7c1f2b75e4a675e21" exitCode=0 Sep 10 18:47:58.223849 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:58.223712 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" event={"ID":"b404b14f-e5c2-4cdc-97bc-e0a9e5964d06","Type":"ContainerDied","Data":"8d62224ed171bd17740b888b6b0c36b5971330d64d8fd4b7c1f2b75e4a675e21"} Sep 10 18:47:58.223849 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:58.223766 2581 scope.go:117] "RemoveContainer" containerID="c061cce360b35a71fc7fadca33dbf9120e1341ea3d6c0d37e6684e06525c3ce8" Sep 10 18:47:58.224116 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:58.224098 2581 scope.go:117] "RemoveContainer" containerID="8d62224ed171bd17740b888b6b0c36b5971330d64d8fd4b7c1f2b75e4a675e21" Sep 10 18:47:58.224296 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:47:58.224278 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5_llm(b404b14f-e5c2-4cdc-97bc-e0a9e5964d06)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" podUID="b404b14f-e5c2-4cdc-97bc-e0a9e5964d06" Sep 10 18:47:59.374673 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:47:59.374642 2581 scope.go:117] "RemoveContainer" containerID="5d5d7731b0e71315ac76d0ff19f38232e785dfb5307d2bf8c5c4f5bc3ccf2b36" Sep 10 18:47:59.375129 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:47:59.374848 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-lgkff_llm(894e00dc-93dd-4245-846f-3529d796f688)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" podUID="894e00dc-93dd-4245-846f-3529d796f688" Sep 10 18:48:02.374804 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:48:02.374770 2581 scope.go:117] "RemoveContainer" containerID="71fc486a8ce5cabb5a34c27799a43dc10ce476a7aefcdeaf80dcc44a2f476ad3" Sep 10 18:48:02.375354 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:48:02.374955 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-jrt76_llm(e1f4af20-061e-4e66-b2c1-d14d0c501e01)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" podUID="e1f4af20-061e-4e66-b2c1-d14d0c501e01" Sep 10 18:48:06.167785 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:48:06.167740 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:48:06.168174 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:48:06.167842 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:48:06.168174 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:48:06.167866 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:48:06.168174 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:48:06.167886 2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:48:06.168281 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:48:06.168181 2581 scope.go:117] "RemoveContainer" containerID="8d62224ed171bd17740b888b6b0c36b5971330d64d8fd4b7c1f2b75e4a675e21" Sep 10 18:48:06.168372 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:48:06.168351 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5_llm(b404b14f-e5c2-4cdc-97bc-e0a9e5964d06)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" podUID="b404b14f-e5c2-4cdc-97bc-e0a9e5964d06" Sep 10 18:48:14.376742 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:48:14.376717 2581 scope.go:117] "RemoveContainer" containerID="5d5d7731b0e71315ac76d0ff19f38232e785dfb5307d2bf8c5c4f5bc3ccf2b36" Sep 10 18:48:14.377142 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:48:14.376866 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-lgkff_llm(894e00dc-93dd-4245-846f-3529d796f688)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" podUID="894e00dc-93dd-4245-846f-3529d796f688" Sep 10 18:48:16.374470 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:48:16.374442 2581 scope.go:117] "RemoveContainer" containerID="71fc486a8ce5cabb5a34c27799a43dc10ce476a7aefcdeaf80dcc44a2f476ad3" Sep 10 18:48:16.374855 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:48:16.374613 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-jrt76_llm(e1f4af20-061e-4e66-b2c1-d14d0c501e01)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" podUID="e1f4af20-061e-4e66-b2c1-d14d0c501e01" Sep 10 18:48:17.374975 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:48:17.374945 2581 scope.go:117] "RemoveContainer" containerID="8d62224ed171bd17740b888b6b0c36b5971330d64d8fd4b7c1f2b75e4a675e21" Sep 10 18:48:17.375367 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:48:17.375108 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5_llm(b404b14f-e5c2-4cdc-97bc-e0a9e5964d06)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" podUID="b404b14f-e5c2-4cdc-97bc-e0a9e5964d06" Sep 10 18:48:28.374835 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:48:28.374803 2581 scope.go:117] "RemoveContainer" containerID="5d5d7731b0e71315ac76d0ff19f38232e785dfb5307d2bf8c5c4f5bc3ccf2b36" Sep 10 18:48:28.375420 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:48:28.375221 2581 scope.go:117] "RemoveContainer" containerID="71fc486a8ce5cabb5a34c27799a43dc10ce476a7aefcdeaf80dcc44a2f476ad3" Sep 10 18:48:29.365856 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:48:29.365826 2581 generic.go:356] "Generic (PLEG): container finished" podID="894e00dc-93dd-4245-846f-3529d796f688" containerID="471c289ca0d545c991cba42c0ebed39ed23b88abaee9144fa6f5b2ff8a8c3f32" exitCode=0 Sep 10 18:48:29.366046 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:48:29.365905 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" event={"ID":"894e00dc-93dd-4245-846f-3529d796f688","Type":"ContainerDied","Data":"471c289ca0d545c991cba42c0ebed39ed23b88abaee9144fa6f5b2ff8a8c3f32"} Sep 10 18:48:29.366046 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:48:29.365964 2581 scope.go:117] "RemoveContainer" containerID="5d5d7731b0e71315ac76d0ff19f38232e785dfb5307d2bf8c5c4f5bc3ccf2b36" Sep 10 18:48:29.366285 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:48:29.366265 2581 scope.go:117] "RemoveContainer" containerID="471c289ca0d545c991cba42c0ebed39ed23b88abaee9144fa6f5b2ff8a8c3f32" Sep 10 18:48:29.366482 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:48:29.366464 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-lgkff_llm(894e00dc-93dd-4245-846f-3529d796f688)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" podUID="894e00dc-93dd-4245-846f-3529d796f688" Sep 10 18:48:29.367769 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:48:29.367750 2581 generic.go:356] "Generic (PLEG): container finished" podID="e1f4af20-061e-4e66-b2c1-d14d0c501e01" containerID="92ed2f3777b80b465a9cf0fe509df826dbe4d15bd3f4282f55d7e1bf6fe721b0" exitCode=0 Sep 10 18:48:29.367854 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:48:29.367824 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" event={"ID":"e1f4af20-061e-4e66-b2c1-d14d0c501e01","Type":"ContainerDied","Data":"92ed2f3777b80b465a9cf0fe509df826dbe4d15bd3f4282f55d7e1bf6fe721b0"} Sep 10 18:48:29.368122 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:48:29.368109 2581 scope.go:117] "RemoveContainer" containerID="92ed2f3777b80b465a9cf0fe509df826dbe4d15bd3f4282f55d7e1bf6fe721b0" Sep 10 18:48:29.368269 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:48:29.368245 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-jrt76_llm(e1f4af20-061e-4e66-b2c1-d14d0c501e01)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" podUID="e1f4af20-061e-4e66-b2c1-d14d0c501e01" Sep 10 18:48:29.377516 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:48:29.377284 2581 scope.go:117] "RemoveContainer" containerID="71fc486a8ce5cabb5a34c27799a43dc10ce476a7aefcdeaf80dcc44a2f476ad3" Sep 10 18:48:30.374515 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:48:30.374486 2581 scope.go:117] "RemoveContainer" containerID="8d62224ed171bd17740b888b6b0c36b5971330d64d8fd4b7c1f2b75e4a675e21" Sep 10 18:48:30.374712 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:48:30.374650 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5_llm(b404b14f-e5c2-4cdc-97bc-e0a9e5964d06)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" podUID="b404b14f-e5c2-4cdc-97bc-e0a9e5964d06" Sep 10 18:48:33.178680 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:48:33.178637 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:48:33.179069 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:48:33.178706 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:48:33.179069 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:48:33.178721 2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:48:33.179069 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:48:33.178733 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" Sep 10 18:48:33.179069 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:48:33.179000 2581 scope.go:117] "RemoveContainer" containerID="471c289ca0d545c991cba42c0ebed39ed23b88abaee9144fa6f5b2ff8a8c3f32" Sep 10 18:48:33.179202 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:48:33.179152 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-lgkff_llm(894e00dc-93dd-4245-846f-3529d796f688)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" podUID="894e00dc-93dd-4245-846f-3529d796f688" Sep 10 18:48:36.380053 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:48:36.380023 2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:48:36.380425 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:48:36.380067 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:48:36.380425 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:48:36.380081 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:48:36.380425 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:48:36.380095 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" Sep 10 18:48:36.380425 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:48:36.380361 2581 scope.go:117] "RemoveContainer" containerID="92ed2f3777b80b465a9cf0fe509df826dbe4d15bd3f4282f55d7e1bf6fe721b0" Sep 10 18:48:36.380550 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:48:36.380487 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-jrt76_llm(e1f4af20-061e-4e66-b2c1-d14d0c501e01)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" podUID="e1f4af20-061e-4e66-b2c1-d14d0c501e01" Sep 10 18:48:44.376515 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:48:44.376485 2581 scope.go:117] "RemoveContainer" containerID="471c289ca0d545c991cba42c0ebed39ed23b88abaee9144fa6f5b2ff8a8c3f32" Sep 10 18:48:44.376892 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:48:44.376743 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-lgkff_llm(894e00dc-93dd-4245-846f-3529d796f688)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" podUID="894e00dc-93dd-4245-846f-3529d796f688" Sep 10 18:48:45.374905 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:48:45.374868 2581 scope.go:117] "RemoveContainer" containerID="8d62224ed171bd17740b888b6b0c36b5971330d64d8fd4b7c1f2b75e4a675e21" Sep 10 18:48:45.375090 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:48:45.375071 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5_llm(b404b14f-e5c2-4cdc-97bc-e0a9e5964d06)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" podUID="b404b14f-e5c2-4cdc-97bc-e0a9e5964d06" Sep 10 18:48:49.375065 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:48:49.375025 2581 scope.go:117] "RemoveContainer" containerID="92ed2f3777b80b465a9cf0fe509df826dbe4d15bd3f4282f55d7e1bf6fe721b0" Sep 10 18:48:49.375562 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:48:49.375258 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-jrt76_llm(e1f4af20-061e-4e66-b2c1-d14d0c501e01)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" podUID="e1f4af20-061e-4e66-b2c1-d14d0c501e01" Sep 10 18:48:58.374878 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:48:58.374838 2581 scope.go:117] "RemoveContainer" containerID="8d62224ed171bd17740b888b6b0c36b5971330d64d8fd4b7c1f2b75e4a675e21" Sep 10 18:48:58.375395 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:48:58.375027 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5_llm(b404b14f-e5c2-4cdc-97bc-e0a9e5964d06)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" podUID="b404b14f-e5c2-4cdc-97bc-e0a9e5964d06" Sep 10 18:48:59.374816 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:48:59.374787 2581 scope.go:117] "RemoveContainer" containerID="471c289ca0d545c991cba42c0ebed39ed23b88abaee9144fa6f5b2ff8a8c3f32" Sep 10 18:48:59.374990 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:48:59.374964 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-lgkff_llm(894e00dc-93dd-4245-846f-3529d796f688)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" podUID="894e00dc-93dd-4245-846f-3529d796f688" Sep 10 18:49:03.375268 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:49:03.375233 2581 scope.go:117] "RemoveContainer" containerID="92ed2f3777b80b465a9cf0fe509df826dbe4d15bd3f4282f55d7e1bf6fe721b0" Sep 10 18:49:03.375818 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:49:03.375505 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-jrt76_llm(e1f4af20-061e-4e66-b2c1-d14d0c501e01)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" podUID="e1f4af20-061e-4e66-b2c1-d14d0c501e01" Sep 10 18:49:10.375182 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:49:10.375101 2581 scope.go:117] "RemoveContainer" containerID="471c289ca0d545c991cba42c0ebed39ed23b88abaee9144fa6f5b2ff8a8c3f32" Sep 10 18:49:10.375585 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:49:10.375282 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-lgkff_llm(894e00dc-93dd-4245-846f-3529d796f688)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" podUID="894e00dc-93dd-4245-846f-3529d796f688" Sep 10 18:49:12.375003 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:49:12.374970 2581 scope.go:117] "RemoveContainer" containerID="8d62224ed171bd17740b888b6b0c36b5971330d64d8fd4b7c1f2b75e4a675e21" Sep 10 18:49:12.375428 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:49:12.375171 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5_llm(b404b14f-e5c2-4cdc-97bc-e0a9e5964d06)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" podUID="b404b14f-e5c2-4cdc-97bc-e0a9e5964d06" Sep 10 18:49:16.374626 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:49:16.374591 2581 scope.go:117] "RemoveContainer" containerID="92ed2f3777b80b465a9cf0fe509df826dbe4d15bd3f4282f55d7e1bf6fe721b0" Sep 10 18:49:16.374991 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:49:16.374818 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-jrt76_llm(e1f4af20-061e-4e66-b2c1-d14d0c501e01)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" podUID="e1f4af20-061e-4e66-b2c1-d14d0c501e01" Sep 10 18:49:23.375387 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:49:23.375352 2581 scope.go:117] "RemoveContainer" containerID="471c289ca0d545c991cba42c0ebed39ed23b88abaee9144fa6f5b2ff8a8c3f32" Sep 10 18:49:23.375765 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:49:23.375539 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-lgkff_llm(894e00dc-93dd-4245-846f-3529d796f688)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" podUID="894e00dc-93dd-4245-846f-3529d796f688" Sep 10 18:49:27.375434 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:49:27.375400 2581 scope.go:117] "RemoveContainer" containerID="8d62224ed171bd17740b888b6b0c36b5971330d64d8fd4b7c1f2b75e4a675e21" Sep 10 18:49:27.632811 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:49:27.632737 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" event={"ID":"b404b14f-e5c2-4cdc-97bc-e0a9e5964d06","Type":"ContainerStarted","Data":"a1dd741e4736e468d5819bb5cce27a1cd6441f8b2e9bb3b4aeaea7d70f26ae88"} Sep 10 18:49:27.648891 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:49:27.648847 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" podStartSLOduration=191.456689523 podStartE2EDuration="3m12.648830315s" totalImagesPullingTime="1.192140792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:46:15 +0000 UTC" imagePullSessionsCount=6 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:49:27.647299342 +0000 UTC m=+1093.733636817" watchObservedRunningTime="2026-09-10 18:49:27.648830315 +0000 UTC m=+1093.735167791" Sep 10 18:49:28.639395 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:49:28.639362 2581 generic.go:356] "Generic (PLEG): container finished" podID="b404b14f-e5c2-4cdc-97bc-e0a9e5964d06" containerID="a1dd741e4736e468d5819bb5cce27a1cd6441f8b2e9bb3b4aeaea7d70f26ae88" exitCode=0 Sep 10 18:49:28.639802 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:49:28.639436 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" event={"ID":"b404b14f-e5c2-4cdc-97bc-e0a9e5964d06","Type":"ContainerDied","Data":"a1dd741e4736e468d5819bb5cce27a1cd6441f8b2e9bb3b4aeaea7d70f26ae88"} Sep 10 18:49:28.639802 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:49:28.639495 2581 scope.go:117] "RemoveContainer" containerID="8d62224ed171bd17740b888b6b0c36b5971330d64d8fd4b7c1f2b75e4a675e21" Sep 10 18:49:28.639919 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:49:28.639875 2581 scope.go:117] "RemoveContainer" containerID="a1dd741e4736e468d5819bb5cce27a1cd6441f8b2e9bb3b4aeaea7d70f26ae88" Sep 10 18:49:28.640100 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:49:28.640077 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5_llm(b404b14f-e5c2-4cdc-97bc-e0a9e5964d06)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" podUID="b404b14f-e5c2-4cdc-97bc-e0a9e5964d06" Sep 10 18:49:30.375130 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:49:30.375094 2581 scope.go:117] "RemoveContainer" containerID="92ed2f3777b80b465a9cf0fe509df826dbe4d15bd3f4282f55d7e1bf6fe721b0" Sep 10 18:49:30.375619 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:49:30.375305 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-jrt76_llm(e1f4af20-061e-4e66-b2c1-d14d0c501e01)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" podUID="e1f4af20-061e-4e66-b2c1-d14d0c501e01" Sep 10 18:49:36.167096 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:49:36.167060 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:49:36.167642 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:49:36.167156 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:49:36.167642 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:49:36.167178 2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:49:36.167642 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:49:36.167195 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" Sep 10 18:49:36.167642 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:49:36.167566 2581 scope.go:117] "RemoveContainer" containerID="a1dd741e4736e468d5819bb5cce27a1cd6441f8b2e9bb3b4aeaea7d70f26ae88" Sep 10 18:49:36.167831 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:49:36.167753 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5_llm(b404b14f-e5c2-4cdc-97bc-e0a9e5964d06)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" podUID="b404b14f-e5c2-4cdc-97bc-e0a9e5964d06" Sep 10 18:49:38.375375 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:49:38.375348 2581 scope.go:117] "RemoveContainer" containerID="471c289ca0d545c991cba42c0ebed39ed23b88abaee9144fa6f5b2ff8a8c3f32" Sep 10 18:49:38.375832 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:49:38.375508 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-lgkff_llm(894e00dc-93dd-4245-846f-3529d796f688)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" podUID="894e00dc-93dd-4245-846f-3529d796f688" Sep 10 18:49:42.374807 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:49:42.374775 2581 scope.go:117] "RemoveContainer" containerID="92ed2f3777b80b465a9cf0fe509df826dbe4d15bd3f4282f55d7e1bf6fe721b0" Sep 10 18:49:42.375209 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:49:42.374936 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-jrt76_llm(e1f4af20-061e-4e66-b2c1-d14d0c501e01)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" podUID="e1f4af20-061e-4e66-b2c1-d14d0c501e01" Sep 10 18:49:47.375282 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:49:47.375247 2581 scope.go:117] "RemoveContainer" containerID="a1dd741e4736e468d5819bb5cce27a1cd6441f8b2e9bb3b4aeaea7d70f26ae88" Sep 10 18:49:47.375708 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:49:47.375446 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5_llm(b404b14f-e5c2-4cdc-97bc-e0a9e5964d06)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" podUID="b404b14f-e5c2-4cdc-97bc-e0a9e5964d06" Sep 10 18:49:52.374642 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:49:52.374612 2581 scope.go:117] "RemoveContainer" containerID="471c289ca0d545c991cba42c0ebed39ed23b88abaee9144fa6f5b2ff8a8c3f32" Sep 10 18:49:52.375045 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:49:52.374784 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-lgkff_llm(894e00dc-93dd-4245-846f-3529d796f688)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" podUID="894e00dc-93dd-4245-846f-3529d796f688" Sep 10 18:49:53.374389 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:49:53.374352 2581 scope.go:117] "RemoveContainer" containerID="92ed2f3777b80b465a9cf0fe509df826dbe4d15bd3f4282f55d7e1bf6fe721b0" Sep 10 18:49:53.374570 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:49:53.374535 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-jrt76_llm(e1f4af20-061e-4e66-b2c1-d14d0c501e01)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" podUID="e1f4af20-061e-4e66-b2c1-d14d0c501e01" Sep 10 18:49:58.375262 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:49:58.375232 2581 scope.go:117] "RemoveContainer" containerID="a1dd741e4736e468d5819bb5cce27a1cd6441f8b2e9bb3b4aeaea7d70f26ae88" Sep 10 18:49:58.375758 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:49:58.375451 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5_llm(b404b14f-e5c2-4cdc-97bc-e0a9e5964d06)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" podUID="b404b14f-e5c2-4cdc-97bc-e0a9e5964d06" Sep 10 18:50:04.377294 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:04.377264 2581 scope.go:117] "RemoveContainer" containerID="471c289ca0d545c991cba42c0ebed39ed23b88abaee9144fa6f5b2ff8a8c3f32" Sep 10 18:50:04.379412 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:04.377369 2581 scope.go:117] "RemoveContainer" containerID="92ed2f3777b80b465a9cf0fe509df826dbe4d15bd3f4282f55d7e1bf6fe721b0" Sep 10 18:50:04.379412 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:50:04.377521 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-jrt76_llm(e1f4af20-061e-4e66-b2c1-d14d0c501e01)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" podUID="e1f4af20-061e-4e66-b2c1-d14d0c501e01" Sep 10 18:50:04.379412 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:50:04.377521 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-lgkff_llm(894e00dc-93dd-4245-846f-3529d796f688)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" podUID="894e00dc-93dd-4245-846f-3529d796f688" Sep 10 18:50:07.611373 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:07.611346 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5_b404b14f-e5c2-4cdc-97bc-e0a9e5964d06/main/5.log" Sep 10 18:50:07.728754 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:07.728724 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5_b404b14f-e5c2-4cdc-97bc-e0a9e5964d06/main/5.log" Sep 10 18:50:08.830480 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:08.830454 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-f4bb7cdb7-4tqbj_adefb8c0-d030-424a-ac8c-a5723f66d6e3/authorino/0.log" Sep 10 18:50:10.375566 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:10.375531 2581 scope.go:117] "RemoveContainer" containerID="a1dd741e4736e468d5819bb5cce27a1cd6441f8b2e9bb3b4aeaea7d70f26ae88" Sep 10 18:50:10.375956 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:50:10.375694 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5_llm(b404b14f-e5c2-4cdc-97bc-e0a9e5964d06)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" podUID="b404b14f-e5c2-4cdc-97bc-e0a9e5964d06" Sep 10 18:50:14.541631 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:14.541605 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-mbc8q_d2a51711-565b-4fac-8dfd-c1932700f12a/manager/0.log" Sep 10 18:50:14.673823 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:14.673789 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-pp5xl_5b716014-647a-4159-8511-0349dda108e1/manager/0.log" Sep 10 18:50:14.958120 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:14.958038 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-958jt_5bc21d47-8dd2-4012-815a-588353e3cd58/server/0.log" Sep 10 18:50:15.090521 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:15.090492 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-9rgkh_7eba2f85-27fd-4fe5-8e01-0facd1e8bc78/manager/0.log" Sep 10 18:50:15.390568 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:15.390540 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4_9b8839e0-8ba6-4f99-94b3-c4202c9e1404/copy-manifests/0.log" Sep 10 18:50:15.403592 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:15.403568 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4_9b8839e0-8ba6-4f99-94b3-c4202c9e1404/manager/0.log" Sep 10 18:50:15.929960 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:15.929931 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-79bbc6b8-xl5l8_c6987549-3bf4-4234-bb10-8b92064a64d7/maas-api/0.log" Sep 10 18:50:16.176180 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:16.176150 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-7cvlb_9de637bc-a885-4dcb-ba47-b6c8be00b70c/postgres/0.log" Sep 10 18:50:16.374621 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:16.374593 2581 scope.go:117] "RemoveContainer" containerID="471c289ca0d545c991cba42c0ebed39ed23b88abaee9144fa6f5b2ff8a8c3f32" Sep 10 18:50:16.374799 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:50:16.374761 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-lgkff_llm(894e00dc-93dd-4245-846f-3529d796f688)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" podUID="894e00dc-93dd-4245-846f-3529d796f688" Sep 10 18:50:16.991269 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:16.991240 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-f4bb7cdb7-4tqbj_adefb8c0-d030-424a-ac8c-a5723f66d6e3/authorino/0.log" Sep 10 18:50:17.108306 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:17.108248 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-6hl2d_df9e3b01-9857-4103-96ce-39a3da42e7d0/manager/0.log" Sep 10 18:50:17.237950 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:17.237924 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-9bf85_674fa0a2-179a-47c9-b8d8-c7e1e947e93b/manager/0.log" Sep 10 18:50:17.342764 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:17.342744 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-8sp4k_57122aa5-9906-4e29-a8b6-0424da9fceb7/kuadrant-console-plugin/0.log" Sep 10 18:50:17.375217 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:17.375199 2581 scope.go:117] "RemoveContainer" containerID="92ed2f3777b80b465a9cf0fe509df826dbe4d15bd3f4282f55d7e1bf6fe721b0" Sep 10 18:50:17.375405 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:50:17.375386 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-jrt76_llm(e1f4af20-061e-4e66-b2c1-d14d0c501e01)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" podUID="e1f4af20-061e-4e66-b2c1-d14d0c501e01" Sep 10 18:50:17.678020 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:17.677948 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-ptlsk_3d15dbde-9c7a-4f40-871f-33179a38f87b/manager/0.log" Sep 10 18:50:18.263633 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:18.263591 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64dfbcccbf-9pgvj_828ca3e2-4140-4cb3-be58-7aa5395d90ac/kube-auth-proxy/0.log" Sep 10 18:50:18.486204 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:18.486183 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-h6mg2_952dcc01-8c30-41c0-9d9f-2964ed5c0760/istio-proxy/1.log" Sep 10 18:50:19.206317 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:19.206291 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-859856b874-jrt76_e1f4af20-061e-4e66-b2c1-d14d0c501e01/main/5.log" Sep 10 18:50:19.323672 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:19.323640 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-6685d86975-lgkff_894e00dc-93dd-4245-846f-3529d796f688/main/5.log" Sep 10 18:50:19.815882 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:19.815851 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5_b404b14f-e5c2-4cdc-97bc-e0a9e5964d06/main/5.log" Sep 10 18:50:23.374686 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:23.374658 2581 scope.go:117] "RemoveContainer" containerID="a1dd741e4736e468d5819bb5cce27a1cd6441f8b2e9bb3b4aeaea7d70f26ae88" Sep 10 18:50:23.375066 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:50:23.374843 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5_llm(b404b14f-e5c2-4cdc-97bc-e0a9e5964d06)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" podUID="b404b14f-e5c2-4cdc-97bc-e0a9e5964d06" Sep 10 18:50:26.150044 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:26.150007 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:50:26.215892 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:26.215859 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:50:28.375392 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:28.375359 2581 scope.go:117] "RemoveContainer" containerID="471c289ca0d545c991cba42c0ebed39ed23b88abaee9144fa6f5b2ff8a8c3f32" Sep 10 18:50:28.375837 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:50:28.375583 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-lgkff_llm(894e00dc-93dd-4245-846f-3529d796f688)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" podUID="894e00dc-93dd-4245-846f-3529d796f688" Sep 10 18:50:29.125677 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:29.125647 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-df44c_0b6bd408-6ab0-48c2-bbb3-04e7e1146ff9/global-pull-secret-syncer/0.log" Sep 10 18:50:29.205086 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:29.205061 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" Sep 10 18:50:29.290050 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:29.290025 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-166.ec2.internal_a6bd606f49b0efaf64eed63f0b2241b9/haproxy/0.log" Sep 10 18:50:31.374695 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:31.374668 2581 scope.go:117] "RemoveContainer" containerID="92ed2f3777b80b465a9cf0fe509df826dbe4d15bd3f4282f55d7e1bf6fe721b0" Sep 10 18:50:31.375066 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:50:31.374823 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-jrt76_llm(e1f4af20-061e-4e66-b2c1-d14d0c501e01)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" podUID="e1f4af20-061e-4e66-b2c1-d14d0c501e01" Sep 10 18:50:32.849499 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:32.849467 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g45x_bd3aed00-8ff3-4247-abc7-4893cfae63dc/konnectivity-agent/0.log" limit=1024 Sep 10 18:50:34.400759 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:34.400729 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-f4bb7cdb7-4tqbj_adefb8c0-d030-424a-ac8c-a5723f66d6e3/authorino/0.log" Sep 10 18:50:34.422932 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:34.422907 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-6hl2d_df9e3b01-9857-4103-96ce-39a3da42e7d0/manager/0.log" Sep 10 18:50:34.452427 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:34.452405 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-9bf85_674fa0a2-179a-47c9-b8d8-c7e1e947e93b/manager/0.log" Sep 10 18:50:34.485205 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:34.485179 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-8sp4k_57122aa5-9906-4e29-a8b6-0424da9fceb7/kuadrant-console-plugin/0.log" Sep 10 18:50:34.584433 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:34.584404 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-ptlsk_3d15dbde-9c7a-4f40-871f-33179a38f87b/manager/0.log" Sep 10 18:50:35.375150 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:35.375118 2581 scope.go:117] "RemoveContainer" containerID="a1dd741e4736e468d5819bb5cce27a1cd6441f8b2e9bb3b4aeaea7d70f26ae88" Sep 10 18:50:35.375353 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:50:35.375285 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5_llm(b404b14f-e5c2-4cdc-97bc-e0a9e5964d06)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" podUID="b404b14f-e5c2-4cdc-97bc-e0a9e5964d06" Sep 10 18:50:36.579309 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:36.579283 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xzs2v_591fba1e-a40f-41a3-8955-fd32464bf2ad/node-exporter/0.log" Sep 10 18:50:36.598557 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:36.598534 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xzs2v_591fba1e-a40f-41a3-8955-fd32464bf2ad/kube-rbac-proxy/0.log" Sep 10 18:50:36.617015 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:36.617001 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xzs2v_591fba1e-a40f-41a3-8955-fd32464bf2ad/init-textfile/0.log" Sep 10 18:50:39.134298 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:39.134261 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ddcqf/perf-node-gather-daemonset-hc6sl"] Sep 10 18:50:39.139629 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:39.139607 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ddcqf/perf-node-gather-daemonset-hc6sl" Sep 10 18:50:39.141590 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:39.141565 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ddcqf\"/\"default-dockercfg-bf9tb\"" Sep 10 18:50:39.141772 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:39.141690 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ddcqf\"/\"kube-root-ca.crt\"" Sep 10 18:50:39.142132 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:39.142111 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ddcqf\"/\"openshift-service-ca.crt\"" Sep 10 18:50:39.143019 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:39.143000 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ddcqf/perf-node-gather-daemonset-hc6sl"] Sep 10 18:50:39.195671 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:39.195648 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/be4566b5-a32a-492d-aad9-82759fbdfe8a-proc\") pod \"perf-node-gather-daemonset-hc6sl\" (UID: \"be4566b5-a32a-492d-aad9-82759fbdfe8a\") " pod="openshift-must-gather-ddcqf/perf-node-gather-daemonset-hc6sl" Sep 10 18:50:39.195785 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:39.195676 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be4566b5-a32a-492d-aad9-82759fbdfe8a-lib-modules\") pod \"perf-node-gather-daemonset-hc6sl\" (UID: \"be4566b5-a32a-492d-aad9-82759fbdfe8a\") " pod="openshift-must-gather-ddcqf/perf-node-gather-daemonset-hc6sl" Sep 10 18:50:39.195785 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:39.195713 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be4566b5-a32a-492d-aad9-82759fbdfe8a-sys\") pod \"perf-node-gather-daemonset-hc6sl\" (UID: \"be4566b5-a32a-492d-aad9-82759fbdfe8a\") " pod="openshift-must-gather-ddcqf/perf-node-gather-daemonset-hc6sl" Sep 10 18:50:39.195785 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:39.195730 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxbc8\" (UniqueName: \"kubernetes.io/projected/be4566b5-a32a-492d-aad9-82759fbdfe8a-kube-api-access-bxbc8\") pod \"perf-node-gather-daemonset-hc6sl\" (UID: \"be4566b5-a32a-492d-aad9-82759fbdfe8a\") " pod="openshift-must-gather-ddcqf/perf-node-gather-daemonset-hc6sl" Sep 10 18:50:39.195895 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:39.195786 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/be4566b5-a32a-492d-aad9-82759fbdfe8a-podres\") pod \"perf-node-gather-daemonset-hc6sl\" (UID: \"be4566b5-a32a-492d-aad9-82759fbdfe8a\") " pod="openshift-must-gather-ddcqf/perf-node-gather-daemonset-hc6sl" Sep 10 18:50:39.297091 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:39.297067 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/be4566b5-a32a-492d-aad9-82759fbdfe8a-proc\") pod \"perf-node-gather-daemonset-hc6sl\" (UID: \"be4566b5-a32a-492d-aad9-82759fbdfe8a\") " pod="openshift-must-gather-ddcqf/perf-node-gather-daemonset-hc6sl" Sep 10 18:50:39.297091 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:39.297093 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be4566b5-a32a-492d-aad9-82759fbdfe8a-lib-modules\") pod \"perf-node-gather-daemonset-hc6sl\" (UID: \"be4566b5-a32a-492d-aad9-82759fbdfe8a\") " pod="openshift-must-gather-ddcqf/perf-node-gather-daemonset-hc6sl" Sep 10 18:50:39.297235 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:39.297122 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be4566b5-a32a-492d-aad9-82759fbdfe8a-sys\") pod \"perf-node-gather-daemonset-hc6sl\" (UID: \"be4566b5-a32a-492d-aad9-82759fbdfe8a\") " pod="openshift-must-gather-ddcqf/perf-node-gather-daemonset-hc6sl" Sep 10 18:50:39.297235 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:39.297140 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxbc8\" (UniqueName: \"kubernetes.io/projected/be4566b5-a32a-492d-aad9-82759fbdfe8a-kube-api-access-bxbc8\") pod \"perf-node-gather-daemonset-hc6sl\" (UID: \"be4566b5-a32a-492d-aad9-82759fbdfe8a\") " pod="openshift-must-gather-ddcqf/perf-node-gather-daemonset-hc6sl" Sep 10 18:50:39.297235 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:39.297159 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/be4566b5-a32a-492d-aad9-82759fbdfe8a-podres\") pod \"perf-node-gather-daemonset-hc6sl\" (UID: \"be4566b5-a32a-492d-aad9-82759fbdfe8a\") " pod="openshift-must-gather-ddcqf/perf-node-gather-daemonset-hc6sl" Sep 10 18:50:39.297235 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:39.297195 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/be4566b5-a32a-492d-aad9-82759fbdfe8a-proc\") pod \"perf-node-gather-daemonset-hc6sl\" (UID: \"be4566b5-a32a-492d-aad9-82759fbdfe8a\") " pod="openshift-must-gather-ddcqf/perf-node-gather-daemonset-hc6sl" Sep 10 18:50:39.297235 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:39.297211 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be4566b5-a32a-492d-aad9-82759fbdfe8a-sys\") pod \"perf-node-gather-daemonset-hc6sl\" (UID: \"be4566b5-a32a-492d-aad9-82759fbdfe8a\") " pod="openshift-must-gather-ddcqf/perf-node-gather-daemonset-hc6sl" Sep 10 18:50:39.297428 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:39.297259 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/be4566b5-a32a-492d-aad9-82759fbdfe8a-podres\") pod \"perf-node-gather-daemonset-hc6sl\" (UID: \"be4566b5-a32a-492d-aad9-82759fbdfe8a\") " pod="openshift-must-gather-ddcqf/perf-node-gather-daemonset-hc6sl" Sep 10 18:50:39.297428 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:39.297260 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be4566b5-a32a-492d-aad9-82759fbdfe8a-lib-modules\") pod \"perf-node-gather-daemonset-hc6sl\" (UID: \"be4566b5-a32a-492d-aad9-82759fbdfe8a\") " pod="openshift-must-gather-ddcqf/perf-node-gather-daemonset-hc6sl" Sep 10 18:50:39.303889 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:39.303864 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxbc8\" (UniqueName: \"kubernetes.io/projected/be4566b5-a32a-492d-aad9-82759fbdfe8a-kube-api-access-bxbc8\") pod \"perf-node-gather-daemonset-hc6sl\" (UID: \"be4566b5-a32a-492d-aad9-82759fbdfe8a\") " pod="openshift-must-gather-ddcqf/perf-node-gather-daemonset-hc6sl" Sep 10 18:50:39.451668 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:39.451609 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ddcqf/perf-node-gather-daemonset-hc6sl" Sep 10 18:50:39.720531 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:39.720507 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ddcqf/perf-node-gather-daemonset-hc6sl"] Sep 10 18:50:39.722915 ip-10-0-134-166 kubenswrapper[2581]: W0910 18:50:39.722888 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbe4566b5_a32a_492d_aad9_82759fbdfe8a.slice/crio-0b41311dc45279db42b62f5e31d7cbeef7c46530f7190173f1f1cf03c0342be2 WatchSource:0}: Error finding container 0b41311dc45279db42b62f5e31d7cbeef7c46530f7190173f1f1cf03c0342be2: Status 404 returned error can't find the container with id 0b41311dc45279db42b62f5e31d7cbeef7c46530f7190173f1f1cf03c0342be2 Sep 10 18:50:39.972402 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:39.972290 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ddcqf/perf-node-gather-daemonset-hc6sl" event={"ID":"be4566b5-a32a-492d-aad9-82759fbdfe8a","Type":"ContainerStarted","Data":"77debdb1a7355daf68c3107c54d6ac715e15895c73f2227dae4f0f2905b5c0ee"} Sep 10 18:50:39.972402 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:39.972363 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ddcqf/perf-node-gather-daemonset-hc6sl" event={"ID":"be4566b5-a32a-492d-aad9-82759fbdfe8a","Type":"ContainerStarted","Data":"0b41311dc45279db42b62f5e31d7cbeef7c46530f7190173f1f1cf03c0342be2"} Sep 10 18:50:39.972610 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:39.972442 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ddcqf/perf-node-gather-daemonset-hc6sl" Sep 10 18:50:39.986191 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:39.986135 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ddcqf/perf-node-gather-daemonset-hc6sl" podStartSLOduration=0.9861173 podStartE2EDuration="986.1173ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 18:50:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:50:39.9846073 +0000 UTC m=+1166.070944775" watchObservedRunningTime="2026-09-10 18:50:39.9861173 +0000 UTC m=+1166.072454776" Sep 10 18:50:40.374756 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:40.374723 2581 scope.go:117] "RemoveContainer" containerID="471c289ca0d545c991cba42c0ebed39ed23b88abaee9144fa6f5b2ff8a8c3f32" Sep 10 18:50:40.375146 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:50:40.374914 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-lgkff_llm(894e00dc-93dd-4245-846f-3529d796f688)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" podUID="894e00dc-93dd-4245-846f-3529d796f688" Sep 10 18:50:41.344158 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:41.344126 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4t7vn_2c41b6ab-b71c-4b51-9cf7-4deba7a3a113/dns/0.log" Sep 10 18:50:41.362696 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:41.362668 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4t7vn_2c41b6ab-b71c-4b51-9cf7-4deba7a3a113/kube-rbac-proxy/0.log" Sep 10 18:50:41.512018 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:41.511995 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gf28x_4d70e39b-2b94-4593-828b-75d5d071560e/dns-node-resolver/0.log" Sep 10 18:50:42.085178 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:42.085153 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5678c4d879-pgdx7_6b082e08-bb40-4b42-aff0-ffc0ccbf2f02/registry/0.log" Sep 10 18:50:42.088515 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:42.088497 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5678c4d879-pgdx7_6b082e08-bb40-4b42-aff0-ffc0ccbf2f02/registry/1.log" Sep 10 18:50:42.146977 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:42.146956 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lh8gr_dc8250af-b1fc-4108-b6fd-092198a098ac/node-ca/0.log" Sep 10 18:50:43.119204 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:43.119178 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64dfbcccbf-9pgvj_828ca3e2-4140-4cb3-be58-7aa5395d90ac/kube-auth-proxy/0.log" Sep 10 18:50:43.164949 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:43.164927 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-h6mg2_952dcc01-8c30-41c0-9d9f-2964ed5c0760/istio-proxy/1.log" Sep 10 18:50:43.165409 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:43.165392 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-h6mg2_952dcc01-8c30-41c0-9d9f-2964ed5c0760/istio-proxy/0.log" Sep 10 18:50:43.843376 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:43.843349 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-drn5f_7470a24c-2464-4678-9015-4689638c6b57/serve-healthcheck-canary/0.log" Sep 10 18:50:44.326675 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:44.326649 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-chkc2_34adc247-d318-44f5-8588-0dba849035e6/kube-rbac-proxy/0.log" Sep 10 18:50:44.345791 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:44.345762 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-chkc2_34adc247-d318-44f5-8588-0dba849035e6/exporter/0.log" Sep 10 18:50:44.364353 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:44.364317 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-chkc2_34adc247-d318-44f5-8588-0dba849035e6/extractor/0.log" Sep 10 18:50:45.987095 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:45.987067 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ddcqf/perf-node-gather-daemonset-hc6sl" Sep 10 18:50:46.374992 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:46.374966 2581 scope.go:117] "RemoveContainer" containerID="92ed2f3777b80b465a9cf0fe509df826dbe4d15bd3f4282f55d7e1bf6fe721b0" Sep 10 18:50:46.375157 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:50:46.375138 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-jrt76_llm(e1f4af20-061e-4e66-b2c1-d14d0c501e01)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-jrt76" podUID="e1f4af20-061e-4e66-b2c1-d14d0c501e01" Sep 10 18:50:46.521247 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:46.521224 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-mbc8q_d2a51711-565b-4fac-8dfd-c1932700f12a/manager/0.log" Sep 10 18:50:46.553425 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:46.553391 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-pp5xl_5b716014-647a-4159-8511-0349dda108e1/manager/0.log" Sep 10 18:50:46.616555 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:46.616528 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-958jt_5bc21d47-8dd2-4012-815a-588353e3cd58/server/0.log" Sep 10 18:50:46.659445 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:46.659382 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-9rgkh_7eba2f85-27fd-4fe5-8e01-0facd1e8bc78/manager/0.log" Sep 10 18:50:46.743810 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:46.743785 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4_9b8839e0-8ba6-4f99-94b3-c4202c9e1404/manager/0.log" Sep 10 18:50:46.758765 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:46.758742 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bb8cfcdd6-ct8g4_9b8839e0-8ba6-4f99-94b3-c4202c9e1404/copy-manifests/0.log" Sep 10 18:50:48.375055 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:48.374987 2581 scope.go:117] "RemoveContainer" containerID="a1dd741e4736e468d5819bb5cce27a1cd6441f8b2e9bb3b4aeaea7d70f26ae88" Sep 10 18:50:48.375410 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:50:48.375161 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5_llm(b404b14f-e5c2-4cdc-97bc-e0a9e5964d06)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tdrr5" podUID="b404b14f-e5c2-4cdc-97bc-e0a9e5964d06" Sep 10 18:50:50.012491 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:50.012437 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64dfbcccbf-9pgvj_828ca3e2-4140-4cb3-be58-7aa5395d90ac/kube-auth-proxy/0.log" Sep 10 18:50:50.051960 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:50.051930 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-h6mg2_952dcc01-8c30-41c0-9d9f-2964ed5c0760/istio-proxy/1.log" Sep 10 18:50:50.052580 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:50.052557 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-h6mg2_952dcc01-8c30-41c0-9d9f-2964ed5c0760/istio-proxy/0.log" Sep 10 18:50:51.393178 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:51.393153 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64dfbcccbf-9pgvj_828ca3e2-4140-4cb3-be58-7aa5395d90ac/kube-auth-proxy/0.log" Sep 10 18:50:51.438570 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:51.438541 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-h6mg2_952dcc01-8c30-41c0-9d9f-2964ed5c0760/istio-proxy/1.log" Sep 10 18:50:51.438914 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:51.438896 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-h6mg2_952dcc01-8c30-41c0-9d9f-2964ed5c0760/istio-proxy/0.log" Sep 10 18:50:54.376415 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:54.376384 2581 scope.go:117] "RemoveContainer" containerID="471c289ca0d545c991cba42c0ebed39ed23b88abaee9144fa6f5b2ff8a8c3f32" Sep 10 18:50:54.376861 ip-10-0-134-166 kubenswrapper[2581]: E0910 18:50:54.376610 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-lgkff_llm(894e00dc-93dd-4245-846f-3529d796f688)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-lgkff" podUID="894e00dc-93dd-4245-846f-3529d796f688" Sep 10 18:50:54.945063 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:54.944995 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qrb6h_747cbe9d-036e-448f-b567-6ffd3c917b2e/kube-multus-additional-cni-plugins/0.log" Sep 10 18:50:54.963661 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:54.963638 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qrb6h_747cbe9d-036e-448f-b567-6ffd3c917b2e/egress-router-binary-copy/0.log" Sep 10 18:50:54.983473 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:54.983451 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qrb6h_747cbe9d-036e-448f-b567-6ffd3c917b2e/cni-plugins/0.log" Sep 10 18:50:55.004659 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:55.004646 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qrb6h_747cbe9d-036e-448f-b567-6ffd3c917b2e/bond-cni-plugin/0.log" Sep 10 18:50:55.026752 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:55.026723 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qrb6h_747cbe9d-036e-448f-b567-6ffd3c917b2e/routeoverride-cni/0.log" Sep 10 18:50:55.045631 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:55.045614 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qrb6h_747cbe9d-036e-448f-b567-6ffd3c917b2e/whereabouts-cni-bincopy/0.log" Sep 10 18:50:55.064514 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:55.064484 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qrb6h_747cbe9d-036e-448f-b567-6ffd3c917b2e/whereabouts-cni/0.log" Sep 10 18:50:55.095783 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:55.095762 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jc45h_70d9c337-84ae-485f-be5d-9a7c3919f3b0/kube-multus/0.log" Sep 10 18:50:55.152559 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:55.152533 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fqmzj_54ef3c93-1624-46e6-ba09-4fd3bb1fc07f/network-metrics-daemon/0.log" Sep 10 18:50:55.170096 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:55.170069 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fqmzj_54ef3c93-1624-46e6-ba09-4fd3bb1fc07f/kube-rbac-proxy/0.log" Sep 10 18:50:55.985106 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:55.985080 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9j6cn_9eac62b3-150f-42a5-9af2-bf313bf92907/ovn-controller/0.log" Sep 10 18:50:56.001930 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:56.001908 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9j6cn_9eac62b3-150f-42a5-9af2-bf313bf92907/ovn-acl-logging/0.log" Sep 10 18:50:56.025856 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:56.025826 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9j6cn_9eac62b3-150f-42a5-9af2-bf313bf92907/kube-rbac-proxy-node/0.log" Sep 10 18:50:56.044576 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:56.044548 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9j6cn_9eac62b3-150f-42a5-9af2-bf313bf92907/kube-rbac-proxy-ovn-metrics/0.log" Sep 10 18:50:56.059541 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:56.059518 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9j6cn_9eac62b3-150f-42a5-9af2-bf313bf92907/northd/0.log" Sep 10 18:50:56.077553 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:56.077533 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9j6cn_9eac62b3-150f-42a5-9af2-bf313bf92907/nbdb/0.log" Sep 10 18:50:56.096016 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:56.095997 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9j6cn_9eac62b3-150f-42a5-9af2-bf313bf92907/sbdb/0.log" Sep 10 18:50:56.130594 ip-10-0-134-166 kubenswrapper[2581]: I0910 18:50:56.130563 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9j6cn_9eac62b3-150f-42a5-9af2-bf313bf92907/ovnkube-controller/0.log"