Sep 06 12:16:13.681133 ip-10-0-132-159 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 06 12:16:13.681150 ip-10-0-132-159 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 06 12:16:13.681160 ip-10-0-132-159 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 06 12:16:13.681398 ip-10-0-132-159 systemd[1]: Failed to start Kubernetes Kubelet. Sep 06 12:16:23.882916 ip-10-0-132-159 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 06 12:16:23.882928 ip-10-0-132-159 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 7389e74355b24dd4933d678da02955ac -- Sep 06 12:18:04.026041 ip-10-0-132-159 systemd[1]: Starting Kubernetes Kubelet... Sep 06 12:18:04.441083 ip-10-0-132-159 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 06 12:18:04.441083 ip-10-0-132-159 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 06 12:18:04.441083 ip-10-0-132-159 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 06 12:18:04.441530 ip-10-0-132-159 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 06 12:18:04.441530 ip-10-0-132-159 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 06 12:18:04.441530 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.441165 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 06 12:18:04.444603 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444588 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 06 12:18:04.444603 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444602 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 06 12:18:04.444669 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444606 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 06 12:18:04.444669 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444609 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 06 12:18:04.444669 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444612 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 06 12:18:04.444669 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444615 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 06 12:18:04.444669 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444618 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 06 12:18:04.444669 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444621 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 06 12:18:04.444669 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444623 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 06 12:18:04.444669 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444626 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 06 12:18:04.444669 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444629 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 06 12:18:04.444669 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444631 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 06 12:18:04.444669 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444634 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 06 12:18:04.444669 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444637 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 06 12:18:04.444669 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444639 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 06 12:18:04.444669 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444642 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 06 12:18:04.444669 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444644 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 06 12:18:04.444669 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444647 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 06 12:18:04.444669 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444649 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 06 12:18:04.444669 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444652 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 06 12:18:04.444669 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444655 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 06 12:18:04.444669 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444657 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 06 12:18:04.445193 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444660 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 06 12:18:04.445193 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444664 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 06 12:18:04.445193 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444666 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 06 12:18:04.445193 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444669 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 06 12:18:04.445193 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444671 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 06 12:18:04.445193 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444674 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 06 12:18:04.445193 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444677 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 06 12:18:04.445193 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444680 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 06 12:18:04.445193 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444683 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 06 12:18:04.445193 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444685 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 06 12:18:04.445193 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444688 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 06 12:18:04.445193 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444690 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 06 12:18:04.445193 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444693 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 06 12:18:04.445193 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444695 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 06 12:18:04.445193 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444699 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 06 12:18:04.445193 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444703 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 06 12:18:04.445193 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444706 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 06 12:18:04.445193 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444709 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 06 12:18:04.445193 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444712 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 06 12:18:04.445649 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444717 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 06 12:18:04.445649 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444720 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 06 12:18:04.445649 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444723 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 06 12:18:04.445649 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444725 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 06 12:18:04.445649 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444728 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 06 12:18:04.445649 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444730 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 06 12:18:04.445649 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444733 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 06 12:18:04.445649 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444735 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 06 12:18:04.445649 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444738 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 06 12:18:04.445649 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444740 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 06 12:18:04.445649 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444742 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 06 12:18:04.445649 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444745 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 06 12:18:04.445649 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444747 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 06 12:18:04.445649 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444762 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 06 12:18:04.445649 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444766 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 06 12:18:04.445649 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444768 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 06 12:18:04.445649 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444771 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 06 12:18:04.445649 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444774 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 06 12:18:04.445649 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444776 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 06 12:18:04.445649 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444779 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 06 12:18:04.446155 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444782 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 06 12:18:04.446155 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444784 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 06 12:18:04.446155 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444787 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 06 12:18:04.446155 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444790 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 06 12:18:04.446155 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444792 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 06 12:18:04.446155 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444795 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 06 12:18:04.446155 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444798 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 06 12:18:04.446155 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444800 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 06 12:18:04.446155 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444802 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 06 12:18:04.446155 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444805 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 06 12:18:04.446155 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444808 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 06 12:18:04.446155 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444810 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 06 12:18:04.446155 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444814 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 06 12:18:04.446155 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444818 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 06 12:18:04.446155 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444820 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 06 12:18:04.446155 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444823 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 06 12:18:04.446155 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444826 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 06 12:18:04.446155 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444829 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 06 12:18:04.446155 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444832 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 06 12:18:04.446614 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444835 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 06 12:18:04.446614 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444838 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 06 12:18:04.446614 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444841 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 06 12:18:04.446614 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444843 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 06 12:18:04.446614 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444845 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 06 12:18:04.446614 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444849 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 06 12:18:04.446614 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444852 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 06 12:18:04.446614 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444855 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 06 12:18:04.446614 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444857 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 06 12:18:04.446614 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444860 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 06 12:18:04.446614 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444862 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 06 12:18:04.446614 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444865 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 06 12:18:04.446614 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444867 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 06 12:18:04.446614 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444869 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 06 12:18:04.446614 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444872 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 06 12:18:04.446614 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444874 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 06 12:18:04.446614 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444877 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 06 12:18:04.446614 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444879 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 06 12:18:04.446614 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.444882 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 06 12:18:04.446614 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445551 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 06 12:18:04.447104 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445558 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 06 12:18:04.447104 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445561 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 06 12:18:04.447104 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445564 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 06 12:18:04.447104 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445567 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 06 12:18:04.447104 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445570 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 06 12:18:04.447104 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445573 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 06 12:18:04.447104 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445576 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 06 12:18:04.447104 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445578 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 06 12:18:04.447104 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445581 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 06 12:18:04.447104 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445584 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 06 12:18:04.447104 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445586 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 06 12:18:04.447104 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445590 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 06 12:18:04.447104 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445594 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 06 12:18:04.447104 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445597 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 06 12:18:04.447104 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445600 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 06 12:18:04.447104 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445603 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 06 12:18:04.447104 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445605 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 06 12:18:04.447104 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445608 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 06 12:18:04.447104 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445610 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 06 12:18:04.447575 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445614 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 06 12:18:04.447575 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445616 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 06 12:18:04.447575 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445619 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 06 12:18:04.447575 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445621 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 06 12:18:04.447575 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445624 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 06 12:18:04.447575 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445626 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 06 12:18:04.447575 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445629 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 06 12:18:04.447575 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445631 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 06 12:18:04.447575 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445634 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 06 12:18:04.447575 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445636 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 06 12:18:04.447575 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445639 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 06 12:18:04.447575 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445641 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 06 12:18:04.447575 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445644 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 06 12:18:04.447575 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445646 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 06 12:18:04.447575 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445649 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 06 12:18:04.447575 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445652 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 06 12:18:04.447575 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445654 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 06 12:18:04.447575 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445657 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 06 12:18:04.447575 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445660 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 06 12:18:04.447575 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445663 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 06 12:18:04.448088 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445665 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 06 12:18:04.448088 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445668 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 06 12:18:04.448088 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445670 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 06 12:18:04.448088 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445673 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 06 12:18:04.448088 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445675 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 06 12:18:04.448088 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445678 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 06 12:18:04.448088 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445680 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 06 12:18:04.448088 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445683 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 06 12:18:04.448088 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445685 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 06 12:18:04.448088 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445688 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 06 12:18:04.448088 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445690 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 06 12:18:04.448088 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445693 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 06 12:18:04.448088 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445696 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 06 12:18:04.448088 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445699 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 06 12:18:04.448088 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445701 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 06 12:18:04.448088 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445705 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 06 12:18:04.448088 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445709 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 06 12:18:04.448088 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445712 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 06 12:18:04.448088 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445715 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 06 12:18:04.448555 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445717 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 06 12:18:04.448555 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445720 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 06 12:18:04.448555 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445723 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 06 12:18:04.448555 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445726 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 06 12:18:04.448555 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445729 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 06 12:18:04.448555 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445732 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 06 12:18:04.448555 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445734 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 06 12:18:04.448555 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445737 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 06 12:18:04.448555 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445739 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 06 12:18:04.448555 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445741 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 06 12:18:04.448555 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445744 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 06 12:18:04.448555 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445747 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 06 12:18:04.448555 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445769 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 06 12:18:04.448555 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445773 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 06 12:18:04.448555 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445777 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 06 12:18:04.448555 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445781 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 06 12:18:04.448555 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445785 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 06 12:18:04.448555 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445788 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 06 12:18:04.448555 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445791 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 06 12:18:04.448555 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445794 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 06 12:18:04.449066 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445796 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 06 12:18:04.449066 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445800 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 06 12:18:04.449066 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445802 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 06 12:18:04.449066 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445805 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 06 12:18:04.449066 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445807 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 06 12:18:04.449066 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445810 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 06 12:18:04.449066 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445813 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 06 12:18:04.449066 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445815 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 06 12:18:04.449066 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445817 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 06 12:18:04.449066 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445820 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 06 12:18:04.449066 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445822 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 06 12:18:04.449066 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445825 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 06 12:18:04.449066 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445829 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 06 12:18:04.449066 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445833 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 06 12:18:04.449066 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445835 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 06 12:18:04.449066 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445838 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 06 12:18:04.449066 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445840 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 06 12:18:04.449066 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445842 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 06 12:18:04.449066 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445845 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 06 12:18:04.449536 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.445848 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 06 12:18:04.449536 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446191 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 06 12:18:04.449536 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446195 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 06 12:18:04.449536 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446198 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 06 12:18:04.449536 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446201 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 06 12:18:04.449536 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446219 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 06 12:18:04.449536 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446223 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 06 12:18:04.449536 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446226 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 06 12:18:04.449536 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446229 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 06 12:18:04.449536 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446232 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 06 12:18:04.449536 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446236 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 06 12:18:04.449536 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446238 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 06 12:18:04.449536 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446241 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 06 12:18:04.449536 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446244 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 06 12:18:04.449536 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446246 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 06 12:18:04.449536 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446249 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 06 12:18:04.449536 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446252 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 06 12:18:04.449536 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446255 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 06 12:18:04.449536 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446257 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 06 12:18:04.449536 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446260 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 06 12:18:04.450053 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446262 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 06 12:18:04.450053 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446265 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 06 12:18:04.450053 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446267 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 06 12:18:04.450053 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446270 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 06 12:18:04.450053 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446272 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 06 12:18:04.450053 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446274 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 06 12:18:04.450053 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446278 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 06 12:18:04.450053 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446282 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 06 12:18:04.450053 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446284 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 06 12:18:04.450053 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446287 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 06 12:18:04.450053 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446289 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 06 12:18:04.450053 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446292 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 06 12:18:04.450053 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446294 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 06 12:18:04.450053 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446297 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 06 12:18:04.450053 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446299 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 06 12:18:04.450053 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446302 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 06 12:18:04.450053 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446304 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 06 12:18:04.450053 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446307 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 06 12:18:04.450053 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446309 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 06 12:18:04.450549 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446312 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 06 12:18:04.450549 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446315 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 06 12:18:04.450549 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446321 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 06 12:18:04.450549 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446324 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 06 12:18:04.450549 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446327 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 06 12:18:04.450549 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446330 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 06 12:18:04.450549 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446333 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 06 12:18:04.450549 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446335 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 06 12:18:04.450549 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446338 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 06 12:18:04.450549 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446340 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 06 12:18:04.450549 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446343 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 06 12:18:04.450549 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446345 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 06 12:18:04.450549 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446347 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 06 12:18:04.450549 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446350 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 06 12:18:04.450549 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446352 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 06 12:18:04.450549 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446355 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 06 12:18:04.450549 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446357 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 06 12:18:04.450549 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446360 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 06 12:18:04.450549 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446363 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 06 12:18:04.451009 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446365 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 06 12:18:04.451009 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446368 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 06 12:18:04.451009 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446370 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 06 12:18:04.451009 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446373 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 06 12:18:04.451009 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446375 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 06 12:18:04.451009 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446378 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 06 12:18:04.451009 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446380 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 06 12:18:04.451009 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446383 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 06 12:18:04.451009 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446385 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 06 12:18:04.451009 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446388 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 06 12:18:04.451009 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446391 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 06 12:18:04.451009 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446393 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 06 12:18:04.451009 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446395 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 06 12:18:04.451009 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446398 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 06 12:18:04.451009 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446401 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 06 12:18:04.451009 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446405 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 06 12:18:04.451009 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446407 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 06 12:18:04.451009 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446410 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 06 12:18:04.451009 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446413 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 06 12:18:04.451468 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446417 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 06 12:18:04.451468 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446420 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 06 12:18:04.451468 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446423 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 06 12:18:04.451468 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446425 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 06 12:18:04.451468 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446428 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 06 12:18:04.451468 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446430 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 06 12:18:04.451468 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446433 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 06 12:18:04.451468 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446435 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 06 12:18:04.451468 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446438 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 06 12:18:04.451468 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446441 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 06 12:18:04.451468 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446443 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 06 12:18:04.451468 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446446 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 06 12:18:04.451468 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446448 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 06 12:18:04.451468 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446451 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 06 12:18:04.451468 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446453 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 06 12:18:04.451468 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446456 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 06 12:18:04.451468 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446458 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 06 12:18:04.451468 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446461 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 06 12:18:04.451468 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446463 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 06 12:18:04.451468 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446466 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 06 12:18:04.452017 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446468 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 06 12:18:04.452017 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446471 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 06 12:18:04.452017 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.446473 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 06 12:18:04.452017 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446537 2576 flags.go:64] FLAG: --address="0.0.0.0" Sep 06 12:18:04.452017 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446578 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 06 12:18:04.452017 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446590 2576 flags.go:64] FLAG: --anonymous-auth="true" Sep 06 12:18:04.452017 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446594 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 06 12:18:04.452017 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446599 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 06 12:18:04.452017 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446604 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 06 12:18:04.452017 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446608 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 06 12:18:04.452017 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446612 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 06 12:18:04.452017 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446615 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 06 12:18:04.452017 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446619 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 06 12:18:04.452017 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446622 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 06 12:18:04.452017 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446626 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 06 12:18:04.452017 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446629 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 06 12:18:04.452017 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446631 2576 flags.go:64] FLAG: --cgroup-root="" Sep 06 12:18:04.452017 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446634 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 06 12:18:04.452017 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446637 2576 flags.go:64] FLAG: --client-ca-file="" Sep 06 12:18:04.452017 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446640 2576 flags.go:64] FLAG: --cloud-provider="external" Sep 06 12:18:04.452017 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446642 2576 flags.go:64] FLAG: --cluster-dns="[]" Sep 06 12:18:04.452017 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446645 2576 flags.go:64] FLAG: --cluster-domain="" Sep 06 12:18:04.452017 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446648 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 06 12:18:04.452017 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446651 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 06 12:18:04.452601 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446654 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 06 12:18:04.452601 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446657 2576 flags.go:64] FLAG: --container-log-max-files="5" Sep 06 12:18:04.452601 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446661 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 06 12:18:04.452601 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446663 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 06 12:18:04.452601 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446666 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 06 12:18:04.452601 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446669 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 06 12:18:04.452601 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446672 2576 flags.go:64] FLAG: --contention-profiling="false" Sep 06 12:18:04.452601 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446675 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 06 12:18:04.452601 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446677 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 06 12:18:04.452601 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446680 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 06 12:18:04.452601 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446683 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 06 12:18:04.452601 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446689 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 06 12:18:04.452601 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446692 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 06 12:18:04.452601 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446695 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 06 12:18:04.452601 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446698 2576 flags.go:64] FLAG: --enable-load-reader="false" Sep 06 12:18:04.452601 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446702 2576 flags.go:64] FLAG: --enable-server="true" Sep 06 12:18:04.452601 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446706 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 06 12:18:04.452601 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446710 2576 flags.go:64] FLAG: --event-burst="100" Sep 06 12:18:04.452601 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446713 2576 flags.go:64] FLAG: --event-qps="50" Sep 06 12:18:04.452601 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446716 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 06 12:18:04.452601 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446719 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 06 12:18:04.452601 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446721 2576 flags.go:64] FLAG: --eviction-hard="" Sep 06 12:18:04.452601 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446725 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 06 12:18:04.452601 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446728 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 06 12:18:04.453182 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446730 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 06 12:18:04.453182 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446733 2576 flags.go:64] FLAG: --eviction-soft="" Sep 06 12:18:04.453182 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446736 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 06 12:18:04.453182 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446739 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 06 12:18:04.453182 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446741 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 06 12:18:04.453182 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446744 2576 flags.go:64] FLAG: --experimental-mounter-path="" Sep 06 12:18:04.453182 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446747 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 06 12:18:04.453182 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446764 2576 flags.go:64] FLAG: --fail-swap-on="true" Sep 06 12:18:04.453182 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446767 2576 flags.go:64] FLAG: --feature-gates="" Sep 06 12:18:04.453182 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446770 2576 flags.go:64] FLAG: --file-check-frequency="20s" Sep 06 12:18:04.453182 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446774 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 06 12:18:04.453182 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446777 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 06 12:18:04.453182 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446780 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 06 12:18:04.453182 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446783 2576 flags.go:64] FLAG: --healthz-port="10248" Sep 06 12:18:04.453182 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446786 2576 flags.go:64] FLAG: --help="false" Sep 06 12:18:04.453182 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446789 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-132-159.ec2.internal" Sep 06 12:18:04.453182 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446791 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 06 12:18:04.453182 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446794 2576 flags.go:64] FLAG: --http-check-frequency="20s" Sep 06 12:18:04.453182 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446797 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 06 12:18:04.453182 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446800 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 06 12:18:04.453182 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446803 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 06 12:18:04.453182 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446806 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 06 12:18:04.453182 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446809 2576 flags.go:64] FLAG: --image-service-endpoint="" Sep 06 12:18:04.453765 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446812 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 06 12:18:04.453765 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446817 2576 flags.go:64] FLAG: --kube-api-burst="100" Sep 06 12:18:04.453765 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446820 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 06 12:18:04.453765 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446823 2576 flags.go:64] FLAG: --kube-api-qps="50" Sep 06 12:18:04.453765 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446826 2576 flags.go:64] FLAG: --kube-reserved="" Sep 06 12:18:04.453765 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446829 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 06 12:18:04.453765 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446832 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 06 12:18:04.453765 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446834 2576 flags.go:64] FLAG: --kubelet-cgroups="" Sep 06 12:18:04.453765 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446837 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 06 12:18:04.453765 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446840 2576 flags.go:64] FLAG: --lock-file="" Sep 06 12:18:04.453765 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446843 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 06 12:18:04.453765 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446846 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 06 12:18:04.453765 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446848 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 06 12:18:04.453765 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446854 2576 flags.go:64] FLAG: --log-json-split-stream="false" Sep 06 12:18:04.453765 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446857 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 06 12:18:04.453765 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446859 2576 flags.go:64] FLAG: --log-text-split-stream="false" Sep 06 12:18:04.453765 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446862 2576 flags.go:64] FLAG: --logging-format="text" Sep 06 12:18:04.453765 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446865 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 06 12:18:04.453765 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446868 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 06 12:18:04.453765 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446870 2576 flags.go:64] FLAG: --manifest-url="" Sep 06 12:18:04.453765 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446873 2576 flags.go:64] FLAG: --manifest-url-header="" Sep 06 12:18:04.453765 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446877 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 06 12:18:04.453765 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446880 2576 flags.go:64] FLAG: --max-open-files="1000000" Sep 06 12:18:04.453765 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446883 2576 flags.go:64] FLAG: --max-pods="110" Sep 06 12:18:04.453765 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446886 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 06 12:18:04.454385 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446889 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 06 12:18:04.454385 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446892 2576 flags.go:64] FLAG: --memory-manager-policy="None" Sep 06 12:18:04.454385 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446894 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 06 12:18:04.454385 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446897 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 06 12:18:04.454385 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446900 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 06 12:18:04.454385 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446903 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 06 12:18:04.454385 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446909 2576 flags.go:64] FLAG: --node-status-max-images="50" Sep 06 12:18:04.454385 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446913 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 06 12:18:04.454385 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446917 2576 flags.go:64] FLAG: --oom-score-adj="-999" Sep 06 12:18:04.454385 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446920 2576 flags.go:64] FLAG: --pod-cidr="" Sep 06 12:18:04.454385 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446923 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 06 12:18:04.454385 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446928 2576 flags.go:64] FLAG: --pod-manifest-path="" Sep 06 12:18:04.454385 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446931 2576 flags.go:64] FLAG: --pod-max-pids="-1" Sep 06 12:18:04.454385 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446934 2576 flags.go:64] FLAG: --pods-per-core="0" Sep 06 12:18:04.454385 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446936 2576 flags.go:64] FLAG: --port="10250" Sep 06 12:18:04.454385 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446939 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 06 12:18:04.454385 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446942 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c8ac3cd6ec2ba862" Sep 06 12:18:04.454385 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446945 2576 flags.go:64] FLAG: --qos-reserved="" Sep 06 12:18:04.454385 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446948 2576 flags.go:64] FLAG: --read-only-port="10255" Sep 06 12:18:04.454385 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446951 2576 flags.go:64] FLAG: --register-node="true" Sep 06 12:18:04.454385 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446953 2576 flags.go:64] FLAG: --register-with-taints="" Sep 06 12:18:04.454385 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446957 2576 flags.go:64] FLAG: --registry-burst="10" Sep 06 12:18:04.454385 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446960 2576 flags.go:64] FLAG: --registry-qps="5" Sep 06 12:18:04.454385 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446963 2576 flags.go:64] FLAG: --reserved-cpus="" Sep 06 12:18:04.454981 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446965 2576 flags.go:64] FLAG: --reserved-memory="" Sep 06 12:18:04.454981 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446969 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 06 12:18:04.454981 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446972 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 06 12:18:04.454981 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446975 2576 flags.go:64] FLAG: --rotate-certificates="false" Sep 06 12:18:04.454981 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446977 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 06 12:18:04.454981 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446980 2576 flags.go:64] FLAG: --runonce="false" Sep 06 12:18:04.454981 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446983 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 06 12:18:04.454981 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446986 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 06 12:18:04.454981 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446989 2576 flags.go:64] FLAG: --seccomp-default="false" Sep 06 12:18:04.454981 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446992 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 06 12:18:04.454981 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446994 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 06 12:18:04.454981 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.446997 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 06 12:18:04.454981 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.447000 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 06 12:18:04.454981 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.447003 2576 flags.go:64] FLAG: --storage-driver-password="root" Sep 06 12:18:04.454981 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.447005 2576 flags.go:64] FLAG: --storage-driver-secure="false" Sep 06 12:18:04.454981 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.447008 2576 flags.go:64] FLAG: --storage-driver-table="stats" Sep 06 12:18:04.454981 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.447013 2576 flags.go:64] FLAG: --storage-driver-user="root" Sep 06 12:18:04.454981 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.447016 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 06 12:18:04.454981 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.447019 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 06 12:18:04.454981 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.447022 2576 flags.go:64] FLAG: --system-cgroups="" Sep 06 12:18:04.454981 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.447025 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 06 12:18:04.454981 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.447030 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 06 12:18:04.454981 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.447033 2576 flags.go:64] FLAG: --tls-cert-file="" Sep 06 12:18:04.454981 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.447035 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 06 12:18:04.454981 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.447039 2576 flags.go:64] FLAG: --tls-min-version="" Sep 06 12:18:04.455626 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.447041 2576 flags.go:64] FLAG: --tls-private-key-file="" Sep 06 12:18:04.455626 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.447044 2576 flags.go:64] FLAG: --topology-manager-policy="none" Sep 06 12:18:04.455626 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.447047 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 06 12:18:04.455626 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.447050 2576 flags.go:64] FLAG: --topology-manager-scope="container" Sep 06 12:18:04.455626 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.447052 2576 flags.go:64] FLAG: --v="2" Sep 06 12:18:04.455626 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.447056 2576 flags.go:64] FLAG: --version="false" Sep 06 12:18:04.455626 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.447059 2576 flags.go:64] FLAG: --vmodule="" Sep 06 12:18:04.455626 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.447063 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 06 12:18:04.455626 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.447066 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 06 12:18:04.455626 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447176 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 06 12:18:04.455626 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447180 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 06 12:18:04.455626 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447184 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 06 12:18:04.455626 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447186 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 06 12:18:04.455626 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447189 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 06 12:18:04.455626 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447192 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 06 12:18:04.455626 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447194 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 06 12:18:04.455626 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447197 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 06 12:18:04.455626 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447199 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 06 12:18:04.455626 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447204 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 06 12:18:04.455626 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447207 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 06 12:18:04.455626 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447210 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 06 12:18:04.455626 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447213 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 06 12:18:04.456237 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447216 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 06 12:18:04.456237 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447220 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 06 12:18:04.456237 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447223 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 06 12:18:04.456237 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447225 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 06 12:18:04.456237 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447228 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 06 12:18:04.456237 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447232 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 06 12:18:04.456237 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447234 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 06 12:18:04.456237 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447238 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 06 12:18:04.456237 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447241 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 06 12:18:04.456237 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447244 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 06 12:18:04.456237 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447248 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 06 12:18:04.456237 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447250 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 06 12:18:04.456237 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447253 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 06 12:18:04.456237 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447256 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 06 12:18:04.456237 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447258 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 06 12:18:04.456237 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447261 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 06 12:18:04.456237 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447264 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 06 12:18:04.456237 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447266 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 06 12:18:04.456237 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447269 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 06 12:18:04.456237 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447271 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 06 12:18:04.456727 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447274 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 06 12:18:04.456727 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447276 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 06 12:18:04.456727 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447279 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 06 12:18:04.456727 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447281 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 06 12:18:04.456727 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447284 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 06 12:18:04.456727 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447286 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 06 12:18:04.456727 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447290 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 06 12:18:04.456727 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447293 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 06 12:18:04.456727 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447296 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 06 12:18:04.456727 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447298 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 06 12:18:04.456727 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447301 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 06 12:18:04.456727 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447303 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 06 12:18:04.456727 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447306 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 06 12:18:04.456727 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447311 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 06 12:18:04.456727 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447313 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 06 12:18:04.456727 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447316 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 06 12:18:04.456727 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447318 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 06 12:18:04.456727 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447320 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 06 12:18:04.456727 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447323 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 06 12:18:04.456727 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447326 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 06 12:18:04.457233 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447329 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 06 12:18:04.457233 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447331 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 06 12:18:04.457233 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447334 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 06 12:18:04.457233 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447336 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 06 12:18:04.457233 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447339 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 06 12:18:04.457233 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447341 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 06 12:18:04.457233 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447344 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 06 12:18:04.457233 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447346 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 06 12:18:04.457233 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447349 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 06 12:18:04.457233 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447351 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 06 12:18:04.457233 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447354 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 06 12:18:04.457233 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447356 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 06 12:18:04.457233 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447359 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 06 12:18:04.457233 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447361 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 06 12:18:04.457233 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447363 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 06 12:18:04.457233 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447366 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 06 12:18:04.457233 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447368 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 06 12:18:04.457233 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447371 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 06 12:18:04.457233 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447373 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 06 12:18:04.457792 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447376 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 06 12:18:04.457792 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447379 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 06 12:18:04.457792 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447381 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 06 12:18:04.457792 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447384 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 06 12:18:04.457792 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447386 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 06 12:18:04.457792 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447389 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 06 12:18:04.457792 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447393 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 06 12:18:04.457792 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447395 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 06 12:18:04.457792 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447398 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 06 12:18:04.457792 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447400 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 06 12:18:04.457792 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447403 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 06 12:18:04.457792 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447406 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 06 12:18:04.457792 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447409 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 06 12:18:04.457792 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447411 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 06 12:18:04.457792 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447414 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 06 12:18:04.457792 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447416 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 06 12:18:04.457792 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447418 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 06 12:18:04.457792 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447421 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 06 12:18:04.457792 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447423 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 06 12:18:04.457792 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447426 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 06 12:18:04.458286 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447428 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 06 12:18:04.458286 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447431 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 06 12:18:04.458286 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447434 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 06 12:18:04.458286 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447437 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 06 12:18:04.458286 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447439 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 06 12:18:04.458286 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447441 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 06 12:18:04.458286 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.447444 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 06 12:18:04.458286 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.448043 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 06 12:18:04.458286 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.454363 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 06 12:18:04.458286 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.454377 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 06 12:18:04.458286 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.456039 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 06 12:18:04.458286 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.456047 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 06 12:18:04.458286 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456107 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 06 12:18:04.458286 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456112 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 06 12:18:04.458286 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456115 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 06 12:18:04.458657 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456118 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 06 12:18:04.458657 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456121 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 06 12:18:04.458657 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456123 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 06 12:18:04.458657 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456126 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 06 12:18:04.458657 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456128 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 06 12:18:04.458657 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456133 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 06 12:18:04.458657 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456136 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 06 12:18:04.458657 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456139 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 06 12:18:04.458657 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456142 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 06 12:18:04.458657 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456146 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 06 12:18:04.458657 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456149 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 06 12:18:04.458657 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456152 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 06 12:18:04.458657 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456155 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 06 12:18:04.458657 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456158 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 06 12:18:04.458657 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456161 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 06 12:18:04.458657 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456164 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 06 12:18:04.458657 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456167 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 06 12:18:04.458657 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456169 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 06 12:18:04.458657 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456172 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 06 12:18:04.459183 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456175 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 06 12:18:04.459183 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456178 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 06 12:18:04.459183 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456180 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 06 12:18:04.459183 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456183 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 06 12:18:04.459183 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456186 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 06 12:18:04.459183 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456189 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 06 12:18:04.459183 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456191 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 06 12:18:04.459183 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456194 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 06 12:18:04.459183 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456197 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 06 12:18:04.459183 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456199 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 06 12:18:04.459183 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456202 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 06 12:18:04.459183 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456204 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 06 12:18:04.459183 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456207 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 06 12:18:04.459183 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456210 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 06 12:18:04.459183 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456212 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 06 12:18:04.459183 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456215 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 06 12:18:04.459183 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456217 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 06 12:18:04.459183 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456220 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 06 12:18:04.459183 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456222 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 06 12:18:04.459183 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456225 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 06 12:18:04.459676 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456227 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 06 12:18:04.459676 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456229 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 06 12:18:04.459676 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456232 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 06 12:18:04.459676 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456235 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 06 12:18:04.459676 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456237 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 06 12:18:04.459676 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456240 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 06 12:18:04.459676 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456242 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 06 12:18:04.459676 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456245 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 06 12:18:04.459676 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456249 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 06 12:18:04.459676 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456253 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 06 12:18:04.459676 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456257 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 06 12:18:04.459676 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456261 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 06 12:18:04.459676 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456265 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 06 12:18:04.459676 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456268 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 06 12:18:04.459676 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456271 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 06 12:18:04.459676 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456274 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 06 12:18:04.459676 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456276 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 06 12:18:04.459676 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456279 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 06 12:18:04.459676 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456282 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 06 12:18:04.460150 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456285 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 06 12:18:04.460150 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456287 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 06 12:18:04.460150 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456290 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 06 12:18:04.460150 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456292 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 06 12:18:04.460150 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456295 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 06 12:18:04.460150 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456297 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 06 12:18:04.460150 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456300 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 06 12:18:04.460150 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456302 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 06 12:18:04.460150 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456304 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 06 12:18:04.460150 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456307 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 06 12:18:04.460150 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456309 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 06 12:18:04.460150 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456312 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 06 12:18:04.460150 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456315 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 06 12:18:04.460150 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456317 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 06 12:18:04.460150 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456319 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 06 12:18:04.460150 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456322 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 06 12:18:04.460150 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456324 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 06 12:18:04.460150 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456327 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 06 12:18:04.460150 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456330 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 06 12:18:04.460150 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456332 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 06 12:18:04.460638 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456335 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 06 12:18:04.460638 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456337 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 06 12:18:04.460638 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456340 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 06 12:18:04.460638 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456342 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 06 12:18:04.460638 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456345 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 06 12:18:04.460638 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456348 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 06 12:18:04.460638 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456350 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 06 12:18:04.460638 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456352 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 06 12:18:04.460638 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456355 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 06 12:18:04.460638 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456358 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 06 12:18:04.460638 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456360 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 06 12:18:04.460638 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456363 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 06 12:18:04.460638 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456365 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 06 12:18:04.460638 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456367 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 06 12:18:04.460638 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456370 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 06 12:18:04.460638 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456372 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 06 12:18:04.460638 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456375 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 06 12:18:04.460638 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456377 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 06 12:18:04.461080 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.456382 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 06 12:18:04.461080 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456482 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 06 12:18:04.461080 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456486 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 06 12:18:04.461080 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456489 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 06 12:18:04.461080 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456492 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 06 12:18:04.461080 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456495 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 06 12:18:04.461080 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456497 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 06 12:18:04.461080 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456500 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 06 12:18:04.461080 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456502 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 06 12:18:04.461080 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456505 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 06 12:18:04.461080 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456507 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 06 12:18:04.461080 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456510 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 06 12:18:04.461080 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456512 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 06 12:18:04.461080 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456515 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 06 12:18:04.461080 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456517 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 06 12:18:04.461443 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456520 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 06 12:18:04.461443 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456523 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 06 12:18:04.461443 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456527 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 06 12:18:04.461443 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456530 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 06 12:18:04.461443 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456533 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 06 12:18:04.461443 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456535 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 06 12:18:04.461443 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456538 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 06 12:18:04.461443 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456540 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 06 12:18:04.461443 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456544 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 06 12:18:04.461443 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456547 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 06 12:18:04.461443 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456550 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 06 12:18:04.461443 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456553 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 06 12:18:04.461443 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456555 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 06 12:18:04.461443 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456558 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 06 12:18:04.461443 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456560 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 06 12:18:04.461443 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456563 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 06 12:18:04.461443 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456565 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 06 12:18:04.461443 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456568 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 06 12:18:04.461443 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456570 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 06 12:18:04.461924 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456572 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 06 12:18:04.461924 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456575 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 06 12:18:04.461924 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456577 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 06 12:18:04.461924 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456579 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 06 12:18:04.461924 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456582 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 06 12:18:04.461924 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456584 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 06 12:18:04.461924 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456587 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 06 12:18:04.461924 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456591 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 06 12:18:04.461924 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456594 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 06 12:18:04.461924 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456597 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 06 12:18:04.461924 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456599 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 06 12:18:04.461924 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456603 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 06 12:18:04.461924 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456606 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 06 12:18:04.461924 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456608 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 06 12:18:04.461924 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456611 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 06 12:18:04.461924 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456613 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 06 12:18:04.461924 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456616 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 06 12:18:04.461924 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456618 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 06 12:18:04.461924 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456621 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 06 12:18:04.462367 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456623 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 06 12:18:04.462367 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456625 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 06 12:18:04.462367 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456628 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 06 12:18:04.462367 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456630 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 06 12:18:04.462367 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456633 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 06 12:18:04.462367 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456635 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 06 12:18:04.462367 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456638 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 06 12:18:04.462367 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456640 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 06 12:18:04.462367 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456642 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 06 12:18:04.462367 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456645 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 06 12:18:04.462367 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456647 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 06 12:18:04.462367 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456649 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 06 12:18:04.462367 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456652 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 06 12:18:04.462367 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456654 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 06 12:18:04.462367 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456657 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 06 12:18:04.462367 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456659 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 06 12:18:04.462367 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456662 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 06 12:18:04.462367 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456664 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 06 12:18:04.462367 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456666 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 06 12:18:04.462367 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456669 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 06 12:18:04.462864 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456671 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 06 12:18:04.462864 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456674 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 06 12:18:04.462864 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456676 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 06 12:18:04.462864 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456678 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 06 12:18:04.462864 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456681 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 06 12:18:04.462864 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456683 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 06 12:18:04.462864 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456686 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 06 12:18:04.462864 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456688 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 06 12:18:04.462864 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456691 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 06 12:18:04.462864 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456693 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 06 12:18:04.462864 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456695 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 06 12:18:04.462864 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456698 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 06 12:18:04.462864 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456700 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 06 12:18:04.462864 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456703 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 06 12:18:04.462864 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456705 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 06 12:18:04.462864 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456707 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 06 12:18:04.462864 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456710 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 06 12:18:04.462864 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456712 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 06 12:18:04.462864 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456714 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 06 12:18:04.462864 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456717 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 06 12:18:04.463346 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456719 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 06 12:18:04.463346 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456721 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 06 12:18:04.463346 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456724 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 06 12:18:04.463346 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456726 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 06 12:18:04.463346 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456729 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 06 12:18:04.463346 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456731 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 06 12:18:04.463346 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:04.456734 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 06 12:18:04.463346 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.456739 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 06 12:18:04.463346 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.456839 2576 server.go:962] "Client rotation is on, will bootstrap in background" Sep 06 12:18:04.463346 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.458841 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 06 12:18:04.463346 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.459893 2576 server.go:1019] "Starting client certificate rotation" Sep 06 12:18:04.463346 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.459988 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 06 12:18:04.463346 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.460030 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 06 12:18:04.463679 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.462671 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 06 12:18:04.464372 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.464346 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 06 12:18:04.473698 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.473679 2576 log.go:25] "Validated CRI v1 runtime API" Sep 06 12:18:04.476769 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.476739 2576 log.go:25] "Validated CRI v1 image API" Sep 06 12:18:04.477576 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.477556 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 06 12:18:04.481042 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.481022 2576 fs.go:135] Filesystem UUIDs: map[0b912428-78a7-41e1-b505-b2dfef675826:/dev/nvme0n1p4 50f070ff-62f0-4499-9bc5-62bdaf27afc0:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Sep 06 12:18:04.481116 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.481041 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 06 12:18:04.487021 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.486900 2576 manager.go:216] Machine: {Timestamp:2026-09-06 12:18:04.4855577 +0000 UTC m=+0.341315598 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098759 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec297c3639e5ca688f951814db96d299 SystemUUID:ec297c36-39e5-ca68-8f95-1814db96d299 BootID:7389e743-55b2-4dd4-933d-678da02955ac Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d6:13:d5:be:c9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d6:13:d5:be:c9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:6c:61:26:cf:7e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 06 12:18:04.487021 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.487013 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 06 12:18:04.487176 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.487107 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 06 12:18:04.490633 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.490618 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 06 12:18:04.490872 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.490850 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 06 12:18:04.491013 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.490877 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-159.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 06 12:18:04.491677 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.491667 2576 topology_manager.go:138] "Creating topology manager with none policy" Sep 06 12:18:04.491716 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.491680 2576 container_manager_linux.go:309] "Creating device plugin manager" Sep 06 12:18:04.491716 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.491688 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 06 12:18:04.492493 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.492480 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 06 12:18:04.494142 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.494132 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 06 12:18:04.497638 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.497626 2576 state_mem.go:36] "Initialized new in-memory state store" Sep 06 12:18:04.497762 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.497740 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 06 12:18:04.497815 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.497782 2576 kubelet.go:486] "Attempting to sync node with API server" Sep 06 12:18:04.497815 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.497793 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 06 12:18:04.497815 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.497808 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 06 12:18:04.497939 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.497819 2576 kubelet.go:398] "Adding apiserver pod source" Sep 06 12:18:04.497939 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.497828 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 06 12:18:04.498151 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.498138 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 06 12:18:04.498186 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.498162 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 06 12:18:04.500439 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.500421 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 06 12:18:04.502198 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.502184 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 06 12:18:04.502256 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.502213 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 06 12:18:04.502934 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.502922 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 06 12:18:04.502967 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.502940 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 06 12:18:04.502967 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.502946 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 06 12:18:04.502967 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.502951 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 06 12:18:04.502967 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.502960 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 06 12:18:04.503105 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.502969 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 06 12:18:04.503105 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.502983 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 06 12:18:04.503105 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.502989 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 06 12:18:04.503105 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.502995 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 06 12:18:04.503105 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.503001 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 06 12:18:04.503105 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.503009 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 06 12:18:04.503105 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.503017 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 06 12:18:04.504996 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.504980 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 06 12:18:04.504996 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.504994 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 06 12:18:04.508620 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.508607 2576 server.go:1268] "Started kubelet" Sep 06 12:18:04.508728 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.508703 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 06 12:18:04.508846 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.508748 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 06 12:18:04.511622 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.511600 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-159.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 06 12:18:04.511806 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:04.511745 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 06 12:18:04.512008 ip-10-0-132-159 systemd[1]: Started Kubernetes Kubelet. Sep 06 12:18:04.512217 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:04.512003 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-159.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 06 12:18:04.512217 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.512148 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 06 12:18:04.512318 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.508707 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 06 12:18:04.514189 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:04.514145 2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 06 12:18:04.515451 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.515434 2576 server.go:310] "Adding debug handlers to kubelet server" Sep 06 12:18:04.517313 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.517293 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 06 12:18:04.517313 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.517304 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 06 12:18:04.517588 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:04.517478 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-159.ec2.internal\" not found" Sep 06 12:18:04.517889 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.517839 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 06 12:18:04.517961 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.517905 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 06 12:18:04.517961 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.517923 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 06 12:18:04.518037 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.517986 2576 reconstruct.go:97] "Volume reconstruction finished" Sep 06 12:18:04.518037 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.517995 2576 reconciler.go:29] "Reconciler: start to sync state" Sep 06 12:18:04.518161 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.518147 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 06 12:18:04.518161 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.518161 2576 factory.go:55] Registering systemd factory Sep 06 12:18:04.518257 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.518168 2576 factory.go:223] Registration of the systemd container factory successfully Sep 06 12:18:04.518322 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:04.517138 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-159.ec2.internal.18d2ba857ce92fdc default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-159.ec2.internal,UID:ip-10-0-132-159.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-159.ec2.internal,},FirstTimestamp:2026-09-06 12:18:04.508581852 +0000 UTC m=+0.364339730,LastTimestamp:2026-09-06 12:18:04.508581852 +0000 UTC m=+0.364339730,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-159.ec2.internal,}" Sep 06 12:18:04.518385 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.518354 2576 factory.go:153] Registering CRI-O factory Sep 06 12:18:04.518385 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.518368 2576 factory.go:223] Registration of the crio container factory successfully Sep 06 12:18:04.518385 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.518383 2576 factory.go:103] Registering Raw factory Sep 06 12:18:04.518497 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.518396 2576 manager.go:1196] Started watching for new ooms in manager Sep 06 12:18:04.518807 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.518795 2576 manager.go:349] Starting recovery of all containers Sep 06 12:18:04.519975 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.519956 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4l8sd" Sep 06 12:18:04.525257 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:04.525167 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 06 12:18:04.525257 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:04.525212 2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-159.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 06 12:18:04.525910 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.525881 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 06 12:18:04.527732 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.527715 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4l8sd" Sep 06 12:18:04.528585 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.528562 2576 manager.go:354] Recovery completed Sep 06 12:18:04.533110 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.533092 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 06 12:18:04.534712 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.534696 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-159.ec2.internal" event="NodeHasSufficientMemory" Sep 06 12:18:04.534782 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.534734 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-159.ec2.internal" event="NodeHasNoDiskPressure" Sep 06 12:18:04.534782 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.534745 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-159.ec2.internal" event="NodeHasSufficientPID" Sep 06 12:18:04.535219 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.535206 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 06 12:18:04.535219 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.535219 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 06 12:18:04.535332 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.535220 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 06 12:18:04.535332 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.535242 2576 state_mem.go:36] "Initialized new in-memory state store" Sep 06 12:18:04.535925 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.535911 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-159.ec2.internal" event="NodeHasSufficientMemory" Sep 06 12:18:04.535966 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.535936 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-159.ec2.internal" event="NodeHasNoDiskPressure" Sep 06 12:18:04.535966 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.535945 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-159.ec2.internal" event="NodeHasSufficientPID" Sep 06 12:18:04.536868 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:04.536807 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-159.ec2.internal.18d2ba857e77e650 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-159.ec2.internal,UID:ip-10-0-132-159.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-159.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-159.ec2.internal,},FirstTimestamp:2026-09-06 12:18:04.534711888 +0000 UTC m=+0.390469768,LastTimestamp:2026-09-06 12:18:04.534711888 +0000 UTC m=+0.390469768,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-159.ec2.internal,}" Sep 06 12:18:04.537366 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.537353 2576 policy_none.go:49] "None policy: Start" Sep 06 12:18:04.537422 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.537372 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 06 12:18:04.537422 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.537385 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 06 12:18:04.539371 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.539359 2576 policy_none.go:47] "Start" Sep 06 12:18:04.580337 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.580319 2576 manager.go:337] "Starting Device Plugin manager" Sep 06 12:18:04.597133 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:04.580392 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 06 12:18:04.597133 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.580405 2576 server.go:87] "Starting device plugin registration server" Sep 06 12:18:04.597133 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.580595 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 06 12:18:04.597133 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.580605 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 06 12:18:04.597133 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.580705 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 06 12:18:04.597133 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.580813 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 06 12:18:04.597133 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.580822 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 06 12:18:04.597133 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:04.581340 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 06 12:18:04.597133 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:04.581391 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-159.ec2.internal\" not found" Sep 06 12:18:04.646300 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.646281 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 06 12:18:04.646390 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.646343 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 06 12:18:04.646390 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.646361 2576 kubelet.go:2448] "Starting kubelet main sync loop" Sep 06 12:18:04.646467 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:04.646392 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 06 12:18:04.648533 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.648517 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 06 12:18:04.681431 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.681413 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 06 12:18:04.682221 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.682205 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-159.ec2.internal" event="NodeHasSufficientMemory" Sep 06 12:18:04.682289 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.682234 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-159.ec2.internal" event="NodeHasNoDiskPressure" Sep 06 12:18:04.682289 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.682245 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-159.ec2.internal" event="NodeHasSufficientPID" Sep 06 12:18:04.682289 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.682267 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-159.ec2.internal" Sep 06 12:18:04.690221 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.690178 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-159.ec2.internal" Sep 06 12:18:04.690221 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:04.690198 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-159.ec2.internal\": node \"ip-10-0-132-159.ec2.internal\" not found" Sep 06 12:18:04.704301 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:04.704281 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-159.ec2.internal\" not found" Sep 06 12:18:04.746763 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.746734 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-159.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-159.ec2.internal"] Sep 06 12:18:04.746824 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.746810 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 06 12:18:04.747537 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.747523 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-159.ec2.internal" event="NodeHasSufficientMemory" Sep 06 12:18:04.747613 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.747551 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-159.ec2.internal" event="NodeHasNoDiskPressure" Sep 06 12:18:04.747613 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.747560 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-159.ec2.internal" event="NodeHasSufficientPID" Sep 06 12:18:04.749721 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.749709 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 06 12:18:04.749866 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.749852 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-159.ec2.internal" Sep 06 12:18:04.749909 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.749881 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 06 12:18:04.750424 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.750407 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-159.ec2.internal" event="NodeHasSufficientMemory" Sep 06 12:18:04.750486 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.750441 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-159.ec2.internal" event="NodeHasNoDiskPressure" Sep 06 12:18:04.750486 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.750447 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-159.ec2.internal" event="NodeHasSufficientMemory" Sep 06 12:18:04.750486 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.750453 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-159.ec2.internal" event="NodeHasSufficientPID" Sep 06 12:18:04.750486 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.750462 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-159.ec2.internal" event="NodeHasNoDiskPressure" Sep 06 12:18:04.750486 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.750471 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-159.ec2.internal" event="NodeHasSufficientPID" Sep 06 12:18:04.752607 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.752588 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-159.ec2.internal" Sep 06 12:18:04.752700 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.752615 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 06 12:18:04.753241 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.753225 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-159.ec2.internal" event="NodeHasSufficientMemory" Sep 06 12:18:04.753312 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.753269 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-159.ec2.internal" event="NodeHasNoDiskPressure" Sep 06 12:18:04.753312 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.753281 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-159.ec2.internal" event="NodeHasSufficientPID" Sep 06 12:18:04.766271 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:04.766246 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-159.ec2.internal\" not found" node="ip-10-0-132-159.ec2.internal" Sep 06 12:18:04.770059 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:04.770043 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-159.ec2.internal\" not found" node="ip-10-0-132-159.ec2.internal" Sep 06 12:18:04.804678 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:04.804662 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-159.ec2.internal\" not found" Sep 06 12:18:04.819979 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.819965 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5c688b417bc75666ecb3d091248e9864-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-159.ec2.internal\" (UID: \"5c688b417bc75666ecb3d091248e9864\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-159.ec2.internal" Sep 06 12:18:04.820044 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.819985 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c688b417bc75666ecb3d091248e9864-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-159.ec2.internal\" (UID: \"5c688b417bc75666ecb3d091248e9864\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-159.ec2.internal" Sep 06 12:18:04.820044 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.820001 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/45bd7c4cff2b3f51ea19564c2fce8332-config\") pod \"kube-apiserver-proxy-ip-10-0-132-159.ec2.internal\" (UID: \"45bd7c4cff2b3f51ea19564c2fce8332\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-159.ec2.internal" Sep 06 12:18:04.905070 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:04.905045 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-159.ec2.internal\" not found" Sep 06 12:18:04.920370 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.920354 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5c688b417bc75666ecb3d091248e9864-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-159.ec2.internal\" (UID: \"5c688b417bc75666ecb3d091248e9864\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-159.ec2.internal" Sep 06 12:18:04.920419 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.920376 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c688b417bc75666ecb3d091248e9864-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-159.ec2.internal\" (UID: \"5c688b417bc75666ecb3d091248e9864\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-159.ec2.internal" Sep 06 12:18:04.920419 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.920392 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/45bd7c4cff2b3f51ea19564c2fce8332-config\") pod \"kube-apiserver-proxy-ip-10-0-132-159.ec2.internal\" (UID: \"45bd7c4cff2b3f51ea19564c2fce8332\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-159.ec2.internal" Sep 06 12:18:04.920494 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.920450 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/45bd7c4cff2b3f51ea19564c2fce8332-config\") pod \"kube-apiserver-proxy-ip-10-0-132-159.ec2.internal\" (UID: \"45bd7c4cff2b3f51ea19564c2fce8332\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-159.ec2.internal" Sep 06 12:18:04.920494 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.920460 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5c688b417bc75666ecb3d091248e9864-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-159.ec2.internal\" (UID: \"5c688b417bc75666ecb3d091248e9864\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-159.ec2.internal" Sep 06 12:18:04.920494 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:04.920482 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c688b417bc75666ecb3d091248e9864-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-159.ec2.internal\" (UID: \"5c688b417bc75666ecb3d091248e9864\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-159.ec2.internal" Sep 06 12:18:05.005841 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:05.005788 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-159.ec2.internal\" not found" Sep 06 12:18:05.069280 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:05.069264 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-159.ec2.internal" Sep 06 12:18:05.072954 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:05.072937 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-159.ec2.internal" Sep 06 12:18:05.106652 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:05.106529 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-159.ec2.internal\" not found" Sep 06 12:18:05.207113 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:05.207087 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-159.ec2.internal\" not found" Sep 06 12:18:05.307622 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:05.307559 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-159.ec2.internal\" not found" Sep 06 12:18:05.408141 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:05.408117 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-159.ec2.internal\" not found" Sep 06 12:18:05.459595 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:05.459574 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 06 12:18:05.459971 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:05.459704 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 06 12:18:05.508939 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:05.508914 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-159.ec2.internal\" not found" Sep 06 12:18:05.531051 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:05.531024 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-05 12:13:04 +0000 UTC" deadline="2028-04-17 21:57:57.630753007 +0000 UTC" Sep 06 12:18:05.531142 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:05.531054 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14145h39m52.099706092s" Sep 06 12:18:05.609307 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:05.609288 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-159.ec2.internal\" not found" Sep 06 12:18:05.610063 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:05.610038 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c688b417bc75666ecb3d091248e9864.slice/crio-83a9e2aff2db9364e6b74a41dd743f5c90330e847a6a87574d3129643e36eb8e WatchSource:0}: Error finding container 83a9e2aff2db9364e6b74a41dd743f5c90330e847a6a87574d3129643e36eb8e: Status 404 returned error can't find the container with id 83a9e2aff2db9364e6b74a41dd743f5c90330e847a6a87574d3129643e36eb8e Sep 06 12:18:05.610298 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:05.610282 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45bd7c4cff2b3f51ea19564c2fce8332.slice/crio-f84a4215991ba0fe421c22ad4e8c74a3bf6cf601ec2a42454f73fa37178aef7b WatchSource:0}: Error finding container f84a4215991ba0fe421c22ad4e8c74a3bf6cf601ec2a42454f73fa37178aef7b: Status 404 returned error can't find the container with id f84a4215991ba0fe421c22ad4e8c74a3bf6cf601ec2a42454f73fa37178aef7b Sep 06 12:18:05.614371 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:05.614352 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 06 12:18:05.648915 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:05.648881 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-159.ec2.internal" event={"ID":"5c688b417bc75666ecb3d091248e9864","Type":"ContainerStarted","Data":"83a9e2aff2db9364e6b74a41dd743f5c90330e847a6a87574d3129643e36eb8e"} Sep 06 12:18:05.650099 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:05.650079 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-159.ec2.internal" event={"ID":"45bd7c4cff2b3f51ea19564c2fce8332","Type":"ContainerStarted","Data":"f84a4215991ba0fe421c22ad4e8c74a3bf6cf601ec2a42454f73fa37178aef7b"} Sep 06 12:18:05.710249 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:05.710230 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-159.ec2.internal\" not found" Sep 06 12:18:05.810701 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:05.810678 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-159.ec2.internal\" not found" Sep 06 12:18:05.876143 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:05.876092 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 06 12:18:05.911648 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:05.911627 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-159.ec2.internal\" not found" Sep 06 12:18:05.952239 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:05.952216 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 06 12:18:06.012273 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:06.012250 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-159.ec2.internal\" not found" Sep 06 12:18:06.073896 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.073874 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 06 12:18:06.117656 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.117636 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-159.ec2.internal" Sep 06 12:18:06.124550 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.124531 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 06 12:18:06.125720 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.125690 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-159.ec2.internal" Sep 06 12:18:06.134120 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.134074 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 06 12:18:06.379257 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.379231 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 06 12:18:06.499023 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.499001 2576 apiserver.go:52] "Watching apiserver" Sep 06 12:18:06.506813 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.506794 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 06 12:18:06.507146 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.507126 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-gqpgt","openshift-multus/network-metrics-daemon-wlhb5","openshift-network-diagnostics/network-check-target-gpxpp","openshift-network-operator/iptables-alerter-wzmsv","openshift-ovn-kubernetes/ovnkube-node-98vz4","kube-system/global-pull-secret-syncer-m6lx6","kube-system/kube-apiserver-proxy-ip-10-0-132-159.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6n7","openshift-cluster-node-tuning-operator/tuned-td2jx","openshift-dns/node-resolver-v6tf7","openshift-image-registry/node-ca-lt7cv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-159.ec2.internal","openshift-multus/multus-57pbd","kube-system/konnectivity-agent-l2nhh"] Sep 06 12:18:06.509488 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.509464 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v6tf7" Sep 06 12:18:06.511448 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.511393 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.511556 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.511540 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 06 12:18:06.511611 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.511581 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 06 12:18:06.511662 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.511609 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r6nr6\"" Sep 06 12:18:06.513394 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.513368 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 06 12:18:06.513473 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.513439 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxpp" Sep 06 12:18:06.513527 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:06.513502 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxpp" podUID="24ab4b74-f789-4642-adb7-a66bb6b75bb2" Sep 06 12:18:06.513781 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.513767 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 06 12:18:06.514271 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.514255 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 06 12:18:06.514271 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.514268 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 06 12:18:06.514271 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.514289 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xdm6x\"" Sep 06 12:18:06.515301 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.515290 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wzmsv" Sep 06 12:18:06.517057 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.517037 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 06 12:18:06.517149 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.517098 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 06 12:18:06.517414 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.517399 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.518020 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.517822 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-m46lq\"" Sep 06 12:18:06.518020 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.517822 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 06 12:18:06.518837 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.518820 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 06 12:18:06.519496 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.519477 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:06.519574 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:06.519540 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6lx6" podUID="e095072c-bd72-49b4-8cba-b49c94e40bf9" Sep 06 12:18:06.519628 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.519604 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 06 12:18:06.519863 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.519845 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 06 12:18:06.519935 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.519864 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 06 12:18:06.519935 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.519885 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 06 12:18:06.520149 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.520134 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 06 12:18:06.520201 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.520176 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 06 12:18:06.520249 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.520232 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hb8r8\"" Sep 06 12:18:06.522047 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.521801 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.523796 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.523778 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2gx9q\"" Sep 06 12:18:06.523888 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.523813 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 06 12:18:06.524208 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.524190 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l2nhh" Sep 06 12:18:06.524350 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.524337 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 06 12:18:06.526185 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.526168 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 06 12:18:06.526363 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.526352 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 06 12:18:06.526557 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.526541 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rt5dn\"" Sep 06 12:18:06.527012 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.526996 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlhb5" Sep 06 12:18:06.527092 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:06.527053 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlhb5" podUID="e8d626d1-3961-441f-829f-2c8a637e63f7" Sep 06 12:18:06.529121 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.529105 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gqpgt" Sep 06 12:18:06.529418 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.529399 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8b677696-8a5d-4304-bf45-2603cfc71903-multus-daemon-config\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.529482 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.529426 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrxzd\" (UniqueName: \"kubernetes.io/projected/8b677696-8a5d-4304-bf45-2603cfc71903-kube-api-access-qrxzd\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.529482 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.529447 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p48n7\" (UniqueName: \"kubernetes.io/projected/24ab4b74-f789-4642-adb7-a66bb6b75bb2-kube-api-access-p48n7\") pod \"network-check-target-gpxpp\" (UID: \"24ab4b74-f789-4642-adb7-a66bb6b75bb2\") " pod="openshift-network-diagnostics/network-check-target-gpxpp" Sep 06 12:18:06.529482 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.529464 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-systemd-units\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.529583 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.529482 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e095072c-bd72-49b4-8cba-b49c94e40bf9-dbus\") pod \"global-pull-secret-syncer-m6lx6\" (UID: \"e095072c-bd72-49b4-8cba-b49c94e40bf9\") " pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:06.529583 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.529509 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm8p8\" (UniqueName: \"kubernetes.io/projected/1c8ae8aa-90e6-4a07-bd2c-8d9f0ab53f78-kube-api-access-tm8p8\") pod \"node-resolver-v6tf7\" (UID: \"1c8ae8aa-90e6-4a07-bd2c-8d9f0ab53f78\") " pod="openshift-dns/node-resolver-v6tf7" Sep 06 12:18:06.529583 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.529537 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-run-systemd\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.529583 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.529559 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-host-cni-bin\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.529583 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.529577 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4103c09f-dd43-419d-92e5-2374af5b5e01-ovnkube-config\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.529824 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.529592 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e095072c-bd72-49b4-8cba-b49c94e40bf9-original-pull-secret\") pod \"global-pull-secret-syncer-m6lx6\" (UID: \"e095072c-bd72-49b4-8cba-b49c94e40bf9\") " pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:06.529824 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.529609 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c8ae8aa-90e6-4a07-bd2c-8d9f0ab53f78-tmp-dir\") pod \"node-resolver-v6tf7\" (UID: \"1c8ae8aa-90e6-4a07-bd2c-8d9f0ab53f78\") " pod="openshift-dns/node-resolver-v6tf7" Sep 06 12:18:06.529824 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.529625 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-host-run-netns\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.529824 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.529642 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-host-var-lib-cni-bin\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.529824 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.529662 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e095072c-bd72-49b4-8cba-b49c94e40bf9-kubelet-config\") pod \"global-pull-secret-syncer-m6lx6\" (UID: \"e095072c-bd72-49b4-8cba-b49c94e40bf9\") " pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:06.529824 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.529690 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-host-kubelet\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.529824 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.529705 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-run-openvswitch\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.529824 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.529718 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-run-ovn\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.529824 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.529736 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-host-cni-netd\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.529824 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.529786 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlcpx\" (UniqueName: \"kubernetes.io/projected/4103c09f-dd43-419d-92e5-2374af5b5e01-kube-api-access-mlcpx\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.529824 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.529820 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-multus-cni-dir\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.530286 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.529842 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b677696-8a5d-4304-bf45-2603cfc71903-cni-binary-copy\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.530286 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.529868 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.530286 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.529890 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4103c09f-dd43-419d-92e5-2374af5b5e01-ovn-node-metrics-cert\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.530286 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.529927 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-host-var-lib-cni-multus\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.530286 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.529965 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-hostroot\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.530286 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.529986 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-multus-conf-dir\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.530286 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.530020 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-host-run-multus-certs\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.530286 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.530036 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-host-run-netns\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.530286 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.530052 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-etc-openvswitch\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.530286 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.530073 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-node-log\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.530286 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.530088 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-host-run-k8s-cni-cncf-io\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.530286 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.530102 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-etc-kubernetes\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.530286 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.530136 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-var-lib-openvswitch\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.530286 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.530169 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4103c09f-dd43-419d-92e5-2374af5b5e01-ovnkube-script-lib\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.530286 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.530196 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-host-var-lib-kubelet\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.530286 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.530221 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5970f568-4799-4f5a-9ee3-79790a2a3c7e-iptables-alerter-script\") pod \"iptables-alerter-wzmsv\" (UID: \"5970f568-4799-4f5a-9ee3-79790a2a3c7e\") " pod="openshift-network-operator/iptables-alerter-wzmsv" Sep 06 12:18:06.530814 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.530245 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drqmt\" (UniqueName: \"kubernetes.io/projected/5970f568-4799-4f5a-9ee3-79790a2a3c7e-kube-api-access-drqmt\") pod \"iptables-alerter-wzmsv\" (UID: \"5970f568-4799-4f5a-9ee3-79790a2a3c7e\") " pod="openshift-network-operator/iptables-alerter-wzmsv" Sep 06 12:18:06.530814 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.530269 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-host-slash\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.530814 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.530291 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-log-socket\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.530814 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.530314 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-host-run-ovn-kubernetes\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.530814 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.530338 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4103c09f-dd43-419d-92e5-2374af5b5e01-env-overrides\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.530814 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.530360 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-cnibin\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.530814 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.530381 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5970f568-4799-4f5a-9ee3-79790a2a3c7e-host-slash\") pod \"iptables-alerter-wzmsv\" (UID: \"5970f568-4799-4f5a-9ee3-79790a2a3c7e\") " pod="openshift-network-operator/iptables-alerter-wzmsv" Sep 06 12:18:06.530814 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.530394 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1c8ae8aa-90e6-4a07-bd2c-8d9f0ab53f78-hosts-file\") pod \"node-resolver-v6tf7\" (UID: \"1c8ae8aa-90e6-4a07-bd2c-8d9f0ab53f78\") " pod="openshift-dns/node-resolver-v6tf7" Sep 06 12:18:06.530814 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.530415 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-system-cni-dir\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.530814 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.530427 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-multus-socket-dir-parent\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.531295 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.531250 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lt7cv" Sep 06 12:18:06.531513 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.531496 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 06 12:18:06.531733 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.531717 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 06 12:18:06.532471 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.532449 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 06 12:18:06.532731 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.532717 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-twztj\"" Sep 06 12:18:06.535126 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.535108 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 06 12:18:06.535334 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.535322 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 06 12:18:06.535570 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.535554 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-n4sc5\"" Sep 06 12:18:06.535904 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.535891 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 06 12:18:06.539010 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.538993 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6n7" Sep 06 12:18:06.540875 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.540850 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 06 12:18:06.540958 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.540925 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 06 12:18:06.540958 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.540940 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 06 12:18:06.541059 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.540927 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zmqzk\"" Sep 06 12:18:06.550303 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.550289 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rr2lw" Sep 06 12:18:06.554982 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.554948 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rr2lw" Sep 06 12:18:06.619118 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.619096 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 06 12:18:06.630549 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.630530 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-lib-modules\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.630624 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.630552 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/39d362d3-b7d6-48b0-9ac4-05f258ee82d4-registration-dir\") pod \"aws-ebs-csi-driver-node-dh6n7\" (UID: \"39d362d3-b7d6-48b0-9ac4-05f258ee82d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6n7" Sep 06 12:18:06.630624 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.630581 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-run-openvswitch\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.630624 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.630605 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-run-ovn\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.630624 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.630622 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-host-cni-netd\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.630775 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.630637 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlcpx\" (UniqueName: \"kubernetes.io/projected/4103c09f-dd43-419d-92e5-2374af5b5e01-kube-api-access-mlcpx\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.630775 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.630651 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-multus-cni-dir\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.630775 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.630664 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b677696-8a5d-4304-bf45-2603cfc71903-cni-binary-copy\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.630775 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.630677 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/39d362d3-b7d6-48b0-9ac4-05f258ee82d4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dh6n7\" (UID: \"39d362d3-b7d6-48b0-9ac4-05f258ee82d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6n7" Sep 06 12:18:06.630775 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.630692 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.630775 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.630705 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4103c09f-dd43-419d-92e5-2374af5b5e01-ovn-node-metrics-cert\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.630775 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.630725 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-etc-modprobe-d\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.630775 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.630748 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-etc-sysconfig\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.631037 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.630785 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-sys\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.631037 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.630803 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-host\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.631037 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.630817 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbxlx\" (UniqueName: \"kubernetes.io/projected/470b9439-8d7e-4a84-ae68-b66b4851c5dc-kube-api-access-cbxlx\") pod \"node-ca-lt7cv\" (UID: \"470b9439-8d7e-4a84-ae68-b66b4851c5dc\") " pod="openshift-image-registry/node-ca-lt7cv" Sep 06 12:18:06.631037 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.630837 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-host-run-netns\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.631037 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.630853 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-node-log\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.631037 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.630867 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-host-run-k8s-cni-cncf-io\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.631037 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.630888 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-etc-kubernetes\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.631037 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.630913 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzfc4\" (UniqueName: \"kubernetes.io/projected/39d362d3-b7d6-48b0-9ac4-05f258ee82d4-kube-api-access-zzfc4\") pod \"aws-ebs-csi-driver-node-dh6n7\" (UID: \"39d362d3-b7d6-48b0-9ac4-05f258ee82d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6n7" Sep 06 12:18:06.631037 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.630950 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-host-var-lib-kubelet\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.631037 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.630971 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db464292-ffbf-4c25-85d3-ff4c4be77552-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gqpgt\" (UID: \"db464292-ffbf-4c25-85d3-ff4c4be77552\") " pod="openshift-multus/multus-additional-cni-plugins-gqpgt" Sep 06 12:18:06.631037 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.630985 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-run\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.631037 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.630996 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-etc-tuned\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.631037 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631009 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-log-socket\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.631037 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631022 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-multus-socket-dir-parent\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.631037 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631036 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db464292-ffbf-4c25-85d3-ff4c4be77552-system-cni-dir\") pod \"multus-additional-cni-plugins-gqpgt\" (UID: \"db464292-ffbf-4c25-85d3-ff4c4be77552\") " pod="openshift-multus/multus-additional-cni-plugins-gqpgt" Sep 06 12:18:06.631600 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631056 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fnjb\" (UniqueName: \"kubernetes.io/projected/db464292-ffbf-4c25-85d3-ff4c4be77552-kube-api-access-9fnjb\") pod \"multus-additional-cni-plugins-gqpgt\" (UID: \"db464292-ffbf-4c25-85d3-ff4c4be77552\") " pod="openshift-multus/multus-additional-cni-plugins-gqpgt" Sep 06 12:18:06.631600 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631069 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-etc-sysctl-d\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.631600 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631088 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-var-lib-kubelet\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.631600 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631115 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-system-cni-dir\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.631600 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631143 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8b677696-8a5d-4304-bf45-2603cfc71903-multus-daemon-config\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.631600 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631156 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/470b9439-8d7e-4a84-ae68-b66b4851c5dc-serviceca\") pod \"node-ca-lt7cv\" (UID: \"470b9439-8d7e-4a84-ae68-b66b4851c5dc\") " pod="openshift-image-registry/node-ca-lt7cv" Sep 06 12:18:06.631600 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631173 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-systemd-units\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.631600 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631187 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e095072c-bd72-49b4-8cba-b49c94e40bf9-dbus\") pod \"global-pull-secret-syncer-m6lx6\" (UID: \"e095072c-bd72-49b4-8cba-b49c94e40bf9\") " pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:06.631600 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631201 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/12708d6c-10bb-472c-b4a3-e8ece6ef008f-konnectivity-ca\") pod \"konnectivity-agent-l2nhh\" (UID: \"12708d6c-10bb-472c-b4a3-e8ece6ef008f\") " pod="kube-system/konnectivity-agent-l2nhh" Sep 06 12:18:06.631600 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631225 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-run-systemd\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.631600 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631239 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e095072c-bd72-49b4-8cba-b49c94e40bf9-original-pull-secret\") pod \"global-pull-secret-syncer-m6lx6\" (UID: \"e095072c-bd72-49b4-8cba-b49c94e40bf9\") " pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:06.631600 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631253 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-host-run-netns\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.631600 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631267 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e095072c-bd72-49b4-8cba-b49c94e40bf9-kubelet-config\") pod \"global-pull-secret-syncer-m6lx6\" (UID: \"e095072c-bd72-49b4-8cba-b49c94e40bf9\") " pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:06.631600 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631279 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-host-kubelet\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.631600 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631300 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-host-var-lib-cni-multus\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.631600 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631320 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-hostroot\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.631600 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631333 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-multus-conf-dir\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.632158 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631351 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db464292-ffbf-4c25-85d3-ff4c4be77552-cni-binary-copy\") pod \"multus-additional-cni-plugins-gqpgt\" (UID: \"db464292-ffbf-4c25-85d3-ff4c4be77552\") " pod="openshift-multus/multus-additional-cni-plugins-gqpgt" Sep 06 12:18:06.632158 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631368 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-etc-kubernetes\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.632158 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631387 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74qcn\" (UniqueName: \"kubernetes.io/projected/e8d626d1-3961-441f-829f-2c8a637e63f7-kube-api-access-74qcn\") pod \"network-metrics-daemon-wlhb5\" (UID: \"e8d626d1-3961-441f-829f-2c8a637e63f7\") " pod="openshift-multus/network-metrics-daemon-wlhb5" Sep 06 12:18:06.632158 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631410 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-host-run-multus-certs\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.632158 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631433 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-etc-systemd\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.632158 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631451 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/39d362d3-b7d6-48b0-9ac4-05f258ee82d4-device-dir\") pod \"aws-ebs-csi-driver-node-dh6n7\" (UID: \"39d362d3-b7d6-48b0-9ac4-05f258ee82d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6n7" Sep 06 12:18:06.632158 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631467 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-etc-openvswitch\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.632158 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631489 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/39d362d3-b7d6-48b0-9ac4-05f258ee82d4-socket-dir\") pod \"aws-ebs-csi-driver-node-dh6n7\" (UID: \"39d362d3-b7d6-48b0-9ac4-05f258ee82d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6n7" Sep 06 12:18:06.632158 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631506 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-var-lib-openvswitch\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.632158 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631544 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4103c09f-dd43-419d-92e5-2374af5b5e01-ovnkube-script-lib\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.632158 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631565 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/db464292-ffbf-4c25-85d3-ff4c4be77552-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gqpgt\" (UID: \"db464292-ffbf-4c25-85d3-ff4c4be77552\") " pod="openshift-multus/multus-additional-cni-plugins-gqpgt" Sep 06 12:18:06.632158 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631579 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-etc-sysctl-conf\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.632158 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631601 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-tmp\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.632158 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631623 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvkf8\" (UniqueName: \"kubernetes.io/projected/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-kube-api-access-gvkf8\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.632158 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631637 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8d626d1-3961-441f-829f-2c8a637e63f7-metrics-certs\") pod \"network-metrics-daemon-wlhb5\" (UID: \"e8d626d1-3961-441f-829f-2c8a637e63f7\") " pod="openshift-multus/network-metrics-daemon-wlhb5" Sep 06 12:18:06.632158 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631655 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5970f568-4799-4f5a-9ee3-79790a2a3c7e-iptables-alerter-script\") pod \"iptables-alerter-wzmsv\" (UID: \"5970f568-4799-4f5a-9ee3-79790a2a3c7e\") " pod="openshift-network-operator/iptables-alerter-wzmsv" Sep 06 12:18:06.632695 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631669 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drqmt\" (UniqueName: \"kubernetes.io/projected/5970f568-4799-4f5a-9ee3-79790a2a3c7e-kube-api-access-drqmt\") pod \"iptables-alerter-wzmsv\" (UID: \"5970f568-4799-4f5a-9ee3-79790a2a3c7e\") " pod="openshift-network-operator/iptables-alerter-wzmsv" Sep 06 12:18:06.632695 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631682 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-host-slash\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.632695 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631704 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-host-run-ovn-kubernetes\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.632695 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631722 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4103c09f-dd43-419d-92e5-2374af5b5e01-env-overrides\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.632695 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631764 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-cnibin\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.632695 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631792 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrxzd\" (UniqueName: \"kubernetes.io/projected/8b677696-8a5d-4304-bf45-2603cfc71903-kube-api-access-qrxzd\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.632695 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631807 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5970f568-4799-4f5a-9ee3-79790a2a3c7e-host-slash\") pod \"iptables-alerter-wzmsv\" (UID: \"5970f568-4799-4f5a-9ee3-79790a2a3c7e\") " pod="openshift-network-operator/iptables-alerter-wzmsv" Sep 06 12:18:06.632695 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631826 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1c8ae8aa-90e6-4a07-bd2c-8d9f0ab53f78-hosts-file\") pod \"node-resolver-v6tf7\" (UID: \"1c8ae8aa-90e6-4a07-bd2c-8d9f0ab53f78\") " pod="openshift-dns/node-resolver-v6tf7" Sep 06 12:18:06.632695 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631848 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/39d362d3-b7d6-48b0-9ac4-05f258ee82d4-etc-selinux\") pod \"aws-ebs-csi-driver-node-dh6n7\" (UID: \"39d362d3-b7d6-48b0-9ac4-05f258ee82d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6n7" Sep 06 12:18:06.632695 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631878 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p48n7\" (UniqueName: \"kubernetes.io/projected/24ab4b74-f789-4642-adb7-a66bb6b75bb2-kube-api-access-p48n7\") pod \"network-check-target-gpxpp\" (UID: \"24ab4b74-f789-4642-adb7-a66bb6b75bb2\") " pod="openshift-network-diagnostics/network-check-target-gpxpp" Sep 06 12:18:06.632695 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631901 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm8p8\" (UniqueName: \"kubernetes.io/projected/1c8ae8aa-90e6-4a07-bd2c-8d9f0ab53f78-kube-api-access-tm8p8\") pod \"node-resolver-v6tf7\" (UID: \"1c8ae8aa-90e6-4a07-bd2c-8d9f0ab53f78\") " pod="openshift-dns/node-resolver-v6tf7" Sep 06 12:18:06.632695 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631921 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/12708d6c-10bb-472c-b4a3-e8ece6ef008f-agent-certs\") pod \"konnectivity-agent-l2nhh\" (UID: \"12708d6c-10bb-472c-b4a3-e8ece6ef008f\") " pod="kube-system/konnectivity-agent-l2nhh" Sep 06 12:18:06.632695 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631935 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db464292-ffbf-4c25-85d3-ff4c4be77552-cnibin\") pod \"multus-additional-cni-plugins-gqpgt\" (UID: \"db464292-ffbf-4c25-85d3-ff4c4be77552\") " pod="openshift-multus/multus-additional-cni-plugins-gqpgt" Sep 06 12:18:06.632695 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631948 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/db464292-ffbf-4c25-85d3-ff4c4be77552-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gqpgt\" (UID: \"db464292-ffbf-4c25-85d3-ff4c4be77552\") " pod="openshift-multus/multus-additional-cni-plugins-gqpgt" Sep 06 12:18:06.632695 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631962 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/470b9439-8d7e-4a84-ae68-b66b4851c5dc-host\") pod \"node-ca-lt7cv\" (UID: \"470b9439-8d7e-4a84-ae68-b66b4851c5dc\") " pod="openshift-image-registry/node-ca-lt7cv" Sep 06 12:18:06.632695 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631976 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/39d362d3-b7d6-48b0-9ac4-05f258ee82d4-sys-fs\") pod \"aws-ebs-csi-driver-node-dh6n7\" (UID: \"39d362d3-b7d6-48b0-9ac4-05f258ee82d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6n7" Sep 06 12:18:06.632695 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.631993 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-host-cni-bin\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.633273 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.632018 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4103c09f-dd43-419d-92e5-2374af5b5e01-ovnkube-config\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.633273 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.632042 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c8ae8aa-90e6-4a07-bd2c-8d9f0ab53f78-tmp-dir\") pod \"node-resolver-v6tf7\" (UID: \"1c8ae8aa-90e6-4a07-bd2c-8d9f0ab53f78\") " pod="openshift-dns/node-resolver-v6tf7" Sep 06 12:18:06.633273 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.632058 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-host-var-lib-cni-bin\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.633273 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.632111 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-host-var-lib-cni-bin\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.633273 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.632146 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-run-openvswitch\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.633273 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.632170 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-run-ovn\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.633273 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.632197 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-host-cni-netd\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.633273 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.632525 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-multus-cni-dir\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.633273 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.632990 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b677696-8a5d-4304-bf45-2603cfc71903-cni-binary-copy\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.633273 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.633078 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.633622 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.633310 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 06 12:18:06.633670 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.633622 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-system-cni-dir\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.633721 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.633684 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-multus-socket-dir-parent\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.633784 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.633742 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-host-slash\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.633888 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.633819 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-host-run-multus-certs\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.633888 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.633862 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-etc-openvswitch\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.633998 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.633906 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-systemd-units\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.633998 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.633920 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-run-systemd\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.633998 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.633946 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e095072c-bd72-49b4-8cba-b49c94e40bf9-dbus\") pod \"global-pull-secret-syncer-m6lx6\" (UID: \"e095072c-bd72-49b4-8cba-b49c94e40bf9\") " pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:06.633998 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:06.633995 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 06 12:18:06.634182 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:06.634042 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e095072c-bd72-49b4-8cba-b49c94e40bf9-original-pull-secret podName:e095072c-bd72-49b4-8cba-b49c94e40bf9 nodeName:}" failed. No retries permitted until 2026-09-06 12:18:07.134028971 +0000 UTC m=+2.989786834 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e095072c-bd72-49b4-8cba-b49c94e40bf9-original-pull-secret") pod "global-pull-secret-syncer-m6lx6" (UID: "e095072c-bd72-49b4-8cba-b49c94e40bf9") : object "kube-system"/"original-pull-secret" not registered Sep 06 12:18:06.634182 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.634101 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8b677696-8a5d-4304-bf45-2603cfc71903-multus-daemon-config\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.634182 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.634155 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-host-kubelet\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.634182 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.634160 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-host-run-netns\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.634182 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.634181 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-host-var-lib-cni-multus\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.634414 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.634204 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-hostroot\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.634414 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.634204 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e095072c-bd72-49b4-8cba-b49c94e40bf9-kubelet-config\") pod \"global-pull-secret-syncer-m6lx6\" (UID: \"e095072c-bd72-49b4-8cba-b49c94e40bf9\") " pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:06.634414 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.634233 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-multus-conf-dir\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.634414 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.634245 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-etc-kubernetes\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.634414 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.634251 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-host-run-k8s-cni-cncf-io\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.634414 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.634281 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-var-lib-openvswitch\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.634414 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.634284 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-host-var-lib-kubelet\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.634414 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.634307 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-host-run-netns\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.634414 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.634332 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-log-socket\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.634414 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.634354 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-node-log\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.634414 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.634390 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-host-run-ovn-kubernetes\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.634971 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.634524 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b677696-8a5d-4304-bf45-2603cfc71903-cnibin\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.634971 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.634566 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4103c09f-dd43-419d-92e5-2374af5b5e01-ovnkube-script-lib\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.634971 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.634575 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5970f568-4799-4f5a-9ee3-79790a2a3c7e-host-slash\") pod \"iptables-alerter-wzmsv\" (UID: \"5970f568-4799-4f5a-9ee3-79790a2a3c7e\") " pod="openshift-network-operator/iptables-alerter-wzmsv" Sep 06 12:18:06.634971 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.634619 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1c8ae8aa-90e6-4a07-bd2c-8d9f0ab53f78-hosts-file\") pod \"node-resolver-v6tf7\" (UID: \"1c8ae8aa-90e6-4a07-bd2c-8d9f0ab53f78\") " pod="openshift-dns/node-resolver-v6tf7" Sep 06 12:18:06.634971 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.634625 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4103c09f-dd43-419d-92e5-2374af5b5e01-host-cni-bin\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.634971 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.634804 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4103c09f-dd43-419d-92e5-2374af5b5e01-env-overrides\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.634971 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.634967 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c8ae8aa-90e6-4a07-bd2c-8d9f0ab53f78-tmp-dir\") pod \"node-resolver-v6tf7\" (UID: \"1c8ae8aa-90e6-4a07-bd2c-8d9f0ab53f78\") " pod="openshift-dns/node-resolver-v6tf7" Sep 06 12:18:06.635217 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.635121 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5970f568-4799-4f5a-9ee3-79790a2a3c7e-iptables-alerter-script\") pod \"iptables-alerter-wzmsv\" (UID: \"5970f568-4799-4f5a-9ee3-79790a2a3c7e\") " pod="openshift-network-operator/iptables-alerter-wzmsv" Sep 06 12:18:06.635217 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.635172 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4103c09f-dd43-419d-92e5-2374af5b5e01-ovnkube-config\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.636955 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.636931 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4103c09f-dd43-419d-92e5-2374af5b5e01-ovn-node-metrics-cert\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.640000 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:06.639826 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 06 12:18:06.640000 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:06.639849 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 06 12:18:06.640000 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:06.639861 2576 projected.go:196] Error preparing data for projected volume kube-api-access-p48n7 for pod openshift-network-diagnostics/network-check-target-gpxpp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 12:18:06.640000 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:06.639977 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24ab4b74-f789-4642-adb7-a66bb6b75bb2-kube-api-access-p48n7 podName:24ab4b74-f789-4642-adb7-a66bb6b75bb2 nodeName:}" failed. No retries permitted until 2026-09-06 12:18:07.139949818 +0000 UTC m=+2.995707702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-p48n7" (UniqueName: "kubernetes.io/projected/24ab4b74-f789-4642-adb7-a66bb6b75bb2-kube-api-access-p48n7") pod "network-check-target-gpxpp" (UID: "24ab4b74-f789-4642-adb7-a66bb6b75bb2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 12:18:06.640268 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.640249 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlcpx\" (UniqueName: \"kubernetes.io/projected/4103c09f-dd43-419d-92e5-2374af5b5e01-kube-api-access-mlcpx\") pod \"ovnkube-node-98vz4\" (UID: \"4103c09f-dd43-419d-92e5-2374af5b5e01\") " pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.642078 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.642057 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrxzd\" (UniqueName: \"kubernetes.io/projected/8b677696-8a5d-4304-bf45-2603cfc71903-kube-api-access-qrxzd\") pod \"multus-57pbd\" (UID: \"8b677696-8a5d-4304-bf45-2603cfc71903\") " pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.642493 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.642471 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm8p8\" (UniqueName: \"kubernetes.io/projected/1c8ae8aa-90e6-4a07-bd2c-8d9f0ab53f78-kube-api-access-tm8p8\") pod \"node-resolver-v6tf7\" (UID: \"1c8ae8aa-90e6-4a07-bd2c-8d9f0ab53f78\") " pod="openshift-dns/node-resolver-v6tf7" Sep 06 12:18:06.642807 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.642774 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drqmt\" (UniqueName: \"kubernetes.io/projected/5970f568-4799-4f5a-9ee3-79790a2a3c7e-kube-api-access-drqmt\") pod \"iptables-alerter-wzmsv\" (UID: \"5970f568-4799-4f5a-9ee3-79790a2a3c7e\") " pod="openshift-network-operator/iptables-alerter-wzmsv" Sep 06 12:18:06.733045 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733014 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db464292-ffbf-4c25-85d3-ff4c4be77552-cni-binary-copy\") pod \"multus-additional-cni-plugins-gqpgt\" (UID: \"db464292-ffbf-4c25-85d3-ff4c4be77552\") " pod="openshift-multus/multus-additional-cni-plugins-gqpgt" Sep 06 12:18:06.733183 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733052 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-etc-kubernetes\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.733183 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733078 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74qcn\" (UniqueName: \"kubernetes.io/projected/e8d626d1-3961-441f-829f-2c8a637e63f7-kube-api-access-74qcn\") pod \"network-metrics-daemon-wlhb5\" (UID: \"e8d626d1-3961-441f-829f-2c8a637e63f7\") " pod="openshift-multus/network-metrics-daemon-wlhb5" Sep 06 12:18:06.733183 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733139 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-etc-kubernetes\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.733334 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733183 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-etc-systemd\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.733334 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733206 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/39d362d3-b7d6-48b0-9ac4-05f258ee82d4-device-dir\") pod \"aws-ebs-csi-driver-node-dh6n7\" (UID: \"39d362d3-b7d6-48b0-9ac4-05f258ee82d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6n7" Sep 06 12:18:06.733334 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733229 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/39d362d3-b7d6-48b0-9ac4-05f258ee82d4-socket-dir\") pod \"aws-ebs-csi-driver-node-dh6n7\" (UID: \"39d362d3-b7d6-48b0-9ac4-05f258ee82d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6n7" Sep 06 12:18:06.733334 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733254 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/db464292-ffbf-4c25-85d3-ff4c4be77552-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gqpgt\" (UID: \"db464292-ffbf-4c25-85d3-ff4c4be77552\") " pod="openshift-multus/multus-additional-cni-plugins-gqpgt" Sep 06 12:18:06.733334 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733273 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-etc-sysctl-conf\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.733334 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733282 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-etc-systemd\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.733334 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733293 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-tmp\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.733334 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733282 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/39d362d3-b7d6-48b0-9ac4-05f258ee82d4-device-dir\") pod \"aws-ebs-csi-driver-node-dh6n7\" (UID: \"39d362d3-b7d6-48b0-9ac4-05f258ee82d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6n7" Sep 06 12:18:06.733334 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733324 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvkf8\" (UniqueName: \"kubernetes.io/projected/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-kube-api-access-gvkf8\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.733788 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733348 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8d626d1-3961-441f-829f-2c8a637e63f7-metrics-certs\") pod \"network-metrics-daemon-wlhb5\" (UID: \"e8d626d1-3961-441f-829f-2c8a637e63f7\") " pod="openshift-multus/network-metrics-daemon-wlhb5" Sep 06 12:18:06.733788 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733378 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/39d362d3-b7d6-48b0-9ac4-05f258ee82d4-etc-selinux\") pod \"aws-ebs-csi-driver-node-dh6n7\" (UID: \"39d362d3-b7d6-48b0-9ac4-05f258ee82d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6n7" Sep 06 12:18:06.733788 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733410 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/12708d6c-10bb-472c-b4a3-e8ece6ef008f-agent-certs\") pod \"konnectivity-agent-l2nhh\" (UID: \"12708d6c-10bb-472c-b4a3-e8ece6ef008f\") " pod="kube-system/konnectivity-agent-l2nhh" Sep 06 12:18:06.733788 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733432 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db464292-ffbf-4c25-85d3-ff4c4be77552-cnibin\") pod \"multus-additional-cni-plugins-gqpgt\" (UID: \"db464292-ffbf-4c25-85d3-ff4c4be77552\") " pod="openshift-multus/multus-additional-cni-plugins-gqpgt" Sep 06 12:18:06.733788 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733452 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/db464292-ffbf-4c25-85d3-ff4c4be77552-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gqpgt\" (UID: \"db464292-ffbf-4c25-85d3-ff4c4be77552\") " pod="openshift-multus/multus-additional-cni-plugins-gqpgt" Sep 06 12:18:06.733788 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733472 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/470b9439-8d7e-4a84-ae68-b66b4851c5dc-host\") pod \"node-ca-lt7cv\" (UID: \"470b9439-8d7e-4a84-ae68-b66b4851c5dc\") " pod="openshift-image-registry/node-ca-lt7cv" Sep 06 12:18:06.733788 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733494 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/39d362d3-b7d6-48b0-9ac4-05f258ee82d4-sys-fs\") pod \"aws-ebs-csi-driver-node-dh6n7\" (UID: \"39d362d3-b7d6-48b0-9ac4-05f258ee82d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6n7" Sep 06 12:18:06.733788 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733517 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-lib-modules\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.733788 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733538 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/39d362d3-b7d6-48b0-9ac4-05f258ee82d4-registration-dir\") pod \"aws-ebs-csi-driver-node-dh6n7\" (UID: \"39d362d3-b7d6-48b0-9ac4-05f258ee82d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6n7" Sep 06 12:18:06.733788 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733568 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/39d362d3-b7d6-48b0-9ac4-05f258ee82d4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dh6n7\" (UID: \"39d362d3-b7d6-48b0-9ac4-05f258ee82d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6n7" Sep 06 12:18:06.733788 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733591 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-etc-modprobe-d\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.733788 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733597 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db464292-ffbf-4c25-85d3-ff4c4be77552-cni-binary-copy\") pod \"multus-additional-cni-plugins-gqpgt\" (UID: \"db464292-ffbf-4c25-85d3-ff4c4be77552\") " pod="openshift-multus/multus-additional-cni-plugins-gqpgt" Sep 06 12:18:06.733788 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733612 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-etc-sysconfig\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.733788 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733633 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-sys\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.733788 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733654 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-host\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.733788 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733675 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbxlx\" (UniqueName: \"kubernetes.io/projected/470b9439-8d7e-4a84-ae68-b66b4851c5dc-kube-api-access-cbxlx\") pod \"node-ca-lt7cv\" (UID: \"470b9439-8d7e-4a84-ae68-b66b4851c5dc\") " pod="openshift-image-registry/node-ca-lt7cv" Sep 06 12:18:06.733788 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733701 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzfc4\" (UniqueName: \"kubernetes.io/projected/39d362d3-b7d6-48b0-9ac4-05f258ee82d4-kube-api-access-zzfc4\") pod \"aws-ebs-csi-driver-node-dh6n7\" (UID: \"39d362d3-b7d6-48b0-9ac4-05f258ee82d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6n7" Sep 06 12:18:06.734521 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733728 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/db464292-ffbf-4c25-85d3-ff4c4be77552-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gqpgt\" (UID: \"db464292-ffbf-4c25-85d3-ff4c4be77552\") " pod="openshift-multus/multus-additional-cni-plugins-gqpgt" Sep 06 12:18:06.734521 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733737 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db464292-ffbf-4c25-85d3-ff4c4be77552-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gqpgt\" (UID: \"db464292-ffbf-4c25-85d3-ff4c4be77552\") " pod="openshift-multus/multus-additional-cni-plugins-gqpgt" Sep 06 12:18:06.734521 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733777 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-run\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.734521 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733784 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-etc-sysctl-conf\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.734521 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733800 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-etc-tuned\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.734521 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733825 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db464292-ffbf-4c25-85d3-ff4c4be77552-system-cni-dir\") pod \"multus-additional-cni-plugins-gqpgt\" (UID: \"db464292-ffbf-4c25-85d3-ff4c4be77552\") " pod="openshift-multus/multus-additional-cni-plugins-gqpgt" Sep 06 12:18:06.734521 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733842 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/39d362d3-b7d6-48b0-9ac4-05f258ee82d4-socket-dir\") pod \"aws-ebs-csi-driver-node-dh6n7\" (UID: \"39d362d3-b7d6-48b0-9ac4-05f258ee82d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6n7" Sep 06 12:18:06.734521 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733847 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fnjb\" (UniqueName: \"kubernetes.io/projected/db464292-ffbf-4c25-85d3-ff4c4be77552-kube-api-access-9fnjb\") pod \"multus-additional-cni-plugins-gqpgt\" (UID: \"db464292-ffbf-4c25-85d3-ff4c4be77552\") " pod="openshift-multus/multus-additional-cni-plugins-gqpgt" Sep 06 12:18:06.734521 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733842 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/39d362d3-b7d6-48b0-9ac4-05f258ee82d4-registration-dir\") pod \"aws-ebs-csi-driver-node-dh6n7\" (UID: \"39d362d3-b7d6-48b0-9ac4-05f258ee82d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6n7" Sep 06 12:18:06.734521 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733868 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-etc-sysctl-d\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.734521 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733889 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-var-lib-kubelet\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.734521 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733912 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/470b9439-8d7e-4a84-ae68-b66b4851c5dc-serviceca\") pod \"node-ca-lt7cv\" (UID: \"470b9439-8d7e-4a84-ae68-b66b4851c5dc\") " pod="openshift-image-registry/node-ca-lt7cv" Sep 06 12:18:06.734521 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:06.733937 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 12:18:06.734521 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.733933 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/12708d6c-10bb-472c-b4a3-e8ece6ef008f-konnectivity-ca\") pod \"konnectivity-agent-l2nhh\" (UID: \"12708d6c-10bb-472c-b4a3-e8ece6ef008f\") " pod="kube-system/konnectivity-agent-l2nhh" Sep 06 12:18:06.734521 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:06.733979 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8d626d1-3961-441f-829f-2c8a637e63f7-metrics-certs podName:e8d626d1-3961-441f-829f-2c8a637e63f7 nodeName:}" failed. No retries permitted until 2026-09-06 12:18:07.233965385 +0000 UTC m=+3.089723265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8d626d1-3961-441f-829f-2c8a637e63f7-metrics-certs") pod "network-metrics-daemon-wlhb5" (UID: "e8d626d1-3961-441f-829f-2c8a637e63f7") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 12:18:06.734521 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.734192 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/39d362d3-b7d6-48b0-9ac4-05f258ee82d4-etc-selinux\") pod \"aws-ebs-csi-driver-node-dh6n7\" (UID: \"39d362d3-b7d6-48b0-9ac4-05f258ee82d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6n7" Sep 06 12:18:06.734521 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.734421 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/12708d6c-10bb-472c-b4a3-e8ece6ef008f-konnectivity-ca\") pod \"konnectivity-agent-l2nhh\" (UID: \"12708d6c-10bb-472c-b4a3-e8ece6ef008f\") " pod="kube-system/konnectivity-agent-l2nhh" Sep 06 12:18:06.735293 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.734483 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/39d362d3-b7d6-48b0-9ac4-05f258ee82d4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dh6n7\" (UID: \"39d362d3-b7d6-48b0-9ac4-05f258ee82d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6n7" Sep 06 12:18:06.735293 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.734542 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-etc-sysconfig\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.735293 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.734582 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-run\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.735293 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.734602 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-lib-modules\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.735293 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.734646 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/470b9439-8d7e-4a84-ae68-b66b4851c5dc-host\") pod \"node-ca-lt7cv\" (UID: \"470b9439-8d7e-4a84-ae68-b66b4851c5dc\") " pod="openshift-image-registry/node-ca-lt7cv" Sep 06 12:18:06.735293 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.734665 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-etc-modprobe-d\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.735293 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.734696 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db464292-ffbf-4c25-85d3-ff4c4be77552-cnibin\") pod \"multus-additional-cni-plugins-gqpgt\" (UID: \"db464292-ffbf-4c25-85d3-ff4c4be77552\") " pod="openshift-multus/multus-additional-cni-plugins-gqpgt" Sep 06 12:18:06.735293 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.734726 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/39d362d3-b7d6-48b0-9ac4-05f258ee82d4-sys-fs\") pod \"aws-ebs-csi-driver-node-dh6n7\" (UID: \"39d362d3-b7d6-48b0-9ac4-05f258ee82d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6n7" Sep 06 12:18:06.735293 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.734794 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-sys\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.735293 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.734832 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-host\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.735293 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.734845 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db464292-ffbf-4c25-85d3-ff4c4be77552-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gqpgt\" (UID: \"db464292-ffbf-4c25-85d3-ff4c4be77552\") " pod="openshift-multus/multus-additional-cni-plugins-gqpgt" Sep 06 12:18:06.735293 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.735063 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-etc-sysctl-d\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.735293 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.735112 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db464292-ffbf-4c25-85d3-ff4c4be77552-system-cni-dir\") pod \"multus-additional-cni-plugins-gqpgt\" (UID: \"db464292-ffbf-4c25-85d3-ff4c4be77552\") " pod="openshift-multus/multus-additional-cni-plugins-gqpgt" Sep 06 12:18:06.735293 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.734471 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/db464292-ffbf-4c25-85d3-ff4c4be77552-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gqpgt\" (UID: \"db464292-ffbf-4c25-85d3-ff4c4be77552\") " pod="openshift-multus/multus-additional-cni-plugins-gqpgt" Sep 06 12:18:06.735797 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.735345 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-var-lib-kubelet\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.735797 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.735556 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/470b9439-8d7e-4a84-ae68-b66b4851c5dc-serviceca\") pod \"node-ca-lt7cv\" (UID: \"470b9439-8d7e-4a84-ae68-b66b4851c5dc\") " pod="openshift-image-registry/node-ca-lt7cv" Sep 06 12:18:06.735951 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.735932 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-tmp\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.736563 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.736542 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/12708d6c-10bb-472c-b4a3-e8ece6ef008f-agent-certs\") pod \"konnectivity-agent-l2nhh\" (UID: \"12708d6c-10bb-472c-b4a3-e8ece6ef008f\") " pod="kube-system/konnectivity-agent-l2nhh" Sep 06 12:18:06.736646 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.736632 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-etc-tuned\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.739957 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.739934 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74qcn\" (UniqueName: \"kubernetes.io/projected/e8d626d1-3961-441f-829f-2c8a637e63f7-kube-api-access-74qcn\") pod \"network-metrics-daemon-wlhb5\" (UID: \"e8d626d1-3961-441f-829f-2c8a637e63f7\") " pod="openshift-multus/network-metrics-daemon-wlhb5" Sep 06 12:18:06.740768 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.740733 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvkf8\" (UniqueName: \"kubernetes.io/projected/11b4dfc5-375d-44ed-aac5-a4ccc5c41a28-kube-api-access-gvkf8\") pod \"tuned-td2jx\" (UID: \"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28\") " pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.742028 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.741985 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzfc4\" (UniqueName: \"kubernetes.io/projected/39d362d3-b7d6-48b0-9ac4-05f258ee82d4-kube-api-access-zzfc4\") pod \"aws-ebs-csi-driver-node-dh6n7\" (UID: \"39d362d3-b7d6-48b0-9ac4-05f258ee82d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6n7" Sep 06 12:18:06.742517 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.742499 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fnjb\" (UniqueName: \"kubernetes.io/projected/db464292-ffbf-4c25-85d3-ff4c4be77552-kube-api-access-9fnjb\") pod \"multus-additional-cni-plugins-gqpgt\" (UID: \"db464292-ffbf-4c25-85d3-ff4c4be77552\") " pod="openshift-multus/multus-additional-cni-plugins-gqpgt" Sep 06 12:18:06.742787 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.742769 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbxlx\" (UniqueName: \"kubernetes.io/projected/470b9439-8d7e-4a84-ae68-b66b4851c5dc-kube-api-access-cbxlx\") pod \"node-ca-lt7cv\" (UID: \"470b9439-8d7e-4a84-ae68-b66b4851c5dc\") " pod="openshift-image-registry/node-ca-lt7cv" Sep 06 12:18:06.822891 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.822807 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v6tf7" Sep 06 12:18:06.828692 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.828671 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-57pbd" Sep 06 12:18:06.842299 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.842272 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wzmsv" Sep 06 12:18:06.845892 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.845874 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:06.853400 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.853382 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-td2jx" Sep 06 12:18:06.859644 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.859629 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l2nhh" Sep 06 12:18:06.868157 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.868141 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gqpgt" Sep 06 12:18:06.874477 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.874464 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lt7cv" Sep 06 12:18:06.878831 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:06.878812 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6n7" Sep 06 12:18:07.137053 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:07.136981 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e095072c-bd72-49b4-8cba-b49c94e40bf9-original-pull-secret\") pod \"global-pull-secret-syncer-m6lx6\" (UID: \"e095072c-bd72-49b4-8cba-b49c94e40bf9\") " pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:07.137198 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:07.137121 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 06 12:18:07.137198 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:07.137180 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e095072c-bd72-49b4-8cba-b49c94e40bf9-original-pull-secret podName:e095072c-bd72-49b4-8cba-b49c94e40bf9 nodeName:}" failed. No retries permitted until 2026-09-06 12:18:08.137163201 +0000 UTC m=+3.992921072 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e095072c-bd72-49b4-8cba-b49c94e40bf9-original-pull-secret") pod "global-pull-secret-syncer-m6lx6" (UID: "e095072c-bd72-49b4-8cba-b49c94e40bf9") : object "kube-system"/"original-pull-secret" not registered Sep 06 12:18:07.238065 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:07.238036 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8d626d1-3961-441f-829f-2c8a637e63f7-metrics-certs\") pod \"network-metrics-daemon-wlhb5\" (UID: \"e8d626d1-3961-441f-829f-2c8a637e63f7\") " pod="openshift-multus/network-metrics-daemon-wlhb5" Sep 06 12:18:07.238224 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:07.238086 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p48n7\" (UniqueName: \"kubernetes.io/projected/24ab4b74-f789-4642-adb7-a66bb6b75bb2-kube-api-access-p48n7\") pod \"network-check-target-gpxpp\" (UID: \"24ab4b74-f789-4642-adb7-a66bb6b75bb2\") " pod="openshift-network-diagnostics/network-check-target-gpxpp" Sep 06 12:18:07.238224 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:07.238192 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 12:18:07.238224 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:07.238218 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 06 12:18:07.238345 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:07.238240 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 06 12:18:07.238345 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:07.238252 2576 projected.go:196] Error preparing data for projected volume kube-api-access-p48n7 for pod openshift-network-diagnostics/network-check-target-gpxpp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 12:18:07.238345 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:07.238260 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8d626d1-3961-441f-829f-2c8a637e63f7-metrics-certs podName:e8d626d1-3961-441f-829f-2c8a637e63f7 nodeName:}" failed. No retries permitted until 2026-09-06 12:18:08.238241598 +0000 UTC m=+4.093999480 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8d626d1-3961-441f-829f-2c8a637e63f7-metrics-certs") pod "network-metrics-daemon-wlhb5" (UID: "e8d626d1-3961-441f-829f-2c8a637e63f7") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 12:18:07.238345 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:07.238296 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24ab4b74-f789-4642-adb7-a66bb6b75bb2-kube-api-access-p48n7 podName:24ab4b74-f789-4642-adb7-a66bb6b75bb2 nodeName:}" failed. No retries permitted until 2026-09-06 12:18:08.238279408 +0000 UTC m=+4.094037274 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-p48n7" (UniqueName: "kubernetes.io/projected/24ab4b74-f789-4642-adb7-a66bb6b75bb2-kube-api-access-p48n7") pod "network-check-target-gpxpp" (UID: "24ab4b74-f789-4642-adb7-a66bb6b75bb2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 12:18:07.551281 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:07.551250 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5970f568_4799_4f5a_9ee3_79790a2a3c7e.slice/crio-84c5988df993127559229988086fb7f9f455f5f1763ec46c02d9740ac2701dea WatchSource:0}: Error finding container 84c5988df993127559229988086fb7f9f455f5f1763ec46c02d9740ac2701dea: Status 404 returned error can't find the container with id 84c5988df993127559229988086fb7f9f455f5f1763ec46c02d9740ac2701dea Sep 06 12:18:07.558201 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:07.558175 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-05 12:13:06 +0000 UTC" deadline="2028-04-12 00:32:32.732950394 +0000 UTC" Sep 06 12:18:07.558201 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:07.558200 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14004h14m25.174752671s" Sep 06 12:18:07.562269 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:07.561367 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11b4dfc5_375d_44ed_aac5_a4ccc5c41a28.slice/crio-d37219ffdd41fccb39f04b24eab982dcc8c860d2e000c506e26c79a86dc7005c WatchSource:0}: Error finding container d37219ffdd41fccb39f04b24eab982dcc8c860d2e000c506e26c79a86dc7005c: Status 404 returned error can't find the container with id d37219ffdd41fccb39f04b24eab982dcc8c860d2e000c506e26c79a86dc7005c Sep 06 12:18:07.562540 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:07.562507 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c8ae8aa_90e6_4a07_bd2c_8d9f0ab53f78.slice/crio-6daec7c1caa76fe507eda3f60575f6e65b50e87955157d18e89c7dce47150ee7 WatchSource:0}: Error finding container 6daec7c1caa76fe507eda3f60575f6e65b50e87955157d18e89c7dce47150ee7: Status 404 returned error can't find the container with id 6daec7c1caa76fe507eda3f60575f6e65b50e87955157d18e89c7dce47150ee7 Sep 06 12:18:07.563938 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:07.563913 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39d362d3_b7d6_48b0_9ac4_05f258ee82d4.slice/crio-05af4f5d95d2d424a296671a315e4c4a12d39216f6c7addd24d69c45d8b7539e WatchSource:0}: Error finding container 05af4f5d95d2d424a296671a315e4c4a12d39216f6c7addd24d69c45d8b7539e: Status 404 returned error can't find the container with id 05af4f5d95d2d424a296671a315e4c4a12d39216f6c7addd24d69c45d8b7539e Sep 06 12:18:07.564717 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:07.564698 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b677696_8a5d_4304_bf45_2603cfc71903.slice/crio-bcff2e5e42b6e2a183fcd6355462fa2014f88fe9b1fb91da9229f66873dc851e WatchSource:0}: Error finding container bcff2e5e42b6e2a183fcd6355462fa2014f88fe9b1fb91da9229f66873dc851e: Status 404 returned error can't find the container with id bcff2e5e42b6e2a183fcd6355462fa2014f88fe9b1fb91da9229f66873dc851e Sep 06 12:18:07.565869 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:07.565848 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod470b9439_8d7e_4a84_ae68_b66b4851c5dc.slice/crio-aa2d1c3358ac4d88407c18f1219a3bd824b598dbd68dce33950bde7beb88366c WatchSource:0}: Error finding container aa2d1c3358ac4d88407c18f1219a3bd824b598dbd68dce33950bde7beb88366c: Status 404 returned error can't find the container with id aa2d1c3358ac4d88407c18f1219a3bd824b598dbd68dce33950bde7beb88366c Sep 06 12:18:07.566462 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:07.566439 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12708d6c_10bb_472c_b4a3_e8ece6ef008f.slice/crio-df1343743383714149f66dbbac4c65fcfd04280aeeb972e8d5d621dc5261f30e WatchSource:0}: Error finding container df1343743383714149f66dbbac4c65fcfd04280aeeb972e8d5d621dc5261f30e: Status 404 returned error can't find the container with id df1343743383714149f66dbbac4c65fcfd04280aeeb972e8d5d621dc5261f30e Sep 06 12:18:07.646892 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:07.646864 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:07.647061 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:07.647011 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6lx6" podUID="e095072c-bd72-49b4-8cba-b49c94e40bf9" Sep 06 12:18:07.654143 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:07.654097 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l2nhh" event={"ID":"12708d6c-10bb-472c-b4a3-e8ece6ef008f","Type":"ContainerStarted","Data":"df1343743383714149f66dbbac4c65fcfd04280aeeb972e8d5d621dc5261f30e"} Sep 06 12:18:07.655467 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:07.655441 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lt7cv" event={"ID":"470b9439-8d7e-4a84-ae68-b66b4851c5dc","Type":"ContainerStarted","Data":"aa2d1c3358ac4d88407c18f1219a3bd824b598dbd68dce33950bde7beb88366c"} Sep 06 12:18:07.658477 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:07.658449 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6n7" event={"ID":"39d362d3-b7d6-48b0-9ac4-05f258ee82d4","Type":"ContainerStarted","Data":"05af4f5d95d2d424a296671a315e4c4a12d39216f6c7addd24d69c45d8b7539e"} Sep 06 12:18:07.660339 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:07.660320 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v6tf7" event={"ID":"1c8ae8aa-90e6-4a07-bd2c-8d9f0ab53f78","Type":"ContainerStarted","Data":"6daec7c1caa76fe507eda3f60575f6e65b50e87955157d18e89c7dce47150ee7"} Sep 06 12:18:07.661534 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:07.661517 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wzmsv" event={"ID":"5970f568-4799-4f5a-9ee3-79790a2a3c7e","Type":"ContainerStarted","Data":"84c5988df993127559229988086fb7f9f455f5f1763ec46c02d9740ac2701dea"} Sep 06 12:18:07.662671 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:07.662651 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-td2jx" event={"ID":"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28","Type":"ContainerStarted","Data":"d37219ffdd41fccb39f04b24eab982dcc8c860d2e000c506e26c79a86dc7005c"} Sep 06 12:18:07.664057 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:07.664037 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" event={"ID":"4103c09f-dd43-419d-92e5-2374af5b5e01","Type":"ContainerStarted","Data":"dda7e98c32aa5bc8aeb19bac06af7e56a2a676ce1398792caaf271f689b0a094"} Sep 06 12:18:07.667188 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:07.667158 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-159.ec2.internal" event={"ID":"45bd7c4cff2b3f51ea19564c2fce8332","Type":"ContainerStarted","Data":"bb78e690fe4bae4f501d12939c57940bfff94948d8e7bb8b5ba778b288f8ccbc"} Sep 06 12:18:07.668521 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:07.668486 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gqpgt" event={"ID":"db464292-ffbf-4c25-85d3-ff4c4be77552","Type":"ContainerStarted","Data":"56de2022faaa63c55b8d689e23ce787c25a32f978480f88f55948c1049cab3f1"} Sep 06 12:18:07.669679 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:07.669655 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-57pbd" event={"ID":"8b677696-8a5d-4304-bf45-2603cfc71903","Type":"ContainerStarted","Data":"bcff2e5e42b6e2a183fcd6355462fa2014f88fe9b1fb91da9229f66873dc851e"} Sep 06 12:18:07.679033 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:07.678991 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-159.ec2.internal" podStartSLOduration=1.6789748389999999 podStartE2EDuration="1.678974839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 12:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:18:07.6788316 +0000 UTC m=+3.534589503" watchObservedRunningTime="2026-09-06 12:18:07.678974839 +0000 UTC m=+3.534732720" Sep 06 12:18:08.145275 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:08.145033 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e095072c-bd72-49b4-8cba-b49c94e40bf9-original-pull-secret\") pod \"global-pull-secret-syncer-m6lx6\" (UID: \"e095072c-bd72-49b4-8cba-b49c94e40bf9\") " pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:08.145448 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:08.145231 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 06 12:18:08.145448 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:08.145363 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e095072c-bd72-49b4-8cba-b49c94e40bf9-original-pull-secret podName:e095072c-bd72-49b4-8cba-b49c94e40bf9 nodeName:}" failed. No retries permitted until 2026-09-06 12:18:10.145343864 +0000 UTC m=+6.001101744 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e095072c-bd72-49b4-8cba-b49c94e40bf9-original-pull-secret") pod "global-pull-secret-syncer-m6lx6" (UID: "e095072c-bd72-49b4-8cba-b49c94e40bf9") : object "kube-system"/"original-pull-secret" not registered Sep 06 12:18:08.246455 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:08.246419 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8d626d1-3961-441f-829f-2c8a637e63f7-metrics-certs\") pod \"network-metrics-daemon-wlhb5\" (UID: \"e8d626d1-3961-441f-829f-2c8a637e63f7\") " pod="openshift-multus/network-metrics-daemon-wlhb5" Sep 06 12:18:08.246602 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:08.246474 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p48n7\" (UniqueName: \"kubernetes.io/projected/24ab4b74-f789-4642-adb7-a66bb6b75bb2-kube-api-access-p48n7\") pod \"network-check-target-gpxpp\" (UID: \"24ab4b74-f789-4642-adb7-a66bb6b75bb2\") " pod="openshift-network-diagnostics/network-check-target-gpxpp" Sep 06 12:18:08.246662 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:08.246620 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 06 12:18:08.246662 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:08.246637 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 06 12:18:08.246662 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:08.246649 2576 projected.go:196] Error preparing data for projected volume kube-api-access-p48n7 for pod openshift-network-diagnostics/network-check-target-gpxpp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 12:18:08.246830 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:08.246703 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24ab4b74-f789-4642-adb7-a66bb6b75bb2-kube-api-access-p48n7 podName:24ab4b74-f789-4642-adb7-a66bb6b75bb2 nodeName:}" failed. No retries permitted until 2026-09-06 12:18:10.246685898 +0000 UTC m=+6.102443779 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-p48n7" (UniqueName: "kubernetes.io/projected/24ab4b74-f789-4642-adb7-a66bb6b75bb2-kube-api-access-p48n7") pod "network-check-target-gpxpp" (UID: "24ab4b74-f789-4642-adb7-a66bb6b75bb2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 12:18:08.247136 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:08.247117 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 12:18:08.247213 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:08.247172 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8d626d1-3961-441f-829f-2c8a637e63f7-metrics-certs podName:e8d626d1-3961-441f-829f-2c8a637e63f7 nodeName:}" failed. No retries permitted until 2026-09-06 12:18:10.247156958 +0000 UTC m=+6.102914826 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8d626d1-3961-441f-829f-2c8a637e63f7-metrics-certs") pod "network-metrics-daemon-wlhb5" (UID: "e8d626d1-3961-441f-829f-2c8a637e63f7") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 12:18:08.558868 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:08.558763 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-05 12:13:06 +0000 UTC" deadline="2028-05-25 20:43:49.705320264 +0000 UTC" Sep 06 12:18:08.558868 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:08.558819 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15056h25m41.14650384s" Sep 06 12:18:08.647901 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:08.647871 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlhb5" Sep 06 12:18:08.648067 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:08.648007 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlhb5" podUID="e8d626d1-3961-441f-829f-2c8a637e63f7" Sep 06 12:18:08.648361 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:08.648343 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxpp" Sep 06 12:18:08.648458 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:08.648439 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxpp" podUID="24ab4b74-f789-4642-adb7-a66bb6b75bb2" Sep 06 12:18:08.698082 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:08.696965 2576 generic.go:356] "Generic (PLEG): container finished" podID="5c688b417bc75666ecb3d091248e9864" containerID="8a1f50becc9da3765783e662599a5025d1c5a1a8d4ce4a985fb133f972b576a0" exitCode=0 Sep 06 12:18:08.698082 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:08.697860 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-159.ec2.internal" event={"ID":"5c688b417bc75666ecb3d091248e9864","Type":"ContainerDied","Data":"8a1f50becc9da3765783e662599a5025d1c5a1a8d4ce4a985fb133f972b576a0"} Sep 06 12:18:09.647558 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:09.647073 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:09.647558 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:09.647200 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6lx6" podUID="e095072c-bd72-49b4-8cba-b49c94e40bf9" Sep 06 12:18:09.728192 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:09.727473 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-159.ec2.internal" event={"ID":"5c688b417bc75666ecb3d091248e9864","Type":"ContainerStarted","Data":"96ea19b452b86413ab028fb90590a40206538c7bc5f429660de6a3d6eac85304"} Sep 06 12:18:09.742109 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:09.741056 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-159.ec2.internal" podStartSLOduration=3.741037777 podStartE2EDuration="3.741037777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 12:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:18:09.740170723 +0000 UTC m=+5.595928627" watchObservedRunningTime="2026-09-06 12:18:09.741037777 +0000 UTC m=+5.596795668" Sep 06 12:18:10.165024 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:10.164987 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e095072c-bd72-49b4-8cba-b49c94e40bf9-original-pull-secret\") pod \"global-pull-secret-syncer-m6lx6\" (UID: \"e095072c-bd72-49b4-8cba-b49c94e40bf9\") " pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:10.165190 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:10.165180 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 06 12:18:10.165268 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:10.165240 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e095072c-bd72-49b4-8cba-b49c94e40bf9-original-pull-secret podName:e095072c-bd72-49b4-8cba-b49c94e40bf9 nodeName:}" failed. No retries permitted until 2026-09-06 12:18:14.165222914 +0000 UTC m=+10.020980784 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e095072c-bd72-49b4-8cba-b49c94e40bf9-original-pull-secret") pod "global-pull-secret-syncer-m6lx6" (UID: "e095072c-bd72-49b4-8cba-b49c94e40bf9") : object "kube-system"/"original-pull-secret" not registered Sep 06 12:18:10.266109 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:10.266066 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8d626d1-3961-441f-829f-2c8a637e63f7-metrics-certs\") pod \"network-metrics-daemon-wlhb5\" (UID: \"e8d626d1-3961-441f-829f-2c8a637e63f7\") " pod="openshift-multus/network-metrics-daemon-wlhb5" Sep 06 12:18:10.266262 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:10.266125 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p48n7\" (UniqueName: \"kubernetes.io/projected/24ab4b74-f789-4642-adb7-a66bb6b75bb2-kube-api-access-p48n7\") pod \"network-check-target-gpxpp\" (UID: \"24ab4b74-f789-4642-adb7-a66bb6b75bb2\") " pod="openshift-network-diagnostics/network-check-target-gpxpp" Sep 06 12:18:10.266337 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:10.266263 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 06 12:18:10.266337 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:10.266285 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 06 12:18:10.266337 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:10.266298 2576 projected.go:196] Error preparing data for projected volume kube-api-access-p48n7 for pod openshift-network-diagnostics/network-check-target-gpxpp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 12:18:10.266473 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:10.266360 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24ab4b74-f789-4642-adb7-a66bb6b75bb2-kube-api-access-p48n7 podName:24ab4b74-f789-4642-adb7-a66bb6b75bb2 nodeName:}" failed. No retries permitted until 2026-09-06 12:18:14.266340746 +0000 UTC m=+10.122098618 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-p48n7" (UniqueName: "kubernetes.io/projected/24ab4b74-f789-4642-adb7-a66bb6b75bb2-kube-api-access-p48n7") pod "network-check-target-gpxpp" (UID: "24ab4b74-f789-4642-adb7-a66bb6b75bb2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 12:18:10.266807 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:10.266773 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 12:18:10.266932 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:10.266830 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8d626d1-3961-441f-829f-2c8a637e63f7-metrics-certs podName:e8d626d1-3961-441f-829f-2c8a637e63f7 nodeName:}" failed. No retries permitted until 2026-09-06 12:18:14.266813351 +0000 UTC m=+10.122571216 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8d626d1-3961-441f-829f-2c8a637e63f7-metrics-certs") pod "network-metrics-daemon-wlhb5" (UID: "e8d626d1-3961-441f-829f-2c8a637e63f7") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 12:18:10.647865 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:10.647839 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlhb5" Sep 06 12:18:10.648322 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:10.647978 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlhb5" podUID="e8d626d1-3961-441f-829f-2c8a637e63f7" Sep 06 12:18:10.648396 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:10.648360 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxpp" Sep 06 12:18:10.648451 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:10.648432 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxpp" podUID="24ab4b74-f789-4642-adb7-a66bb6b75bb2" Sep 06 12:18:11.646926 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:11.646880 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:11.647079 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:11.647011 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6lx6" podUID="e095072c-bd72-49b4-8cba-b49c94e40bf9" Sep 06 12:18:12.647424 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:12.647391 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlhb5" Sep 06 12:18:12.647800 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:12.647537 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlhb5" podUID="e8d626d1-3961-441f-829f-2c8a637e63f7" Sep 06 12:18:12.647948 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:12.647926 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxpp" Sep 06 12:18:12.648078 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:12.648029 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxpp" podUID="24ab4b74-f789-4642-adb7-a66bb6b75bb2" Sep 06 12:18:13.646621 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:13.646588 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:13.646810 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:13.646701 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6lx6" podUID="e095072c-bd72-49b4-8cba-b49c94e40bf9" Sep 06 12:18:14.200673 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:14.200633 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e095072c-bd72-49b4-8cba-b49c94e40bf9-original-pull-secret\") pod \"global-pull-secret-syncer-m6lx6\" (UID: \"e095072c-bd72-49b4-8cba-b49c94e40bf9\") " pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:14.201136 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:14.200856 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 06 12:18:14.201136 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:14.200916 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e095072c-bd72-49b4-8cba-b49c94e40bf9-original-pull-secret podName:e095072c-bd72-49b4-8cba-b49c94e40bf9 nodeName:}" failed. No retries permitted until 2026-09-06 12:18:22.200897836 +0000 UTC m=+18.056655702 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e095072c-bd72-49b4-8cba-b49c94e40bf9-original-pull-secret") pod "global-pull-secret-syncer-m6lx6" (UID: "e095072c-bd72-49b4-8cba-b49c94e40bf9") : object "kube-system"/"original-pull-secret" not registered Sep 06 12:18:14.301545 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:14.301509 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8d626d1-3961-441f-829f-2c8a637e63f7-metrics-certs\") pod \"network-metrics-daemon-wlhb5\" (UID: \"e8d626d1-3961-441f-829f-2c8a637e63f7\") " pod="openshift-multus/network-metrics-daemon-wlhb5" Sep 06 12:18:14.301691 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:14.301562 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p48n7\" (UniqueName: \"kubernetes.io/projected/24ab4b74-f789-4642-adb7-a66bb6b75bb2-kube-api-access-p48n7\") pod \"network-check-target-gpxpp\" (UID: \"24ab4b74-f789-4642-adb7-a66bb6b75bb2\") " pod="openshift-network-diagnostics/network-check-target-gpxpp" Sep 06 12:18:14.301774 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:14.301709 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 06 12:18:14.301774 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:14.301729 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 06 12:18:14.301774 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:14.301741 2576 projected.go:196] Error preparing data for projected volume kube-api-access-p48n7 for pod openshift-network-diagnostics/network-check-target-gpxpp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 12:18:14.301970 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:14.301809 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24ab4b74-f789-4642-adb7-a66bb6b75bb2-kube-api-access-p48n7 podName:24ab4b74-f789-4642-adb7-a66bb6b75bb2 nodeName:}" failed. No retries permitted until 2026-09-06 12:18:22.301796045 +0000 UTC m=+18.157553910 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-p48n7" (UniqueName: "kubernetes.io/projected/24ab4b74-f789-4642-adb7-a66bb6b75bb2-kube-api-access-p48n7") pod "network-check-target-gpxpp" (UID: "24ab4b74-f789-4642-adb7-a66bb6b75bb2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 12:18:14.302098 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:14.302083 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 12:18:14.302150 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:14.302120 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8d626d1-3961-441f-829f-2c8a637e63f7-metrics-certs podName:e8d626d1-3961-441f-829f-2c8a637e63f7 nodeName:}" failed. No retries permitted until 2026-09-06 12:18:22.302110983 +0000 UTC m=+18.157868848 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8d626d1-3961-441f-829f-2c8a637e63f7-metrics-certs") pod "network-metrics-daemon-wlhb5" (UID: "e8d626d1-3961-441f-829f-2c8a637e63f7") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 12:18:14.648195 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:14.648166 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxpp" Sep 06 12:18:14.648364 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:14.648268 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxpp" podUID="24ab4b74-f789-4642-adb7-a66bb6b75bb2" Sep 06 12:18:14.648718 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:14.648699 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlhb5" Sep 06 12:18:14.648851 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:14.648829 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlhb5" podUID="e8d626d1-3961-441f-829f-2c8a637e63f7" Sep 06 12:18:15.647039 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:15.647006 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:15.647498 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:15.647118 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6lx6" podUID="e095072c-bd72-49b4-8cba-b49c94e40bf9" Sep 06 12:18:16.647398 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:16.647360 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxpp" Sep 06 12:18:16.647797 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:16.647479 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxpp" podUID="24ab4b74-f789-4642-adb7-a66bb6b75bb2" Sep 06 12:18:16.647865 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:16.647850 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlhb5" Sep 06 12:18:16.647970 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:16.647950 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlhb5" podUID="e8d626d1-3961-441f-829f-2c8a637e63f7" Sep 06 12:18:17.647266 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:17.647110 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:17.647266 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:17.647230 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6lx6" podUID="e095072c-bd72-49b4-8cba-b49c94e40bf9" Sep 06 12:18:18.647155 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:18.647128 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxpp" Sep 06 12:18:18.647327 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:18.647134 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlhb5" Sep 06 12:18:18.647380 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:18.647321 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlhb5" podUID="e8d626d1-3961-441f-829f-2c8a637e63f7" Sep 06 12:18:18.647380 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:18.647223 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxpp" podUID="24ab4b74-f789-4642-adb7-a66bb6b75bb2" Sep 06 12:18:19.647361 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:19.647331 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:19.647815 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:19.647450 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6lx6" podUID="e095072c-bd72-49b4-8cba-b49c94e40bf9" Sep 06 12:18:20.647148 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:20.647109 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlhb5" Sep 06 12:18:20.647336 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:20.647248 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlhb5" podUID="e8d626d1-3961-441f-829f-2c8a637e63f7" Sep 06 12:18:20.647400 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:20.647343 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxpp" Sep 06 12:18:20.647723 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:20.647451 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxpp" podUID="24ab4b74-f789-4642-adb7-a66bb6b75bb2" Sep 06 12:18:21.647364 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:21.647332 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:21.647535 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:21.647442 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6lx6" podUID="e095072c-bd72-49b4-8cba-b49c94e40bf9" Sep 06 12:18:22.260586 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:22.260548 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e095072c-bd72-49b4-8cba-b49c94e40bf9-original-pull-secret\") pod \"global-pull-secret-syncer-m6lx6\" (UID: \"e095072c-bd72-49b4-8cba-b49c94e40bf9\") " pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:22.260772 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:22.260711 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 06 12:18:22.260835 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:22.260796 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e095072c-bd72-49b4-8cba-b49c94e40bf9-original-pull-secret podName:e095072c-bd72-49b4-8cba-b49c94e40bf9 nodeName:}" failed. No retries permitted until 2026-09-06 12:18:38.26077985 +0000 UTC m=+34.116537714 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e095072c-bd72-49b4-8cba-b49c94e40bf9-original-pull-secret") pod "global-pull-secret-syncer-m6lx6" (UID: "e095072c-bd72-49b4-8cba-b49c94e40bf9") : object "kube-system"/"original-pull-secret" not registered Sep 06 12:18:22.361146 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:22.361116 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8d626d1-3961-441f-829f-2c8a637e63f7-metrics-certs\") pod \"network-metrics-daemon-wlhb5\" (UID: \"e8d626d1-3961-441f-829f-2c8a637e63f7\") " pod="openshift-multus/network-metrics-daemon-wlhb5" Sep 06 12:18:22.361307 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:22.361164 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p48n7\" (UniqueName: \"kubernetes.io/projected/24ab4b74-f789-4642-adb7-a66bb6b75bb2-kube-api-access-p48n7\") pod \"network-check-target-gpxpp\" (UID: \"24ab4b74-f789-4642-adb7-a66bb6b75bb2\") " pod="openshift-network-diagnostics/network-check-target-gpxpp" Sep 06 12:18:22.361307 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:22.361284 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 12:18:22.361307 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:22.361303 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 06 12:18:22.361478 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:22.361324 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 06 12:18:22.361478 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:22.361336 2576 projected.go:196] Error preparing data for projected volume kube-api-access-p48n7 for pod openshift-network-diagnostics/network-check-target-gpxpp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 12:18:22.361478 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:22.361356 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8d626d1-3961-441f-829f-2c8a637e63f7-metrics-certs podName:e8d626d1-3961-441f-829f-2c8a637e63f7 nodeName:}" failed. No retries permitted until 2026-09-06 12:18:38.361336995 +0000 UTC m=+34.217094876 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8d626d1-3961-441f-829f-2c8a637e63f7-metrics-certs") pod "network-metrics-daemon-wlhb5" (UID: "e8d626d1-3961-441f-829f-2c8a637e63f7") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 12:18:22.361478 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:22.361382 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24ab4b74-f789-4642-adb7-a66bb6b75bb2-kube-api-access-p48n7 podName:24ab4b74-f789-4642-adb7-a66bb6b75bb2 nodeName:}" failed. No retries permitted until 2026-09-06 12:18:38.361366658 +0000 UTC m=+34.217124535 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-p48n7" (UniqueName: "kubernetes.io/projected/24ab4b74-f789-4642-adb7-a66bb6b75bb2-kube-api-access-p48n7") pod "network-check-target-gpxpp" (UID: "24ab4b74-f789-4642-adb7-a66bb6b75bb2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 12:18:22.647558 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:22.647526 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxpp" Sep 06 12:18:22.648047 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:22.647657 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxpp" podUID="24ab4b74-f789-4642-adb7-a66bb6b75bb2" Sep 06 12:18:22.648047 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:22.647729 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlhb5" Sep 06 12:18:22.648047 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:22.647879 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlhb5" podUID="e8d626d1-3961-441f-829f-2c8a637e63f7" Sep 06 12:18:23.647340 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:23.647309 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:23.647513 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:23.647412 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6lx6" podUID="e095072c-bd72-49b4-8cba-b49c94e40bf9" Sep 06 12:18:24.648263 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:24.648220 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxpp" Sep 06 12:18:24.648670 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:24.648345 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxpp" podUID="24ab4b74-f789-4642-adb7-a66bb6b75bb2" Sep 06 12:18:24.648670 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:24.648437 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlhb5" Sep 06 12:18:24.648670 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:24.648528 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlhb5" podUID="e8d626d1-3961-441f-829f-2c8a637e63f7" Sep 06 12:18:25.647507 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:25.647119 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:25.647507 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:25.647419 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6lx6" podUID="e095072c-bd72-49b4-8cba-b49c94e40bf9" Sep 06 12:18:25.759643 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:25.759605 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lt7cv" event={"ID":"470b9439-8d7e-4a84-ae68-b66b4851c5dc","Type":"ContainerStarted","Data":"ad091c5c4bf7bbe7cda8df6c000810dfd113f30f2089c91f19034ec04a80ec37"} Sep 06 12:18:25.761375 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:25.761349 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6n7" event={"ID":"39d362d3-b7d6-48b0-9ac4-05f258ee82d4","Type":"ContainerStarted","Data":"4cef492aa06c958547896b479fd1599090076bae3aa173dadc9e1378b3a14fe8"} Sep 06 12:18:25.763048 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:25.762968 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v6tf7" event={"ID":"1c8ae8aa-90e6-4a07-bd2c-8d9f0ab53f78","Type":"ContainerStarted","Data":"00daafd06db6c953e7dd3cfa6bf03ebe3c2c842b7eba3d60eeb791fa8bd5e695"} Sep 06 12:18:25.764785 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:25.764739 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-td2jx" event={"ID":"11b4dfc5-375d-44ed-aac5-a4ccc5c41a28","Type":"ContainerStarted","Data":"cd3093b42a2104daa116b9b3beadbac7f244f0f61db226b6293143d6edbaa9e6"} Sep 06 12:18:25.767011 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:25.766995 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98vz4_4103c09f-dd43-419d-92e5-2374af5b5e01/ovn-acl-logging/0.log" Sep 06 12:18:25.767472 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:25.767451 2576 generic.go:356] "Generic (PLEG): container finished" podID="4103c09f-dd43-419d-92e5-2374af5b5e01" containerID="6df1dc496a25144dcf080dea280de7e86292086b104b27de1921e6cef013aa9d" exitCode=1 Sep 06 12:18:25.767571 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:25.767521 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" event={"ID":"4103c09f-dd43-419d-92e5-2374af5b5e01","Type":"ContainerStarted","Data":"474f3ceebb043a785e13c6c94454240f15596e2fe39eb7cc11277e23fbb38b3f"} Sep 06 12:18:25.767647 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:25.767598 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" event={"ID":"4103c09f-dd43-419d-92e5-2374af5b5e01","Type":"ContainerDied","Data":"6df1dc496a25144dcf080dea280de7e86292086b104b27de1921e6cef013aa9d"} Sep 06 12:18:25.767700 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:25.767655 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" event={"ID":"4103c09f-dd43-419d-92e5-2374af5b5e01","Type":"ContainerStarted","Data":"3120ea0de9612eac54861fc387382f7b621046b460f62e572c671a46bae9ba5c"} Sep 06 12:18:25.769274 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:25.769239 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gqpgt" event={"ID":"db464292-ffbf-4c25-85d3-ff4c4be77552","Type":"ContainerStarted","Data":"657bfeccfe83e57d236f7c27cd34d1bf37e95f1975013abc5b745e610ce88b0a"} Sep 06 12:18:25.770964 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:25.770944 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-57pbd" event={"ID":"8b677696-8a5d-4304-bf45-2603cfc71903","Type":"ContainerStarted","Data":"3dac562edbc9d7b554f9988124ff5f2654efde3c10ae95d99664c0330068f165"} Sep 06 12:18:25.772650 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:25.772628 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l2nhh" event={"ID":"12708d6c-10bb-472c-b4a3-e8ece6ef008f","Type":"ContainerStarted","Data":"11e9cfbb7eabca2bdddeccf24b46c74a913ef8d417f58f41a766692f068d9311"} Sep 06 12:18:25.779348 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:25.779310 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lt7cv" podStartSLOduration=4.117172072 podStartE2EDuration="21.779297199s" totalImagesPullingTime="17.662125127s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 12:18:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:18:25.778909426 +0000 UTC m=+21.634667336" watchObservedRunningTime="2026-09-06 12:18:25.779297199 +0000 UTC m=+21.635055086" Sep 06 12:18:25.796297 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:25.796259 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-l2nhh" podStartSLOduration=11.561990934 podStartE2EDuration="21.796247097s" totalImagesPullingTime="10.234256163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:18:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:18:25.795528331 +0000 UTC m=+21.651286218" watchObservedRunningTime="2026-09-06 12:18:25.796247097 +0000 UTC m=+21.652005229" Sep 06 12:18:25.836500 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:25.836405 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-v6tf7" podStartSLOduration=4.170845689 podStartE2EDuration="21.836389801s" totalImagesPullingTime="17.665544112s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 12:18:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:18:25.835950365 +0000 UTC m=+21.691708270" watchObservedRunningTime="2026-09-06 12:18:25.836389801 +0000 UTC m=+21.692147687" Sep 06 12:18:25.855775 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:25.855699 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-57pbd" podStartSLOduration=4.15323611 podStartE2EDuration="21.855684273s" totalImagesPullingTime="17.702448163s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 12:18:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:18:25.855627442 +0000 UTC m=+21.711385328" watchObservedRunningTime="2026-09-06 12:18:25.855684273 +0000 UTC m=+21.711442159" Sep 06 12:18:25.973877 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:25.973737 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v6tf7_1c8ae8aa-90e6-4a07-bd2c-8d9f0ab53f78/dns-node-resolver/0.log" Sep 06 12:18:26.238645 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:26.238608 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-l2nhh" Sep 06 12:18:26.239209 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:26.239191 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-l2nhh" Sep 06 12:18:26.253454 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:26.253402 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-td2jx" podStartSLOduration=4.585339352 podStartE2EDuration="22.253384737s" totalImagesPullingTime="17.668045385s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 12:18:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:18:25.872915681 +0000 UTC m=+21.728673618" watchObservedRunningTime="2026-09-06 12:18:26.253384737 +0000 UTC m=+22.109142603" Sep 06 12:18:26.647355 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:26.647326 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxpp" Sep 06 12:18:26.647511 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:26.647451 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxpp" podUID="24ab4b74-f789-4642-adb7-a66bb6b75bb2" Sep 06 12:18:26.647575 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:26.647512 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlhb5" Sep 06 12:18:26.647662 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:26.647638 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlhb5" podUID="e8d626d1-3961-441f-829f-2c8a637e63f7" Sep 06 12:18:26.757130 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:26.757109 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lt7cv_470b9439-8d7e-4a84-ae68-b66b4851c5dc/node-ca/0.log" Sep 06 12:18:26.775846 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:26.775783 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wzmsv" event={"ID":"5970f568-4799-4f5a-9ee3-79790a2a3c7e","Type":"ContainerStarted","Data":"6e3808e24ea9a8d0c7c71fdab6545e9c54b6968d3dcddf488c1e1dbd6e1c18f2"} Sep 06 12:18:26.778583 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:26.778566 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98vz4_4103c09f-dd43-419d-92e5-2374af5b5e01/ovn-acl-logging/0.log" Sep 06 12:18:26.778972 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:26.778949 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" event={"ID":"4103c09f-dd43-419d-92e5-2374af5b5e01","Type":"ContainerStarted","Data":"88dc9d6c36118db6567760b66835bfd429cb794a153bbcbe903ce38c67901a9f"} Sep 06 12:18:26.779042 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:26.779022 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" event={"ID":"4103c09f-dd43-419d-92e5-2374af5b5e01","Type":"ContainerStarted","Data":"b753d23702fb59ad57087c2fcac3365469896a0cd016468b2d28605b9d76def7"} Sep 06 12:18:26.779081 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:26.779058 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" event={"ID":"4103c09f-dd43-419d-92e5-2374af5b5e01","Type":"ContainerStarted","Data":"d455dbb1144bc70ce64f258ca2c49c99ed46487838312e25a717adbbc5631b07"} Sep 06 12:18:26.780363 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:26.780344 2576 generic.go:356] "Generic (PLEG): container finished" podID="db464292-ffbf-4c25-85d3-ff4c4be77552" containerID="657bfeccfe83e57d236f7c27cd34d1bf37e95f1975013abc5b745e610ce88b0a" exitCode=0 Sep 06 12:18:26.780473 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:26.780438 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gqpgt" event={"ID":"db464292-ffbf-4c25-85d3-ff4c4be77552","Type":"ContainerDied","Data":"657bfeccfe83e57d236f7c27cd34d1bf37e95f1975013abc5b745e610ce88b0a"} Sep 06 12:18:26.794420 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:26.794384 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wzmsv" podStartSLOduration=5.144812997 podStartE2EDuration="22.794372078s" totalImagesPullingTime="17.649559081s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 12:18:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:18:26.793997372 +0000 UTC m=+22.649755271" watchObservedRunningTime="2026-09-06 12:18:26.794372078 +0000 UTC m=+22.650129964" Sep 06 12:18:27.166943 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:27.166903 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 06 12:18:27.593833 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:27.593729 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-06T12:18:27.166922092Z","UUID":"11ddcebd-a98b-432d-afa7-4c4ac82f1cba","Handler":null,"Name":"","Endpoint":""} Sep 06 12:18:27.595400 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:27.595374 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 06 12:18:27.595400 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:27.595404 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 06 12:18:27.646960 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:27.646936 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:27.647104 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:27.647046 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6lx6" podUID="e095072c-bd72-49b4-8cba-b49c94e40bf9" Sep 06 12:18:27.785613 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:27.785584 2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 06 12:18:27.786068 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:27.785585 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6n7" event={"ID":"39d362d3-b7d6-48b0-9ac4-05f258ee82d4","Type":"ContainerStarted","Data":"420ab9d764305284cad7957f59c847c59a2b84cd506cbc5f07d6ad2bb659e590"} Sep 06 12:18:28.646773 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:28.646727 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlhb5" Sep 06 12:18:28.646873 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:28.646777 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxpp" Sep 06 12:18:28.646925 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:28.646888 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlhb5" podUID="e8d626d1-3961-441f-829f-2c8a637e63f7" Sep 06 12:18:28.647027 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:28.647002 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxpp" podUID="24ab4b74-f789-4642-adb7-a66bb6b75bb2" Sep 06 12:18:28.790375 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:28.790300 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6n7" event={"ID":"39d362d3-b7d6-48b0-9ac4-05f258ee82d4","Type":"ContainerStarted","Data":"da4a57ff8073fed72e798d0bca11ba1a2f0f8df1d66bad1ef5713fbd1c598f33"} Sep 06 12:18:28.794276 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:28.794259 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98vz4_4103c09f-dd43-419d-92e5-2374af5b5e01/ovn-acl-logging/0.log" Sep 06 12:18:28.794710 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:28.794687 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" event={"ID":"4103c09f-dd43-419d-92e5-2374af5b5e01","Type":"ContainerStarted","Data":"5f5bd0af5e378520d16b650373ca4fd97bd77e1462a80af6e80a7ec463aecedc"} Sep 06 12:18:29.647504 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:29.647270 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:29.647645 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:29.647589 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6lx6" podUID="e095072c-bd72-49b4-8cba-b49c94e40bf9" Sep 06 12:18:30.647486 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:30.647457 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlhb5" Sep 06 12:18:30.647486 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:30.647485 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxpp" Sep 06 12:18:30.647993 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:30.647620 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxpp" podUID="24ab4b74-f789-4642-adb7-a66bb6b75bb2" Sep 06 12:18:30.647993 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:30.647664 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlhb5" podUID="e8d626d1-3961-441f-829f-2c8a637e63f7" Sep 06 12:18:31.646532 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:31.646482 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:31.646632 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:31.646577 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6lx6" podUID="e095072c-bd72-49b4-8cba-b49c94e40bf9" Sep 06 12:18:31.807299 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:31.807252 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98vz4_4103c09f-dd43-419d-92e5-2374af5b5e01/ovn-acl-logging/0.log" Sep 06 12:18:31.808029 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:31.807633 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" event={"ID":"4103c09f-dd43-419d-92e5-2374af5b5e01","Type":"ContainerStarted","Data":"1c43d84824612ec5552f56224ab54614650a6dfc5c6aaebb1ebb14733a182d8c"} Sep 06 12:18:31.808086 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:31.808034 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:31.808086 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:31.808043 2576 scope.go:117] "RemoveContainer" containerID="6df1dc496a25144dcf080dea280de7e86292086b104b27de1921e6cef013aa9d" Sep 06 12:18:31.808153 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:31.808119 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:31.809704 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:31.809680 2576 generic.go:356] "Generic (PLEG): container finished" podID="db464292-ffbf-4c25-85d3-ff4c4be77552" containerID="10efbc6c66db12990e96fc777965ce3bf6889ed17c7a742da7e524aab5f7f9e8" exitCode=0 Sep 06 12:18:31.809817 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:31.809722 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gqpgt" event={"ID":"db464292-ffbf-4c25-85d3-ff4c4be77552","Type":"ContainerDied","Data":"10efbc6c66db12990e96fc777965ce3bf6889ed17c7a742da7e524aab5f7f9e8"} Sep 06 12:18:31.824253 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:31.824235 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:31.830664 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:31.830627 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6n7" podStartSLOduration=7.284837349 podStartE2EDuration="27.830615753s" totalImagesPullingTime="20.545778404s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 12:18:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:18:28.810968888 +0000 UTC m=+24.666726776" watchObservedRunningTime="2026-09-06 12:18:31.830615753 +0000 UTC m=+27.686373639" Sep 06 12:18:32.650223 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:32.650202 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlhb5" Sep 06 12:18:32.650316 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:32.650297 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlhb5" podUID="e8d626d1-3961-441f-829f-2c8a637e63f7" Sep 06 12:18:32.650368 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:32.650354 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxpp" Sep 06 12:18:32.650453 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:32.650436 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxpp" podUID="24ab4b74-f789-4642-adb7-a66bb6b75bb2" Sep 06 12:18:32.814457 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:32.814405 2576 generic.go:356] "Generic (PLEG): container finished" podID="db464292-ffbf-4c25-85d3-ff4c4be77552" containerID="1c31851362aa90e8269a1eda6c52ae4b82f92c670e4e4b3f4d37bf0aa3b78f28" exitCode=0 Sep 06 12:18:32.814812 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:32.814468 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gqpgt" event={"ID":"db464292-ffbf-4c25-85d3-ff4c4be77552","Type":"ContainerDied","Data":"1c31851362aa90e8269a1eda6c52ae4b82f92c670e4e4b3f4d37bf0aa3b78f28"} Sep 06 12:18:32.818813 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:32.818795 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98vz4_4103c09f-dd43-419d-92e5-2374af5b5e01/ovn-acl-logging/0.log" Sep 06 12:18:32.819176 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:32.819143 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" event={"ID":"4103c09f-dd43-419d-92e5-2374af5b5e01","Type":"ContainerStarted","Data":"020f8af760df3efbd2b6ed6c73f2801f6ba16837d42081a78ce0b20853f8caa7"} Sep 06 12:18:32.819352 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:32.819331 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:32.837684 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:32.836140 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:18:32.859675 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:32.859641 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" podStartSLOduration=11.160848827 podStartE2EDuration="28.859629929s" totalImagesPullingTime="17.698781102s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 12:18:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:18:32.858279409 +0000 UTC m=+28.714037296" watchObservedRunningTime="2026-09-06 12:18:32.859629929 +0000 UTC m=+28.715387816" Sep 06 12:18:33.647261 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:33.647238 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:33.647402 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:33.647345 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6lx6" podUID="e095072c-bd72-49b4-8cba-b49c94e40bf9" Sep 06 12:18:33.824151 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:33.824080 2576 generic.go:356] "Generic (PLEG): container finished" podID="db464292-ffbf-4c25-85d3-ff4c4be77552" containerID="863e58956df2a2d66ffca67a81656c36cdc0e59a5e53862a520f9f0526e9bae3" exitCode=0 Sep 06 12:18:33.824513 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:33.824174 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gqpgt" event={"ID":"db464292-ffbf-4c25-85d3-ff4c4be77552","Type":"ContainerDied","Data":"863e58956df2a2d66ffca67a81656c36cdc0e59a5e53862a520f9f0526e9bae3"} Sep 06 12:18:34.524960 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:34.524928 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-l2nhh" Sep 06 12:18:34.525111 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:34.525053 2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 06 12:18:34.525463 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:34.525434 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l2nhh" Sep 06 12:18:34.650975 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:34.648952 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlhb5" Sep 06 12:18:34.650975 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:34.649094 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlhb5" podUID="e8d626d1-3961-441f-829f-2c8a637e63f7" Sep 06 12:18:34.650975 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:34.649128 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxpp" Sep 06 12:18:34.650975 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:34.649216 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxpp" podUID="24ab4b74-f789-4642-adb7-a66bb6b75bb2" Sep 06 12:18:35.646999 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:35.646971 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:35.647397 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:35.647078 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6lx6" podUID="e095072c-bd72-49b4-8cba-b49c94e40bf9" Sep 06 12:18:36.646630 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:36.646595 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlhb5" Sep 06 12:18:36.646803 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:36.646735 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlhb5" podUID="e8d626d1-3961-441f-829f-2c8a637e63f7" Sep 06 12:18:36.646870 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:36.646822 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxpp" Sep 06 12:18:36.646953 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:36.646929 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxpp" podUID="24ab4b74-f789-4642-adb7-a66bb6b75bb2" Sep 06 12:18:37.646808 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:37.646773 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:37.647250 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:37.646902 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6lx6" podUID="e095072c-bd72-49b4-8cba-b49c94e40bf9" Sep 06 12:18:38.279692 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:38.279668 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e095072c-bd72-49b4-8cba-b49c94e40bf9-original-pull-secret\") pod \"global-pull-secret-syncer-m6lx6\" (UID: \"e095072c-bd72-49b4-8cba-b49c94e40bf9\") " pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:38.279821 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:38.279807 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 06 12:18:38.279868 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:38.279859 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e095072c-bd72-49b4-8cba-b49c94e40bf9-original-pull-secret podName:e095072c-bd72-49b4-8cba-b49c94e40bf9 nodeName:}" failed. No retries permitted until 2026-09-06 12:19:10.279846471 +0000 UTC m=+66.135604337 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e095072c-bd72-49b4-8cba-b49c94e40bf9-original-pull-secret") pod "global-pull-secret-syncer-m6lx6" (UID: "e095072c-bd72-49b4-8cba-b49c94e40bf9") : object "kube-system"/"original-pull-secret" not registered Sep 06 12:18:38.380852 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:38.380799 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8d626d1-3961-441f-829f-2c8a637e63f7-metrics-certs\") pod \"network-metrics-daemon-wlhb5\" (UID: \"e8d626d1-3961-441f-829f-2c8a637e63f7\") " pod="openshift-multus/network-metrics-daemon-wlhb5" Sep 06 12:18:38.380852 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:38.380831 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p48n7\" (UniqueName: \"kubernetes.io/projected/24ab4b74-f789-4642-adb7-a66bb6b75bb2-kube-api-access-p48n7\") pod \"network-check-target-gpxpp\" (UID: \"24ab4b74-f789-4642-adb7-a66bb6b75bb2\") " pod="openshift-network-diagnostics/network-check-target-gpxpp" Sep 06 12:18:38.380978 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:38.380924 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 06 12:18:38.380978 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:38.380929 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 12:18:38.380978 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:38.380937 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 06 12:18:38.380978 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:38.380958 2576 projected.go:196] Error preparing data for projected volume kube-api-access-p48n7 for pod openshift-network-diagnostics/network-check-target-gpxpp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 12:18:38.381095 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:38.380988 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8d626d1-3961-441f-829f-2c8a637e63f7-metrics-certs podName:e8d626d1-3961-441f-829f-2c8a637e63f7 nodeName:}" failed. No retries permitted until 2026-09-06 12:19:10.380969238 +0000 UTC m=+66.236727118 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8d626d1-3961-441f-829f-2c8a637e63f7-metrics-certs") pod "network-metrics-daemon-wlhb5" (UID: "e8d626d1-3961-441f-829f-2c8a637e63f7") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 12:18:38.381095 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:38.381003 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24ab4b74-f789-4642-adb7-a66bb6b75bb2-kube-api-access-p48n7 podName:24ab4b74-f789-4642-adb7-a66bb6b75bb2 nodeName:}" failed. No retries permitted until 2026-09-06 12:19:10.380996392 +0000 UTC m=+66.236754256 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-p48n7" (UniqueName: "kubernetes.io/projected/24ab4b74-f789-4642-adb7-a66bb6b75bb2-kube-api-access-p48n7") pod "network-check-target-gpxpp" (UID: "24ab4b74-f789-4642-adb7-a66bb6b75bb2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 12:18:38.653508 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:38.651050 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxpp" Sep 06 12:18:38.653508 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:38.651197 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxpp" podUID="24ab4b74-f789-4642-adb7-a66bb6b75bb2" Sep 06 12:18:38.653508 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:38.651616 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlhb5" Sep 06 12:18:38.653508 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:38.651791 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlhb5" podUID="e8d626d1-3961-441f-829f-2c8a637e63f7" Sep 06 12:18:38.838850 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:38.838819 2576 generic.go:356] "Generic (PLEG): container finished" podID="db464292-ffbf-4c25-85d3-ff4c4be77552" containerID="5beffe1c347bcc898d304692c458a80e3cf73307e28aa8e48d6992af9b509868" exitCode=0 Sep 06 12:18:38.838961 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:38.838859 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gqpgt" event={"ID":"db464292-ffbf-4c25-85d3-ff4c4be77552","Type":"ContainerDied","Data":"5beffe1c347bcc898d304692c458a80e3cf73307e28aa8e48d6992af9b509868"} Sep 06 12:18:39.647169 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:39.647128 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:39.647365 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:39.647243 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6lx6" podUID="e095072c-bd72-49b4-8cba-b49c94e40bf9" Sep 06 12:18:39.847918 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:39.847875 2576 generic.go:356] "Generic (PLEG): container finished" podID="db464292-ffbf-4c25-85d3-ff4c4be77552" containerID="73af35af0a8eb42a89dcaa4d5b2e381c37905916c760621260899f69af9af578" exitCode=0 Sep 06 12:18:39.848286 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:39.847927 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gqpgt" event={"ID":"db464292-ffbf-4c25-85d3-ff4c4be77552","Type":"ContainerDied","Data":"73af35af0a8eb42a89dcaa4d5b2e381c37905916c760621260899f69af9af578"} Sep 06 12:18:40.649646 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:40.649616 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxpp" Sep 06 12:18:40.649785 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:40.649700 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxpp" podUID="24ab4b74-f789-4642-adb7-a66bb6b75bb2" Sep 06 12:18:40.649841 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:40.649789 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlhb5" Sep 06 12:18:40.649899 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:40.649882 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlhb5" podUID="e8d626d1-3961-441f-829f-2c8a637e63f7" Sep 06 12:18:40.853550 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:40.853514 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gqpgt" event={"ID":"db464292-ffbf-4c25-85d3-ff4c4be77552","Type":"ContainerStarted","Data":"70a4ae7f89aa52e0e3972137bfb629e1044f2ee87bb5e3255f99de3ede003766"} Sep 06 12:18:40.873788 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:40.873722 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gqpgt" podStartSLOduration=8.783582258 podStartE2EDuration="36.873707739s" totalImagesPullingTime="28.090125481s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 12:18:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:18:40.873335222 +0000 UTC m=+36.729093108" watchObservedRunningTime="2026-09-06 12:18:40.873707739 +0000 UTC m=+36.729465626" Sep 06 12:18:41.647444 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:41.647409 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:41.647618 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:41.647512 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6lx6" podUID="e095072c-bd72-49b4-8cba-b49c94e40bf9" Sep 06 12:18:42.647296 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:42.647260 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxpp" Sep 06 12:18:42.647296 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:42.647290 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlhb5" Sep 06 12:18:42.647808 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:42.647364 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxpp" podUID="24ab4b74-f789-4642-adb7-a66bb6b75bb2" Sep 06 12:18:42.647808 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:42.647507 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlhb5" podUID="e8d626d1-3961-441f-829f-2c8a637e63f7" Sep 06 12:18:43.647063 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:43.647028 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:43.647246 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:43.647133 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6lx6" podUID="e095072c-bd72-49b4-8cba-b49c94e40bf9" Sep 06 12:18:44.648118 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:44.648079 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxpp" Sep 06 12:18:44.648466 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:44.648166 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxpp" podUID="24ab4b74-f789-4642-adb7-a66bb6b75bb2" Sep 06 12:18:44.648466 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:44.648252 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlhb5" Sep 06 12:18:44.648466 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:44.648340 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlhb5" podUID="e8d626d1-3961-441f-829f-2c8a637e63f7" Sep 06 12:18:45.646804 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:45.646774 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:45.646971 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:45.646865 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6lx6" podUID="e095072c-bd72-49b4-8cba-b49c94e40bf9" Sep 06 12:18:46.470565 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:46.470381 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m6lx6"] Sep 06 12:18:46.471048 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:46.470631 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:46.471048 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:46.470717 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6lx6" podUID="e095072c-bd72-49b4-8cba-b49c94e40bf9" Sep 06 12:18:46.473270 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:46.473246 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wlhb5"] Sep 06 12:18:46.473387 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:46.473357 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlhb5" Sep 06 12:18:46.473621 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:46.473465 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlhb5" podUID="e8d626d1-3961-441f-829f-2c8a637e63f7" Sep 06 12:18:46.473954 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:46.473937 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gpxpp"] Sep 06 12:18:46.474052 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:46.474027 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxpp" Sep 06 12:18:46.474150 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:46.474125 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxpp" podUID="24ab4b74-f789-4642-adb7-a66bb6b75bb2" Sep 06 12:18:47.646712 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:47.646687 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlhb5" Sep 06 12:18:47.647118 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:47.646810 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlhb5" podUID="e8d626d1-3961-441f-829f-2c8a637e63f7" Sep 06 12:18:48.650563 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:48.650529 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:48.650978 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:48.650665 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6lx6" podUID="e095072c-bd72-49b4-8cba-b49c94e40bf9" Sep 06 12:18:48.650978 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:48.650702 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxpp" Sep 06 12:18:48.650978 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:48.650797 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxpp" podUID="24ab4b74-f789-4642-adb7-a66bb6b75bb2" Sep 06 12:18:49.646862 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:49.646831 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlhb5" Sep 06 12:18:49.647044 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:49.646950 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlhb5" podUID="e8d626d1-3961-441f-829f-2c8a637e63f7" Sep 06 12:18:50.646856 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:50.646830 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:50.646856 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:50.646845 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxpp" Sep 06 12:18:50.647339 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:50.646949 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6lx6" podUID="e095072c-bd72-49b4-8cba-b49c94e40bf9" Sep 06 12:18:50.647339 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:18:50.647065 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxpp" podUID="24ab4b74-f789-4642-adb7-a66bb6b75bb2" Sep 06 12:18:51.415865 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.415837 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-159.ec2.internal" event="NodeReady" Sep 06 12:18:51.416004 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.415949 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 06 12:18:51.458078 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.458046 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-l9sr9"] Sep 06 12:18:51.482194 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.482175 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xdrwd"] Sep 06 12:18:51.482307 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.482292 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l9sr9" Sep 06 12:18:51.484501 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.484473 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 06 12:18:51.484655 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.484541 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 06 12:18:51.484655 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.484548 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 06 12:18:51.484655 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.484560 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-spt24\"" Sep 06 12:18:51.484655 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.484554 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 06 12:18:51.496278 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.496202 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l9sr9"] Sep 06 12:18:51.496278 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.496256 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xdrwd"] Sep 06 12:18:51.496413 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.496323 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xdrwd" Sep 06 12:18:51.498706 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.498683 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 06 12:18:51.498864 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.498846 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 06 12:18:51.498930 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.498909 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nkgs4\"" Sep 06 12:18:51.498986 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.498911 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 06 12:18:51.559076 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.559053 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-c2mh4"] Sep 06 12:18:51.580458 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.580388 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c2mh4"] Sep 06 12:18:51.580458 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.580446 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c2mh4" Sep 06 12:18:51.582468 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.582453 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 06 12:18:51.582654 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.582635 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-x2nk9\"" Sep 06 12:18:51.582725 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.582661 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 06 12:18:51.585591 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.585439 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2ftf\" (UniqueName: \"kubernetes.io/projected/ccbb27a1-69be-4e30-9eff-95cd7280e7c1-kube-api-access-x2ftf\") pod \"ingress-canary-xdrwd\" (UID: \"ccbb27a1-69be-4e30-9eff-95cd7280e7c1\") " pod="openshift-ingress-canary/ingress-canary-xdrwd" Sep 06 12:18:51.585591 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.585468 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ccbb27a1-69be-4e30-9eff-95cd7280e7c1-cert\") pod \"ingress-canary-xdrwd\" (UID: \"ccbb27a1-69be-4e30-9eff-95cd7280e7c1\") " pod="openshift-ingress-canary/ingress-canary-xdrwd" Sep 06 12:18:51.585591 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.585536 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ccd2b1bb-e49b-4450-92dd-8c7d97a6ebe2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l9sr9\" (UID: \"ccd2b1bb-e49b-4450-92dd-8c7d97a6ebe2\") " pod="openshift-insights/insights-runtime-extractor-l9sr9" Sep 06 12:18:51.585775 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.585606 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ccd2b1bb-e49b-4450-92dd-8c7d97a6ebe2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l9sr9\" (UID: \"ccd2b1bb-e49b-4450-92dd-8c7d97a6ebe2\") " pod="openshift-insights/insights-runtime-extractor-l9sr9" Sep 06 12:18:51.585775 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.585630 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ccd2b1bb-e49b-4450-92dd-8c7d97a6ebe2-crio-socket\") pod \"insights-runtime-extractor-l9sr9\" (UID: \"ccd2b1bb-e49b-4450-92dd-8c7d97a6ebe2\") " pod="openshift-insights/insights-runtime-extractor-l9sr9" Sep 06 12:18:51.585775 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.585655 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79xgs\" (UniqueName: \"kubernetes.io/projected/ccd2b1bb-e49b-4450-92dd-8c7d97a6ebe2-kube-api-access-79xgs\") pod \"insights-runtime-extractor-l9sr9\" (UID: \"ccd2b1bb-e49b-4450-92dd-8c7d97a6ebe2\") " pod="openshift-insights/insights-runtime-extractor-l9sr9" Sep 06 12:18:51.585775 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.585678 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ccd2b1bb-e49b-4450-92dd-8c7d97a6ebe2-data-volume\") pod \"insights-runtime-extractor-l9sr9\" (UID: \"ccd2b1bb-e49b-4450-92dd-8c7d97a6ebe2\") " pod="openshift-insights/insights-runtime-extractor-l9sr9" Sep 06 12:18:51.647028 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.647010 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlhb5" Sep 06 12:18:51.650721 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.650701 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qrr2p\"" Sep 06 12:18:51.650833 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.650789 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 06 12:18:51.686706 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.686654 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ccd2b1bb-e49b-4450-92dd-8c7d97a6ebe2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l9sr9\" (UID: \"ccd2b1bb-e49b-4450-92dd-8c7d97a6ebe2\") " pod="openshift-insights/insights-runtime-extractor-l9sr9" Sep 06 12:18:51.686706 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.686683 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ccd2b1bb-e49b-4450-92dd-8c7d97a6ebe2-crio-socket\") pod \"insights-runtime-extractor-l9sr9\" (UID: \"ccd2b1bb-e49b-4450-92dd-8c7d97a6ebe2\") " pod="openshift-insights/insights-runtime-extractor-l9sr9" Sep 06 12:18:51.686706 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.686701 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79xgs\" (UniqueName: \"kubernetes.io/projected/ccd2b1bb-e49b-4450-92dd-8c7d97a6ebe2-kube-api-access-79xgs\") pod \"insights-runtime-extractor-l9sr9\" (UID: \"ccd2b1bb-e49b-4450-92dd-8c7d97a6ebe2\") " pod="openshift-insights/insights-runtime-extractor-l9sr9" Sep 06 12:18:51.686906 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.686717 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ccd2b1bb-e49b-4450-92dd-8c7d97a6ebe2-data-volume\") pod \"insights-runtime-extractor-l9sr9\" (UID: \"ccd2b1bb-e49b-4450-92dd-8c7d97a6ebe2\") " pod="openshift-insights/insights-runtime-extractor-l9sr9" Sep 06 12:18:51.686906 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.686739 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2ftf\" (UniqueName: \"kubernetes.io/projected/ccbb27a1-69be-4e30-9eff-95cd7280e7c1-kube-api-access-x2ftf\") pod \"ingress-canary-xdrwd\" (UID: \"ccbb27a1-69be-4e30-9eff-95cd7280e7c1\") " pod="openshift-ingress-canary/ingress-canary-xdrwd" Sep 06 12:18:51.686906 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.686774 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5925c1bc-97af-42ac-a824-f6c78a448578-config-volume\") pod \"dns-default-c2mh4\" (UID: \"5925c1bc-97af-42ac-a824-f6c78a448578\") " pod="openshift-dns/dns-default-c2mh4" Sep 06 12:18:51.686906 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.686819 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ccbb27a1-69be-4e30-9eff-95cd7280e7c1-cert\") pod \"ingress-canary-xdrwd\" (UID: \"ccbb27a1-69be-4e30-9eff-95cd7280e7c1\") " pod="openshift-ingress-canary/ingress-canary-xdrwd" Sep 06 12:18:51.686906 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.686844 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfp9j\" (UniqueName: \"kubernetes.io/projected/5925c1bc-97af-42ac-a824-f6c78a448578-kube-api-access-dfp9j\") pod \"dns-default-c2mh4\" (UID: \"5925c1bc-97af-42ac-a824-f6c78a448578\") " pod="openshift-dns/dns-default-c2mh4" Sep 06 12:18:51.687125 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.686913 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ccd2b1bb-e49b-4450-92dd-8c7d97a6ebe2-crio-socket\") pod \"insights-runtime-extractor-l9sr9\" (UID: \"ccd2b1bb-e49b-4450-92dd-8c7d97a6ebe2\") " pod="openshift-insights/insights-runtime-extractor-l9sr9" Sep 06 12:18:51.687125 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.686949 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ccd2b1bb-e49b-4450-92dd-8c7d97a6ebe2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l9sr9\" (UID: \"ccd2b1bb-e49b-4450-92dd-8c7d97a6ebe2\") " pod="openshift-insights/insights-runtime-extractor-l9sr9" Sep 06 12:18:51.687125 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.687038 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ccd2b1bb-e49b-4450-92dd-8c7d97a6ebe2-data-volume\") pod \"insights-runtime-extractor-l9sr9\" (UID: \"ccd2b1bb-e49b-4450-92dd-8c7d97a6ebe2\") " pod="openshift-insights/insights-runtime-extractor-l9sr9" Sep 06 12:18:51.687125 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.687098 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5925c1bc-97af-42ac-a824-f6c78a448578-metrics-tls\") pod \"dns-default-c2mh4\" (UID: \"5925c1bc-97af-42ac-a824-f6c78a448578\") " pod="openshift-dns/dns-default-c2mh4" Sep 06 12:18:51.687266 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.687128 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5925c1bc-97af-42ac-a824-f6c78a448578-tmp-dir\") pod \"dns-default-c2mh4\" (UID: \"5925c1bc-97af-42ac-a824-f6c78a448578\") " pod="openshift-dns/dns-default-c2mh4" Sep 06 12:18:51.687475 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.687453 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ccd2b1bb-e49b-4450-92dd-8c7d97a6ebe2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l9sr9\" (UID: \"ccd2b1bb-e49b-4450-92dd-8c7d97a6ebe2\") " pod="openshift-insights/insights-runtime-extractor-l9sr9" Sep 06 12:18:51.690673 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.690651 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ccd2b1bb-e49b-4450-92dd-8c7d97a6ebe2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l9sr9\" (UID: \"ccd2b1bb-e49b-4450-92dd-8c7d97a6ebe2\") " pod="openshift-insights/insights-runtime-extractor-l9sr9" Sep 06 12:18:51.690931 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.690912 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ccbb27a1-69be-4e30-9eff-95cd7280e7c1-cert\") pod \"ingress-canary-xdrwd\" (UID: \"ccbb27a1-69be-4e30-9eff-95cd7280e7c1\") " pod="openshift-ingress-canary/ingress-canary-xdrwd" Sep 06 12:18:51.694078 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.694057 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79xgs\" (UniqueName: \"kubernetes.io/projected/ccd2b1bb-e49b-4450-92dd-8c7d97a6ebe2-kube-api-access-79xgs\") pod \"insights-runtime-extractor-l9sr9\" (UID: \"ccd2b1bb-e49b-4450-92dd-8c7d97a6ebe2\") " pod="openshift-insights/insights-runtime-extractor-l9sr9" Sep 06 12:18:51.694243 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.694221 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2ftf\" (UniqueName: \"kubernetes.io/projected/ccbb27a1-69be-4e30-9eff-95cd7280e7c1-kube-api-access-x2ftf\") pod \"ingress-canary-xdrwd\" (UID: \"ccbb27a1-69be-4e30-9eff-95cd7280e7c1\") " pod="openshift-ingress-canary/ingress-canary-xdrwd" Sep 06 12:18:51.788266 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.788244 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5925c1bc-97af-42ac-a824-f6c78a448578-metrics-tls\") pod \"dns-default-c2mh4\" (UID: \"5925c1bc-97af-42ac-a824-f6c78a448578\") " pod="openshift-dns/dns-default-c2mh4" Sep 06 12:18:51.788361 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.788268 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5925c1bc-97af-42ac-a824-f6c78a448578-tmp-dir\") pod \"dns-default-c2mh4\" (UID: \"5925c1bc-97af-42ac-a824-f6c78a448578\") " pod="openshift-dns/dns-default-c2mh4" Sep 06 12:18:51.788361 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.788311 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5925c1bc-97af-42ac-a824-f6c78a448578-config-volume\") pod \"dns-default-c2mh4\" (UID: \"5925c1bc-97af-42ac-a824-f6c78a448578\") " pod="openshift-dns/dns-default-c2mh4" Sep 06 12:18:51.788478 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.788357 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfp9j\" (UniqueName: \"kubernetes.io/projected/5925c1bc-97af-42ac-a824-f6c78a448578-kube-api-access-dfp9j\") pod \"dns-default-c2mh4\" (UID: \"5925c1bc-97af-42ac-a824-f6c78a448578\") " pod="openshift-dns/dns-default-c2mh4" Sep 06 12:18:51.788605 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.788585 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5925c1bc-97af-42ac-a824-f6c78a448578-tmp-dir\") pod \"dns-default-c2mh4\" (UID: \"5925c1bc-97af-42ac-a824-f6c78a448578\") " pod="openshift-dns/dns-default-c2mh4" Sep 06 12:18:51.788720 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.788704 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5925c1bc-97af-42ac-a824-f6c78a448578-config-volume\") pod \"dns-default-c2mh4\" (UID: \"5925c1bc-97af-42ac-a824-f6c78a448578\") " pod="openshift-dns/dns-default-c2mh4" Sep 06 12:18:51.790416 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.790393 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5925c1bc-97af-42ac-a824-f6c78a448578-metrics-tls\") pod \"dns-default-c2mh4\" (UID: \"5925c1bc-97af-42ac-a824-f6c78a448578\") " pod="openshift-dns/dns-default-c2mh4" Sep 06 12:18:51.791303 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.791287 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l9sr9" Sep 06 12:18:51.795430 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.795411 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfp9j\" (UniqueName: \"kubernetes.io/projected/5925c1bc-97af-42ac-a824-f6c78a448578-kube-api-access-dfp9j\") pod \"dns-default-c2mh4\" (UID: \"5925c1bc-97af-42ac-a824-f6c78a448578\") " pod="openshift-dns/dns-default-c2mh4" Sep 06 12:18:51.804336 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.804316 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xdrwd" Sep 06 12:18:51.888185 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:51.888152 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c2mh4" Sep 06 12:18:52.125286 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:52.124459 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xdrwd"] Sep 06 12:18:52.126874 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:52.126853 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l9sr9"] Sep 06 12:18:52.128256 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:52.128232 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccbb27a1_69be_4e30_9eff_95cd7280e7c1.slice/crio-554ab70c94292688688b7ee872eba674eaa0244d76c5ba349ea8c3de70254dc1 WatchSource:0}: Error finding container 554ab70c94292688688b7ee872eba674eaa0244d76c5ba349ea8c3de70254dc1: Status 404 returned error can't find the container with id 554ab70c94292688688b7ee872eba674eaa0244d76c5ba349ea8c3de70254dc1 Sep 06 12:18:52.129716 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:52.129688 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccd2b1bb_e49b_4450_92dd_8c7d97a6ebe2.slice/crio-15807716ce2adc88a77ca1fac0e4c71ace806cf8f89beb20498a4adbcc4c1bbf WatchSource:0}: Error finding container 15807716ce2adc88a77ca1fac0e4c71ace806cf8f89beb20498a4adbcc4c1bbf: Status 404 returned error can't find the container with id 15807716ce2adc88a77ca1fac0e4c71ace806cf8f89beb20498a4adbcc4c1bbf Sep 06 12:18:52.175148 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:52.175113 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c2mh4"] Sep 06 12:18:52.178984 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:52.178964 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5925c1bc_97af_42ac_a824_f6c78a448578.slice/crio-8ec9015523acb9c595554d51fdbed1e54c9141bbc59ee97cb17d42b086d600e8 WatchSource:0}: Error finding container 8ec9015523acb9c595554d51fdbed1e54c9141bbc59ee97cb17d42b086d600e8: Status 404 returned error can't find the container with id 8ec9015523acb9c595554d51fdbed1e54c9141bbc59ee97cb17d42b086d600e8 Sep 06 12:18:52.647473 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:52.647443 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:18:52.648019 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:52.647455 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxpp" Sep 06 12:18:52.650422 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:52.650402 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 06 12:18:52.651349 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:52.651192 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 06 12:18:52.651349 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:52.651202 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vrsdl\"" Sep 06 12:18:52.651510 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:52.651446 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 06 12:18:52.883018 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:52.882972 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xdrwd" event={"ID":"ccbb27a1-69be-4e30-9eff-95cd7280e7c1","Type":"ContainerStarted","Data":"554ab70c94292688688b7ee872eba674eaa0244d76c5ba349ea8c3de70254dc1"} Sep 06 12:18:52.884978 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:52.884952 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l9sr9" event={"ID":"ccd2b1bb-e49b-4450-92dd-8c7d97a6ebe2","Type":"ContainerStarted","Data":"e823bbc041a47f5e7cd3630a51ea425dc843e2e767bdd5579355afc34ff1a689"} Sep 06 12:18:52.885101 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:52.885001 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l9sr9" event={"ID":"ccd2b1bb-e49b-4450-92dd-8c7d97a6ebe2","Type":"ContainerStarted","Data":"15807716ce2adc88a77ca1fac0e4c71ace806cf8f89beb20498a4adbcc4c1bbf"} Sep 06 12:18:52.886439 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:52.886413 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c2mh4" event={"ID":"5925c1bc-97af-42ac-a824-f6c78a448578","Type":"ContainerStarted","Data":"8ec9015523acb9c595554d51fdbed1e54c9141bbc59ee97cb17d42b086d600e8"} Sep 06 12:18:53.141512 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:53.141477 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55ff767878-tg6ft"] Sep 06 12:18:53.161990 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:53.161956 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55ff767878-tg6ft"] Sep 06 12:18:53.162143 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:53.162053 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55ff767878-tg6ft" Sep 06 12:18:53.165450 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:53.165431 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 06 12:18:53.167066 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:53.166338 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 06 12:18:53.167066 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:53.166386 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-cncnp\"" Sep 06 12:18:53.167066 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:53.166546 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 06 12:18:53.167066 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:53.166596 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 06 12:18:53.167066 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:53.166806 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 06 12:18:53.167066 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:53.167001 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 06 12:18:53.167404 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:53.167310 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 06 12:18:53.200126 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:53.200099 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b8df13-45dd-4577-af01-5dda0cfab94f-service-ca\") pod \"console-55ff767878-tg6ft\" (UID: \"b1b8df13-45dd-4577-af01-5dda0cfab94f\") " pod="openshift-console/console-55ff767878-tg6ft" Sep 06 12:18:53.200313 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:53.200147 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v25km\" (UniqueName: \"kubernetes.io/projected/b1b8df13-45dd-4577-af01-5dda0cfab94f-kube-api-access-v25km\") pod \"console-55ff767878-tg6ft\" (UID: \"b1b8df13-45dd-4577-af01-5dda0cfab94f\") " pod="openshift-console/console-55ff767878-tg6ft" Sep 06 12:18:53.200313 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:53.200241 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1b8df13-45dd-4577-af01-5dda0cfab94f-console-serving-cert\") pod \"console-55ff767878-tg6ft\" (UID: \"b1b8df13-45dd-4577-af01-5dda0cfab94f\") " pod="openshift-console/console-55ff767878-tg6ft" Sep 06 12:18:53.200313 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:53.200307 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b1b8df13-45dd-4577-af01-5dda0cfab94f-console-oauth-config\") pod \"console-55ff767878-tg6ft\" (UID: \"b1b8df13-45dd-4577-af01-5dda0cfab94f\") " pod="openshift-console/console-55ff767878-tg6ft" Sep 06 12:18:53.200551 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:53.200353 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b1b8df13-45dd-4577-af01-5dda0cfab94f-console-config\") pod \"console-55ff767878-tg6ft\" (UID: \"b1b8df13-45dd-4577-af01-5dda0cfab94f\") " pod="openshift-console/console-55ff767878-tg6ft" Sep 06 12:18:53.200551 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:53.200381 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b1b8df13-45dd-4577-af01-5dda0cfab94f-oauth-serving-cert\") pod \"console-55ff767878-tg6ft\" (UID: \"b1b8df13-45dd-4577-af01-5dda0cfab94f\") " pod="openshift-console/console-55ff767878-tg6ft" Sep 06 12:18:53.301077 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:53.301041 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b1b8df13-45dd-4577-af01-5dda0cfab94f-console-config\") pod \"console-55ff767878-tg6ft\" (UID: \"b1b8df13-45dd-4577-af01-5dda0cfab94f\") " pod="openshift-console/console-55ff767878-tg6ft" Sep 06 12:18:53.301077 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:53.301074 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b1b8df13-45dd-4577-af01-5dda0cfab94f-oauth-serving-cert\") pod \"console-55ff767878-tg6ft\" (UID: \"b1b8df13-45dd-4577-af01-5dda0cfab94f\") " pod="openshift-console/console-55ff767878-tg6ft" Sep 06 12:18:53.301283 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:53.301114 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b8df13-45dd-4577-af01-5dda0cfab94f-service-ca\") pod \"console-55ff767878-tg6ft\" (UID: \"b1b8df13-45dd-4577-af01-5dda0cfab94f\") " pod="openshift-console/console-55ff767878-tg6ft" Sep 06 12:18:53.301283 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:53.301143 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v25km\" (UniqueName: \"kubernetes.io/projected/b1b8df13-45dd-4577-af01-5dda0cfab94f-kube-api-access-v25km\") pod \"console-55ff767878-tg6ft\" (UID: \"b1b8df13-45dd-4577-af01-5dda0cfab94f\") " pod="openshift-console/console-55ff767878-tg6ft" Sep 06 12:18:53.301283 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:53.301189 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1b8df13-45dd-4577-af01-5dda0cfab94f-console-serving-cert\") pod \"console-55ff767878-tg6ft\" (UID: \"b1b8df13-45dd-4577-af01-5dda0cfab94f\") " pod="openshift-console/console-55ff767878-tg6ft" Sep 06 12:18:53.301283 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:53.301233 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b1b8df13-45dd-4577-af01-5dda0cfab94f-console-oauth-config\") pod \"console-55ff767878-tg6ft\" (UID: \"b1b8df13-45dd-4577-af01-5dda0cfab94f\") " pod="openshift-console/console-55ff767878-tg6ft" Sep 06 12:18:53.301742 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:53.301719 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b1b8df13-45dd-4577-af01-5dda0cfab94f-console-config\") pod \"console-55ff767878-tg6ft\" (UID: \"b1b8df13-45dd-4577-af01-5dda0cfab94f\") " pod="openshift-console/console-55ff767878-tg6ft" Sep 06 12:18:53.301870 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:53.301844 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b8df13-45dd-4577-af01-5dda0cfab94f-service-ca\") pod \"console-55ff767878-tg6ft\" (UID: \"b1b8df13-45dd-4577-af01-5dda0cfab94f\") " pod="openshift-console/console-55ff767878-tg6ft" Sep 06 12:18:53.301947 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:53.301882 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b1b8df13-45dd-4577-af01-5dda0cfab94f-oauth-serving-cert\") pod \"console-55ff767878-tg6ft\" (UID: \"b1b8df13-45dd-4577-af01-5dda0cfab94f\") " pod="openshift-console/console-55ff767878-tg6ft" Sep 06 12:18:53.305301 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:53.305271 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1b8df13-45dd-4577-af01-5dda0cfab94f-console-serving-cert\") pod \"console-55ff767878-tg6ft\" (UID: \"b1b8df13-45dd-4577-af01-5dda0cfab94f\") " pod="openshift-console/console-55ff767878-tg6ft" Sep 06 12:18:53.305627 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:53.305607 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b1b8df13-45dd-4577-af01-5dda0cfab94f-console-oauth-config\") pod \"console-55ff767878-tg6ft\" (UID: \"b1b8df13-45dd-4577-af01-5dda0cfab94f\") " pod="openshift-console/console-55ff767878-tg6ft" Sep 06 12:18:53.308888 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:53.308866 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v25km\" (UniqueName: \"kubernetes.io/projected/b1b8df13-45dd-4577-af01-5dda0cfab94f-kube-api-access-v25km\") pod \"console-55ff767878-tg6ft\" (UID: \"b1b8df13-45dd-4577-af01-5dda0cfab94f\") " pod="openshift-console/console-55ff767878-tg6ft" Sep 06 12:18:53.475334 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:53.475248 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55ff767878-tg6ft" Sep 06 12:18:54.235690 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:54.235507 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55ff767878-tg6ft"] Sep 06 12:18:54.894338 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:54.894299 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l9sr9" event={"ID":"ccd2b1bb-e49b-4450-92dd-8c7d97a6ebe2","Type":"ContainerStarted","Data":"717e5dea2b0a4244a77c38a587de65f376ac943e85cef8ec201beb26ac2e53b9"} Sep 06 12:18:55.067089 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:55.067054 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b8df13_45dd_4577_af01_5dda0cfab94f.slice/crio-73ba8971546d57373b95580858385875e38f3dc7aa4763a30dba404de2eb997c WatchSource:0}: Error finding container 73ba8971546d57373b95580858385875e38f3dc7aa4763a30dba404de2eb997c: Status 404 returned error can't find the container with id 73ba8971546d57373b95580858385875e38f3dc7aa4763a30dba404de2eb997c Sep 06 12:18:55.491636 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:55.490783 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zrzx8"] Sep 06 12:18:55.507583 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:55.507557 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zrzx8"] Sep 06 12:18:55.507742 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:55.507646 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zrzx8" Sep 06 12:18:55.511138 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:55.511036 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 06 12:18:55.511381 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:55.511354 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-t8cvz\"" Sep 06 12:18:55.619940 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:55.619908 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8befde00-706b-42ac-9fa7-1bf421fe4783-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-zrzx8\" (UID: \"8befde00-706b-42ac-9fa7-1bf421fe4783\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zrzx8" Sep 06 12:18:55.724314 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:55.721322 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8befde00-706b-42ac-9fa7-1bf421fe4783-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-zrzx8\" (UID: \"8befde00-706b-42ac-9fa7-1bf421fe4783\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zrzx8" Sep 06 12:18:55.727876 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:55.727844 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8befde00-706b-42ac-9fa7-1bf421fe4783-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-zrzx8\" (UID: \"8befde00-706b-42ac-9fa7-1bf421fe4783\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zrzx8" Sep 06 12:18:55.821285 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:55.821256 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zrzx8" Sep 06 12:18:55.903459 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:55.903321 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c2mh4" event={"ID":"5925c1bc-97af-42ac-a824-f6c78a448578","Type":"ContainerStarted","Data":"970aa114152481216fde00a732315cc8776327bbc5a44decc41d372ed193727e"} Sep 06 12:18:55.903567 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:55.903493 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c2mh4" event={"ID":"5925c1bc-97af-42ac-a824-f6c78a448578","Type":"ContainerStarted","Data":"a72b2848b34368f8d988b0f37b6b910c7aa6c4b9a63ac080d27adf7d2ac9076c"} Sep 06 12:18:55.903964 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:55.903710 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-c2mh4" Sep 06 12:18:55.906360 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:55.905742 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xdrwd" event={"ID":"ccbb27a1-69be-4e30-9eff-95cd7280e7c1","Type":"ContainerStarted","Data":"59cd1ac10e2c1d4a95251c4b2538a7e6ad96f443599bf3cb2abc9e2ddaa8638a"} Sep 06 12:18:55.907681 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:55.907608 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55ff767878-tg6ft" event={"ID":"b1b8df13-45dd-4577-af01-5dda0cfab94f","Type":"ContainerStarted","Data":"73ba8971546d57373b95580858385875e38f3dc7aa4763a30dba404de2eb997c"} Sep 06 12:18:55.919328 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:55.919281 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-c2mh4" podStartSLOduration=2.001692863 podStartE2EDuration="4.919264783s" totalImagesPullingTime="2.91757192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:18:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:18:55.918200878 +0000 UTC m=+51.773958762" watchObservedRunningTime="2026-09-06 12:18:55.919264783 +0000 UTC m=+51.775022670" Sep 06 12:18:55.931194 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:55.931159 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xdrwd" podStartSLOduration=1.68843467 podStartE2EDuration="4.931146198s" totalImagesPullingTime="3.242711528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:18:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:18:55.93062585 +0000 UTC m=+51.786383738" watchObservedRunningTime="2026-09-06 12:18:55.931146198 +0000 UTC m=+51.786904085" Sep 06 12:18:56.120211 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:56.120168 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zrzx8"] Sep 06 12:18:56.506683 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:18:56.506653 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8befde00_706b_42ac_9fa7_1bf421fe4783.slice/crio-be3b5289ce5f6d25e918494ad8dedbc50d6d98a5c59e75312b6e828870dc132a WatchSource:0}: Error finding container be3b5289ce5f6d25e918494ad8dedbc50d6d98a5c59e75312b6e828870dc132a: Status 404 returned error can't find the container with id be3b5289ce5f6d25e918494ad8dedbc50d6d98a5c59e75312b6e828870dc132a Sep 06 12:18:56.914422 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:56.914361 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zrzx8" event={"ID":"8befde00-706b-42ac-9fa7-1bf421fe4783","Type":"ContainerStarted","Data":"be3b5289ce5f6d25e918494ad8dedbc50d6d98a5c59e75312b6e828870dc132a"} Sep 06 12:18:58.921718 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:58.921470 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l9sr9" event={"ID":"ccd2b1bb-e49b-4450-92dd-8c7d97a6ebe2","Type":"ContainerStarted","Data":"340a3cf5e253d3508307ac521f02ebaa11b357c4895450fb5bb42e0080f15b2c"} Sep 06 12:18:58.923244 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:58.923220 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55ff767878-tg6ft" event={"ID":"b1b8df13-45dd-4577-af01-5dda0cfab94f","Type":"ContainerStarted","Data":"25e6a11faa86b34996d9bb1062ca245b81a77ebb3208663d4373a9889ec3943d"} Sep 06 12:18:58.938026 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:58.937983 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-l9sr9" podStartSLOduration=2.419190595 podStartE2EDuration="7.937968485s" totalImagesPullingTime="5.51877789s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 12:18:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:18:58.937572957 +0000 UTC m=+54.793330844" watchObservedRunningTime="2026-09-06 12:18:58.937968485 +0000 UTC m=+54.793726373" Sep 06 12:18:58.951770 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:58.951719 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55ff767878-tg6ft" podStartSLOduration=2.9688138090000002 podStartE2EDuration="5.951705918s" totalImagesPullingTime="2.982892109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:18:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:18:58.951350676 +0000 UTC m=+54.807108573" watchObservedRunningTime="2026-09-06 12:18:58.951705918 +0000 UTC m=+54.807463805" Sep 06 12:18:59.929000 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:59.928950 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zrzx8" event={"ID":"8befde00-706b-42ac-9fa7-1bf421fe4783","Type":"ContainerStarted","Data":"139abfaa8e4cd624de25939af05039e5e4407fbb6627255169b782aa68cf9c2b"} Sep 06 12:18:59.943258 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:18:59.943208 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zrzx8" podStartSLOduration=2.417833928 podStartE2EDuration="4.943192041s" totalImagesPullingTime="2.525358113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:18:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:18:59.942402045 +0000 UTC m=+55.798159945" watchObservedRunningTime="2026-09-06 12:18:59.943192041 +0000 UTC m=+55.798949927" Sep 06 12:19:00.119280 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:00.119246 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8454d65567-kn6mk"] Sep 06 12:19:00.121217 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:00.121195 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8454d65567-kn6mk" Sep 06 12:19:00.132269 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:00.132006 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 06 12:19:00.134270 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:00.134248 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8454d65567-kn6mk"] Sep 06 12:19:00.154511 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:00.154487 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bcae8b57-e511-448f-975c-08772a4af285-console-oauth-config\") pod \"console-8454d65567-kn6mk\" (UID: \"bcae8b57-e511-448f-975c-08772a4af285\") " pod="openshift-console/console-8454d65567-kn6mk" Sep 06 12:19:00.154617 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:00.154526 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bcae8b57-e511-448f-975c-08772a4af285-console-serving-cert\") pod \"console-8454d65567-kn6mk\" (UID: \"bcae8b57-e511-448f-975c-08772a4af285\") " pod="openshift-console/console-8454d65567-kn6mk" Sep 06 12:19:00.154617 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:00.154554 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bcae8b57-e511-448f-975c-08772a4af285-oauth-serving-cert\") pod \"console-8454d65567-kn6mk\" (UID: \"bcae8b57-e511-448f-975c-08772a4af285\") " pod="openshift-console/console-8454d65567-kn6mk" Sep 06 12:19:00.154617 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:00.154607 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcae8b57-e511-448f-975c-08772a4af285-trusted-ca-bundle\") pod \"console-8454d65567-kn6mk\" (UID: \"bcae8b57-e511-448f-975c-08772a4af285\") " pod="openshift-console/console-8454d65567-kn6mk" Sep 06 12:19:00.154770 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:00.154638 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bcae8b57-e511-448f-975c-08772a4af285-console-config\") pod \"console-8454d65567-kn6mk\" (UID: \"bcae8b57-e511-448f-975c-08772a4af285\") " pod="openshift-console/console-8454d65567-kn6mk" Sep 06 12:19:00.154770 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:00.154695 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bcae8b57-e511-448f-975c-08772a4af285-service-ca\") pod \"console-8454d65567-kn6mk\" (UID: \"bcae8b57-e511-448f-975c-08772a4af285\") " pod="openshift-console/console-8454d65567-kn6mk" Sep 06 12:19:00.154770 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:00.154710 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tw5x\" (UniqueName: \"kubernetes.io/projected/bcae8b57-e511-448f-975c-08772a4af285-kube-api-access-7tw5x\") pod \"console-8454d65567-kn6mk\" (UID: \"bcae8b57-e511-448f-975c-08772a4af285\") " pod="openshift-console/console-8454d65567-kn6mk" Sep 06 12:19:00.255055 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:00.254997 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bcae8b57-e511-448f-975c-08772a4af285-console-serving-cert\") pod \"console-8454d65567-kn6mk\" (UID: \"bcae8b57-e511-448f-975c-08772a4af285\") " pod="openshift-console/console-8454d65567-kn6mk" Sep 06 12:19:00.255055 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:00.255026 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bcae8b57-e511-448f-975c-08772a4af285-oauth-serving-cert\") pod \"console-8454d65567-kn6mk\" (UID: \"bcae8b57-e511-448f-975c-08772a4af285\") " pod="openshift-console/console-8454d65567-kn6mk" Sep 06 12:19:00.255055 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:00.255043 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcae8b57-e511-448f-975c-08772a4af285-trusted-ca-bundle\") pod \"console-8454d65567-kn6mk\" (UID: \"bcae8b57-e511-448f-975c-08772a4af285\") " pod="openshift-console/console-8454d65567-kn6mk" Sep 06 12:19:00.255271 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:00.255059 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bcae8b57-e511-448f-975c-08772a4af285-console-config\") pod \"console-8454d65567-kn6mk\" (UID: \"bcae8b57-e511-448f-975c-08772a4af285\") " pod="openshift-console/console-8454d65567-kn6mk" Sep 06 12:19:00.255343 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:00.255320 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bcae8b57-e511-448f-975c-08772a4af285-service-ca\") pod \"console-8454d65567-kn6mk\" (UID: \"bcae8b57-e511-448f-975c-08772a4af285\") " pod="openshift-console/console-8454d65567-kn6mk" Sep 06 12:19:00.255396 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:00.255354 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tw5x\" (UniqueName: \"kubernetes.io/projected/bcae8b57-e511-448f-975c-08772a4af285-kube-api-access-7tw5x\") pod \"console-8454d65567-kn6mk\" (UID: \"bcae8b57-e511-448f-975c-08772a4af285\") " pod="openshift-console/console-8454d65567-kn6mk" Sep 06 12:19:00.255451 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:00.255407 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bcae8b57-e511-448f-975c-08772a4af285-console-oauth-config\") pod \"console-8454d65567-kn6mk\" (UID: \"bcae8b57-e511-448f-975c-08772a4af285\") " pod="openshift-console/console-8454d65567-kn6mk" Sep 06 12:19:00.255792 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:00.255738 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bcae8b57-e511-448f-975c-08772a4af285-console-config\") pod \"console-8454d65567-kn6mk\" (UID: \"bcae8b57-e511-448f-975c-08772a4af285\") " pod="openshift-console/console-8454d65567-kn6mk" Sep 06 12:19:00.255899 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:00.255789 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bcae8b57-e511-448f-975c-08772a4af285-oauth-serving-cert\") pod \"console-8454d65567-kn6mk\" (UID: \"bcae8b57-e511-448f-975c-08772a4af285\") " pod="openshift-console/console-8454d65567-kn6mk" Sep 06 12:19:00.256012 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:00.255990 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bcae8b57-e511-448f-975c-08772a4af285-service-ca\") pod \"console-8454d65567-kn6mk\" (UID: \"bcae8b57-e511-448f-975c-08772a4af285\") " pod="openshift-console/console-8454d65567-kn6mk" Sep 06 12:19:00.256148 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:00.256127 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcae8b57-e511-448f-975c-08772a4af285-trusted-ca-bundle\") pod \"console-8454d65567-kn6mk\" (UID: \"bcae8b57-e511-448f-975c-08772a4af285\") " pod="openshift-console/console-8454d65567-kn6mk" Sep 06 12:19:00.259686 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:00.259658 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bcae8b57-e511-448f-975c-08772a4af285-console-oauth-config\") pod \"console-8454d65567-kn6mk\" (UID: \"bcae8b57-e511-448f-975c-08772a4af285\") " pod="openshift-console/console-8454d65567-kn6mk" Sep 06 12:19:00.259804 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:00.259707 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bcae8b57-e511-448f-975c-08772a4af285-console-serving-cert\") pod \"console-8454d65567-kn6mk\" (UID: \"bcae8b57-e511-448f-975c-08772a4af285\") " pod="openshift-console/console-8454d65567-kn6mk" Sep 06 12:19:00.262322 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:00.262297 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tw5x\" (UniqueName: \"kubernetes.io/projected/bcae8b57-e511-448f-975c-08772a4af285-kube-api-access-7tw5x\") pod \"console-8454d65567-kn6mk\" (UID: \"bcae8b57-e511-448f-975c-08772a4af285\") " pod="openshift-console/console-8454d65567-kn6mk" Sep 06 12:19:00.435193 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:00.435152 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8454d65567-kn6mk" Sep 06 12:19:00.705860 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:00.705833 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8454d65567-kn6mk"] Sep 06 12:19:00.708658 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:19:00.708633 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcae8b57_e511_448f_975c_08772a4af285.slice/crio-b7be584a3b985072a51b723a4c17951286d5851c17448cc0f3eedd6196e3319a WatchSource:0}: Error finding container b7be584a3b985072a51b723a4c17951286d5851c17448cc0f3eedd6196e3319a: Status 404 returned error can't find the container with id b7be584a3b985072a51b723a4c17951286d5851c17448cc0f3eedd6196e3319a Sep 06 12:19:00.934716 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:00.934679 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8454d65567-kn6mk" event={"ID":"bcae8b57-e511-448f-975c-08772a4af285","Type":"ContainerStarted","Data":"1e7e9edff93c1f210ae9d66f3564a1c2971de569e3a1e7e34eeb699b38bd594a"} Sep 06 12:19:00.935137 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:00.934733 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8454d65567-kn6mk" event={"ID":"bcae8b57-e511-448f-975c-08772a4af285","Type":"ContainerStarted","Data":"b7be584a3b985072a51b723a4c17951286d5851c17448cc0f3eedd6196e3319a"} Sep 06 12:19:00.935137 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:00.934870 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zrzx8" Sep 06 12:19:00.939631 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:00.939611 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zrzx8" Sep 06 12:19:00.949941 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:00.949896 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8454d65567-kn6mk" podStartSLOduration=0.949879836 podStartE2EDuration="949.879836ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:19:00.949858866 +0000 UTC m=+56.805616778" watchObservedRunningTime="2026-09-06 12:19:00.949879836 +0000 UTC m=+56.805637702" Sep 06 12:19:01.548537 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:01.548505 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-xdf8m"] Sep 06 12:19:01.550785 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:01.550768 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-xdf8m" Sep 06 12:19:01.552935 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:01.552903 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 06 12:19:01.553243 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:01.553224 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 06 12:19:01.553956 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:01.553935 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 06 12:19:01.554013 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:01.553935 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-5sh5p\"" Sep 06 12:19:01.554270 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:01.554254 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 06 12:19:01.554333 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:01.554301 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 06 12:19:01.559529 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:01.559500 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-xdf8m"] Sep 06 12:19:01.664099 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:01.664066 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e34034de-85ae-4d49-8e58-ea01c00a894d-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-xdf8m\" (UID: \"e34034de-85ae-4d49-8e58-ea01c00a894d\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xdf8m" Sep 06 12:19:01.664213 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:01.664138 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e34034de-85ae-4d49-8e58-ea01c00a894d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-xdf8m\" (UID: \"e34034de-85ae-4d49-8e58-ea01c00a894d\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xdf8m" Sep 06 12:19:01.664213 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:01.664196 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e34034de-85ae-4d49-8e58-ea01c00a894d-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-xdf8m\" (UID: \"e34034de-85ae-4d49-8e58-ea01c00a894d\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xdf8m" Sep 06 12:19:01.664303 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:01.664269 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l58dd\" (UniqueName: \"kubernetes.io/projected/e34034de-85ae-4d49-8e58-ea01c00a894d-kube-api-access-l58dd\") pod \"prometheus-operator-6f4946db56-xdf8m\" (UID: \"e34034de-85ae-4d49-8e58-ea01c00a894d\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xdf8m" Sep 06 12:19:01.765562 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:01.765533 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e34034de-85ae-4d49-8e58-ea01c00a894d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-xdf8m\" (UID: \"e34034de-85ae-4d49-8e58-ea01c00a894d\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xdf8m" Sep 06 12:19:01.765677 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:01.765571 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e34034de-85ae-4d49-8e58-ea01c00a894d-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-xdf8m\" (UID: \"e34034de-85ae-4d49-8e58-ea01c00a894d\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xdf8m" Sep 06 12:19:01.765677 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:01.765609 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l58dd\" (UniqueName: \"kubernetes.io/projected/e34034de-85ae-4d49-8e58-ea01c00a894d-kube-api-access-l58dd\") pod \"prometheus-operator-6f4946db56-xdf8m\" (UID: \"e34034de-85ae-4d49-8e58-ea01c00a894d\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xdf8m" Sep 06 12:19:01.765677 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:01.765627 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e34034de-85ae-4d49-8e58-ea01c00a894d-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-xdf8m\" (UID: \"e34034de-85ae-4d49-8e58-ea01c00a894d\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xdf8m" Sep 06 12:19:01.766386 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:01.766358 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e34034de-85ae-4d49-8e58-ea01c00a894d-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-xdf8m\" (UID: \"e34034de-85ae-4d49-8e58-ea01c00a894d\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xdf8m" Sep 06 12:19:01.768363 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:01.768329 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e34034de-85ae-4d49-8e58-ea01c00a894d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-xdf8m\" (UID: \"e34034de-85ae-4d49-8e58-ea01c00a894d\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xdf8m" Sep 06 12:19:01.768651 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:01.768626 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e34034de-85ae-4d49-8e58-ea01c00a894d-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-xdf8m\" (UID: \"e34034de-85ae-4d49-8e58-ea01c00a894d\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xdf8m" Sep 06 12:19:01.772934 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:01.772907 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l58dd\" (UniqueName: \"kubernetes.io/projected/e34034de-85ae-4d49-8e58-ea01c00a894d-kube-api-access-l58dd\") pod \"prometheus-operator-6f4946db56-xdf8m\" (UID: \"e34034de-85ae-4d49-8e58-ea01c00a894d\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xdf8m" Sep 06 12:19:01.860685 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:01.860616 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-xdf8m" Sep 06 12:19:02.127883 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:02.127813 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-xdf8m"] Sep 06 12:19:02.131037 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:19:02.131012 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode34034de_85ae_4d49_8e58_ea01c00a894d.slice/crio-1dbef1de106a1e9a51eee4389dd3cc55b43d42100a80ef6ec8f988c20c67a882 WatchSource:0}: Error finding container 1dbef1de106a1e9a51eee4389dd3cc55b43d42100a80ef6ec8f988c20c67a882: Status 404 returned error can't find the container with id 1dbef1de106a1e9a51eee4389dd3cc55b43d42100a80ef6ec8f988c20c67a882 Sep 06 12:19:02.943699 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:02.943660 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-xdf8m" event={"ID":"e34034de-85ae-4d49-8e58-ea01c00a894d","Type":"ContainerStarted","Data":"1dbef1de106a1e9a51eee4389dd3cc55b43d42100a80ef6ec8f988c20c67a882"} Sep 06 12:19:03.476245 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:03.476215 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55ff767878-tg6ft" Sep 06 12:19:03.476550 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:03.476298 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55ff767878-tg6ft" Sep 06 12:19:03.477456 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:03.477425 2576 patch_prober.go:28] interesting pod/console-55ff767878-tg6ft container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.9:8443/health\": dial tcp 10.134.0.9:8443: connect: connection refused" start-of-body= Sep 06 12:19:03.477552 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:03.477474 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-55ff767878-tg6ft" podUID="b1b8df13-45dd-4577-af01-5dda0cfab94f" containerName="console" probeResult="failure" output="Get \"https://10.134.0.9:8443/health\": dial tcp 10.134.0.9:8443: connect: connection refused" Sep 06 12:19:03.949302 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:03.949262 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-xdf8m" event={"ID":"e34034de-85ae-4d49-8e58-ea01c00a894d","Type":"ContainerStarted","Data":"665b2853c3ee179d43647ed43bd3e9fb9558fa98e26ae504a329465a486c7ac7"} Sep 06 12:19:03.949577 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:03.949319 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-xdf8m" event={"ID":"e34034de-85ae-4d49-8e58-ea01c00a894d","Type":"ContainerStarted","Data":"a44dde05676014ca7e961a103760d8473868cd99f685aa4ab8e9e2e34f6b3c1c"} Sep 06 12:19:03.966101 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:03.966054 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-xdf8m" podStartSLOduration=1.7939846529999999 podStartE2EDuration="2.966033948s" totalImagesPullingTime="1.172049295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:19:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:19:03.964239142 +0000 UTC m=+59.819997028" watchObservedRunningTime="2026-09-06 12:19:03.966033948 +0000 UTC m=+59.821791847" Sep 06 12:19:04.838317 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:04.838280 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-98vz4" Sep 06 12:19:05.892705 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:05.892675 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-5ddg8"] Sep 06 12:19:05.895390 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:05.895373 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5ddg8" Sep 06 12:19:05.897921 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:05.897900 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 06 12:19:05.898109 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:05.898086 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-pqpnq\"" Sep 06 12:19:05.898184 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:05.898114 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 06 12:19:05.912642 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:05.912623 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-5ddg8"] Sep 06 12:19:05.917041 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:05.917021 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-c2mh4" Sep 06 12:19:05.943032 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:05.943012 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-bf74d"] Sep 06 12:19:05.944844 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:05.944826 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-bf74d" Sep 06 12:19:05.947115 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:05.947093 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 06 12:19:05.947115 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:05.947107 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-hx4nj\"" Sep 06 12:19:05.947115 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:05.947111 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 06 12:19:05.947299 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:05.947216 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 06 12:19:05.958043 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:05.958022 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-n825g"] Sep 06 12:19:05.959939 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:05.959911 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n825g" Sep 06 12:19:05.961466 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:05.961441 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-bf74d"] Sep 06 12:19:05.961943 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:05.961920 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 06 12:19:05.962199 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:05.962181 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 06 12:19:05.962312 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:05.962187 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pmdxk\"" Sep 06 12:19:05.962407 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:05.962392 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 06 12:19:05.996006 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:05.995982 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/be7870d2-35ee-43d8-9b02-74ad8ee286b1-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-bf74d\" (UID: \"be7870d2-35ee-43d8-9b02-74ad8ee286b1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bf74d" Sep 06 12:19:05.996139 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:05.996018 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/212e2510-1c27-4fdf-ad5f-51b8b334106a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-5ddg8\" (UID: \"212e2510-1c27-4fdf-ad5f-51b8b334106a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5ddg8" Sep 06 12:19:05.996139 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:05.996055 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be7870d2-35ee-43d8-9b02-74ad8ee286b1-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-bf74d\" (UID: \"be7870d2-35ee-43d8-9b02-74ad8ee286b1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bf74d" Sep 06 12:19:05.996139 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:05.996081 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be7870d2-35ee-43d8-9b02-74ad8ee286b1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-bf74d\" (UID: \"be7870d2-35ee-43d8-9b02-74ad8ee286b1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bf74d" Sep 06 12:19:05.996139 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:05.996108 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-456g6\" (UniqueName: \"kubernetes.io/projected/be7870d2-35ee-43d8-9b02-74ad8ee286b1-kube-api-access-456g6\") pod \"kube-state-metrics-74cb57df7-bf74d\" (UID: \"be7870d2-35ee-43d8-9b02-74ad8ee286b1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bf74d" Sep 06 12:19:05.996357 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:05.996187 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/212e2510-1c27-4fdf-ad5f-51b8b334106a-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-5ddg8\" (UID: \"212e2510-1c27-4fdf-ad5f-51b8b334106a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5ddg8" Sep 06 12:19:05.996357 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:05.996261 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/be7870d2-35ee-43d8-9b02-74ad8ee286b1-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-bf74d\" (UID: \"be7870d2-35ee-43d8-9b02-74ad8ee286b1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bf74d" Sep 06 12:19:05.996357 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:05.996294 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/be7870d2-35ee-43d8-9b02-74ad8ee286b1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-bf74d\" (UID: \"be7870d2-35ee-43d8-9b02-74ad8ee286b1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bf74d" Sep 06 12:19:05.996357 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:05.996322 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz7z2\" (UniqueName: \"kubernetes.io/projected/212e2510-1c27-4fdf-ad5f-51b8b334106a-kube-api-access-qz7z2\") pod \"openshift-state-metrics-657fd6c54-5ddg8\" (UID: \"212e2510-1c27-4fdf-ad5f-51b8b334106a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5ddg8" Sep 06 12:19:05.996507 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:05.996383 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/212e2510-1c27-4fdf-ad5f-51b8b334106a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-5ddg8\" (UID: \"212e2510-1c27-4fdf-ad5f-51b8b334106a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5ddg8" Sep 06 12:19:06.097153 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.097129 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz7z2\" (UniqueName: \"kubernetes.io/projected/212e2510-1c27-4fdf-ad5f-51b8b334106a-kube-api-access-qz7z2\") pod \"openshift-state-metrics-657fd6c54-5ddg8\" (UID: \"212e2510-1c27-4fdf-ad5f-51b8b334106a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5ddg8" Sep 06 12:19:06.097281 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.097161 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/529c654b-f937-4604-8271-488f66663e84-sys\") pod \"node-exporter-n825g\" (UID: \"529c654b-f937-4604-8271-488f66663e84\") " pod="openshift-monitoring/node-exporter-n825g" Sep 06 12:19:06.097281 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.097192 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/212e2510-1c27-4fdf-ad5f-51b8b334106a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-5ddg8\" (UID: \"212e2510-1c27-4fdf-ad5f-51b8b334106a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5ddg8" Sep 06 12:19:06.097281 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.097217 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7snvp\" (UniqueName: \"kubernetes.io/projected/529c654b-f937-4604-8271-488f66663e84-kube-api-access-7snvp\") pod \"node-exporter-n825g\" (UID: \"529c654b-f937-4604-8271-488f66663e84\") " pod="openshift-monitoring/node-exporter-n825g" Sep 06 12:19:06.097445 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.097279 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/be7870d2-35ee-43d8-9b02-74ad8ee286b1-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-bf74d\" (UID: \"be7870d2-35ee-43d8-9b02-74ad8ee286b1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bf74d" Sep 06 12:19:06.097445 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:19:06.097298 2576 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Sep 06 12:19:06.097445 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.097314 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/529c654b-f937-4604-8271-488f66663e84-node-exporter-tls\") pod \"node-exporter-n825g\" (UID: \"529c654b-f937-4604-8271-488f66663e84\") " pod="openshift-monitoring/node-exporter-n825g" Sep 06 12:19:06.097445 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:19:06.097354 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/212e2510-1c27-4fdf-ad5f-51b8b334106a-openshift-state-metrics-tls podName:212e2510-1c27-4fdf-ad5f-51b8b334106a nodeName:}" failed. No retries permitted until 2026-09-06 12:19:06.597335964 +0000 UTC m=+62.453093835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/212e2510-1c27-4fdf-ad5f-51b8b334106a-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-5ddg8" (UID: "212e2510-1c27-4fdf-ad5f-51b8b334106a") : secret "openshift-state-metrics-tls" not found Sep 06 12:19:06.097445 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:19:06.097410 2576 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Sep 06 12:19:06.097445 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.097422 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/212e2510-1c27-4fdf-ad5f-51b8b334106a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-5ddg8\" (UID: \"212e2510-1c27-4fdf-ad5f-51b8b334106a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5ddg8" Sep 06 12:19:06.097717 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:19:06.097460 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be7870d2-35ee-43d8-9b02-74ad8ee286b1-kube-state-metrics-tls podName:be7870d2-35ee-43d8-9b02-74ad8ee286b1 nodeName:}" failed. No retries permitted until 2026-09-06 12:19:06.597445954 +0000 UTC m=+62.453203819 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/be7870d2-35ee-43d8-9b02-74ad8ee286b1-kube-state-metrics-tls") pod "kube-state-metrics-74cb57df7-bf74d" (UID: "be7870d2-35ee-43d8-9b02-74ad8ee286b1") : secret "kube-state-metrics-tls" not found Sep 06 12:19:06.097717 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.097493 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be7870d2-35ee-43d8-9b02-74ad8ee286b1-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-bf74d\" (UID: \"be7870d2-35ee-43d8-9b02-74ad8ee286b1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bf74d" Sep 06 12:19:06.097717 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.097515 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/529c654b-f937-4604-8271-488f66663e84-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n825g\" (UID: \"529c654b-f937-4604-8271-488f66663e84\") " pod="openshift-monitoring/node-exporter-n825g" Sep 06 12:19:06.097717 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.097538 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be7870d2-35ee-43d8-9b02-74ad8ee286b1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-bf74d\" (UID: \"be7870d2-35ee-43d8-9b02-74ad8ee286b1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bf74d" Sep 06 12:19:06.097717 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.097588 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-456g6\" (UniqueName: \"kubernetes.io/projected/be7870d2-35ee-43d8-9b02-74ad8ee286b1-kube-api-access-456g6\") pod \"kube-state-metrics-74cb57df7-bf74d\" (UID: \"be7870d2-35ee-43d8-9b02-74ad8ee286b1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bf74d" Sep 06 12:19:06.097717 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.097624 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/212e2510-1c27-4fdf-ad5f-51b8b334106a-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-5ddg8\" (UID: \"212e2510-1c27-4fdf-ad5f-51b8b334106a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5ddg8" Sep 06 12:19:06.097717 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.097656 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/529c654b-f937-4604-8271-488f66663e84-node-exporter-wtmp\") pod \"node-exporter-n825g\" (UID: \"529c654b-f937-4604-8271-488f66663e84\") " pod="openshift-monitoring/node-exporter-n825g" Sep 06 12:19:06.097717 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.097675 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/529c654b-f937-4604-8271-488f66663e84-node-exporter-accelerators-collector-config\") pod \"node-exporter-n825g\" (UID: \"529c654b-f937-4604-8271-488f66663e84\") " pod="openshift-monitoring/node-exporter-n825g" Sep 06 12:19:06.097717 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.097713 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/be7870d2-35ee-43d8-9b02-74ad8ee286b1-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-bf74d\" (UID: \"be7870d2-35ee-43d8-9b02-74ad8ee286b1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bf74d" Sep 06 12:19:06.098175 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.097744 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/be7870d2-35ee-43d8-9b02-74ad8ee286b1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-bf74d\" (UID: \"be7870d2-35ee-43d8-9b02-74ad8ee286b1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bf74d" Sep 06 12:19:06.098175 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.097796 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/529c654b-f937-4604-8271-488f66663e84-root\") pod \"node-exporter-n825g\" (UID: \"529c654b-f937-4604-8271-488f66663e84\") " pod="openshift-monitoring/node-exporter-n825g" Sep 06 12:19:06.098175 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.097820 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/529c654b-f937-4604-8271-488f66663e84-node-exporter-textfile\") pod \"node-exporter-n825g\" (UID: \"529c654b-f937-4604-8271-488f66663e84\") " pod="openshift-monitoring/node-exporter-n825g" Sep 06 12:19:06.098175 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.097845 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/529c654b-f937-4604-8271-488f66663e84-metrics-client-ca\") pod \"node-exporter-n825g\" (UID: \"529c654b-f937-4604-8271-488f66663e84\") " pod="openshift-monitoring/node-exporter-n825g" Sep 06 12:19:06.098375 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.098293 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/be7870d2-35ee-43d8-9b02-74ad8ee286b1-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-bf74d\" (UID: \"be7870d2-35ee-43d8-9b02-74ad8ee286b1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bf74d" Sep 06 12:19:06.098469 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.098449 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be7870d2-35ee-43d8-9b02-74ad8ee286b1-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-bf74d\" (UID: \"be7870d2-35ee-43d8-9b02-74ad8ee286b1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bf74d" Sep 06 12:19:06.098641 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.098622 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/212e2510-1c27-4fdf-ad5f-51b8b334106a-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-5ddg8\" (UID: \"212e2510-1c27-4fdf-ad5f-51b8b334106a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5ddg8" Sep 06 12:19:06.098689 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.098622 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/be7870d2-35ee-43d8-9b02-74ad8ee286b1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-bf74d\" (UID: \"be7870d2-35ee-43d8-9b02-74ad8ee286b1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bf74d" Sep 06 12:19:06.099977 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.099959 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/212e2510-1c27-4fdf-ad5f-51b8b334106a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-5ddg8\" (UID: \"212e2510-1c27-4fdf-ad5f-51b8b334106a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5ddg8" Sep 06 12:19:06.100328 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.100307 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be7870d2-35ee-43d8-9b02-74ad8ee286b1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-bf74d\" (UID: \"be7870d2-35ee-43d8-9b02-74ad8ee286b1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bf74d" Sep 06 12:19:06.106293 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.106271 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz7z2\" (UniqueName: \"kubernetes.io/projected/212e2510-1c27-4fdf-ad5f-51b8b334106a-kube-api-access-qz7z2\") pod \"openshift-state-metrics-657fd6c54-5ddg8\" (UID: \"212e2510-1c27-4fdf-ad5f-51b8b334106a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5ddg8" Sep 06 12:19:06.106794 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.106773 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-456g6\" (UniqueName: \"kubernetes.io/projected/be7870d2-35ee-43d8-9b02-74ad8ee286b1-kube-api-access-456g6\") pod \"kube-state-metrics-74cb57df7-bf74d\" (UID: \"be7870d2-35ee-43d8-9b02-74ad8ee286b1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bf74d" Sep 06 12:19:06.198224 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.198159 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7snvp\" (UniqueName: \"kubernetes.io/projected/529c654b-f937-4604-8271-488f66663e84-kube-api-access-7snvp\") pod \"node-exporter-n825g\" (UID: \"529c654b-f937-4604-8271-488f66663e84\") " pod="openshift-monitoring/node-exporter-n825g" Sep 06 12:19:06.198224 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.198207 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/529c654b-f937-4604-8271-488f66663e84-node-exporter-tls\") pod \"node-exporter-n825g\" (UID: \"529c654b-f937-4604-8271-488f66663e84\") " pod="openshift-monitoring/node-exporter-n825g" Sep 06 12:19:06.198378 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.198256 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/529c654b-f937-4604-8271-488f66663e84-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n825g\" (UID: \"529c654b-f937-4604-8271-488f66663e84\") " pod="openshift-monitoring/node-exporter-n825g" Sep 06 12:19:06.198378 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.198302 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/529c654b-f937-4604-8271-488f66663e84-node-exporter-wtmp\") pod \"node-exporter-n825g\" (UID: \"529c654b-f937-4604-8271-488f66663e84\") " pod="openshift-monitoring/node-exporter-n825g" Sep 06 12:19:06.198378 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.198326 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/529c654b-f937-4604-8271-488f66663e84-node-exporter-accelerators-collector-config\") pod \"node-exporter-n825g\" (UID: \"529c654b-f937-4604-8271-488f66663e84\") " pod="openshift-monitoring/node-exporter-n825g" Sep 06 12:19:06.198378 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.198355 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/529c654b-f937-4604-8271-488f66663e84-root\") pod \"node-exporter-n825g\" (UID: \"529c654b-f937-4604-8271-488f66663e84\") " pod="openshift-monitoring/node-exporter-n825g" Sep 06 12:19:06.198562 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.198377 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/529c654b-f937-4604-8271-488f66663e84-node-exporter-textfile\") pod \"node-exporter-n825g\" (UID: \"529c654b-f937-4604-8271-488f66663e84\") " pod="openshift-monitoring/node-exporter-n825g" Sep 06 12:19:06.198562 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:19:06.198395 2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 06 12:19:06.198562 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.198414 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/529c654b-f937-4604-8271-488f66663e84-metrics-client-ca\") pod \"node-exporter-n825g\" (UID: \"529c654b-f937-4604-8271-488f66663e84\") " pod="openshift-monitoring/node-exporter-n825g" Sep 06 12:19:06.198562 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.198430 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/529c654b-f937-4604-8271-488f66663e84-node-exporter-wtmp\") pod \"node-exporter-n825g\" (UID: \"529c654b-f937-4604-8271-488f66663e84\") " pod="openshift-monitoring/node-exporter-n825g" Sep 06 12:19:06.198562 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:19:06.198456 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/529c654b-f937-4604-8271-488f66663e84-node-exporter-tls podName:529c654b-f937-4604-8271-488f66663e84 nodeName:}" failed. No retries permitted until 2026-09-06 12:19:06.698437801 +0000 UTC m=+62.554195667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/529c654b-f937-4604-8271-488f66663e84-node-exporter-tls") pod "node-exporter-n825g" (UID: "529c654b-f937-4604-8271-488f66663e84") : secret "node-exporter-tls" not found Sep 06 12:19:06.198562 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.198453 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/529c654b-f937-4604-8271-488f66663e84-root\") pod \"node-exporter-n825g\" (UID: \"529c654b-f937-4604-8271-488f66663e84\") " pod="openshift-monitoring/node-exporter-n825g" Sep 06 12:19:06.198562 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.198480 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/529c654b-f937-4604-8271-488f66663e84-sys\") pod \"node-exporter-n825g\" (UID: \"529c654b-f937-4604-8271-488f66663e84\") " pod="openshift-monitoring/node-exporter-n825g" Sep 06 12:19:06.198955 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.198584 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/529c654b-f937-4604-8271-488f66663e84-sys\") pod \"node-exporter-n825g\" (UID: \"529c654b-f937-4604-8271-488f66663e84\") " pod="openshift-monitoring/node-exporter-n825g" Sep 06 12:19:06.198955 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.198744 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/529c654b-f937-4604-8271-488f66663e84-node-exporter-textfile\") pod \"node-exporter-n825g\" (UID: \"529c654b-f937-4604-8271-488f66663e84\") " pod="openshift-monitoring/node-exporter-n825g" Sep 06 12:19:06.199102 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.199071 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/529c654b-f937-4604-8271-488f66663e84-metrics-client-ca\") pod \"node-exporter-n825g\" (UID: \"529c654b-f937-4604-8271-488f66663e84\") " pod="openshift-monitoring/node-exporter-n825g" Sep 06 12:19:06.199157 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.199116 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/529c654b-f937-4604-8271-488f66663e84-node-exporter-accelerators-collector-config\") pod \"node-exporter-n825g\" (UID: \"529c654b-f937-4604-8271-488f66663e84\") " pod="openshift-monitoring/node-exporter-n825g" Sep 06 12:19:06.200497 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.200479 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/529c654b-f937-4604-8271-488f66663e84-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n825g\" (UID: \"529c654b-f937-4604-8271-488f66663e84\") " pod="openshift-monitoring/node-exporter-n825g" Sep 06 12:19:06.207945 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.207925 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7snvp\" (UniqueName: \"kubernetes.io/projected/529c654b-f937-4604-8271-488f66663e84-kube-api-access-7snvp\") pod \"node-exporter-n825g\" (UID: \"529c654b-f937-4604-8271-488f66663e84\") " pod="openshift-monitoring/node-exporter-n825g" Sep 06 12:19:06.602065 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.601984 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/212e2510-1c27-4fdf-ad5f-51b8b334106a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-5ddg8\" (UID: \"212e2510-1c27-4fdf-ad5f-51b8b334106a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5ddg8" Sep 06 12:19:06.602065 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.602034 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/be7870d2-35ee-43d8-9b02-74ad8ee286b1-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-bf74d\" (UID: \"be7870d2-35ee-43d8-9b02-74ad8ee286b1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bf74d" Sep 06 12:19:06.604483 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.604448 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/212e2510-1c27-4fdf-ad5f-51b8b334106a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-5ddg8\" (UID: \"212e2510-1c27-4fdf-ad5f-51b8b334106a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5ddg8" Sep 06 12:19:06.604590 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.604577 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/be7870d2-35ee-43d8-9b02-74ad8ee286b1-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-bf74d\" (UID: \"be7870d2-35ee-43d8-9b02-74ad8ee286b1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bf74d" Sep 06 12:19:06.702499 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.702475 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/529c654b-f937-4604-8271-488f66663e84-node-exporter-tls\") pod \"node-exporter-n825g\" (UID: \"529c654b-f937-4604-8271-488f66663e84\") " pod="openshift-monitoring/node-exporter-n825g" Sep 06 12:19:06.704656 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.704634 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/529c654b-f937-4604-8271-488f66663e84-node-exporter-tls\") pod \"node-exporter-n825g\" (UID: \"529c654b-f937-4604-8271-488f66663e84\") " pod="openshift-monitoring/node-exporter-n825g" Sep 06 12:19:06.803330 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.803301 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5ddg8" Sep 06 12:19:06.853375 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.853056 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-bf74d" Sep 06 12:19:06.869477 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.869227 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n825g" Sep 06 12:19:06.879825 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:19:06.879797 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod529c654b_f937_4604_8271_488f66663e84.slice/crio-8fdef22c76fa28c331d9338052723b4fc8c0b19e18a4023b08c538e3fe5b1f08 WatchSource:0}: Error finding container 8fdef22c76fa28c331d9338052723b4fc8c0b19e18a4023b08c538e3fe5b1f08: Status 404 returned error can't find the container with id 8fdef22c76fa28c331d9338052723b4fc8c0b19e18a4023b08c538e3fe5b1f08 Sep 06 12:19:06.962613 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.962508 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n825g" event={"ID":"529c654b-f937-4604-8271-488f66663e84","Type":"ContainerStarted","Data":"8fdef22c76fa28c331d9338052723b4fc8c0b19e18a4023b08c538e3fe5b1f08"} Sep 06 12:19:06.973188 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.973157 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 06 12:19:06.976050 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.976035 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:06.978466 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.978444 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-5tbgd\"" Sep 06 12:19:06.978730 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.978566 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 06 12:19:06.978730 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.978577 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 06 12:19:06.978730 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.978637 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 06 12:19:06.978730 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.978645 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 06 12:19:06.979032 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.979013 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 06 12:19:06.979140 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.979032 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 06 12:19:06.979140 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.979040 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 06 12:19:06.979140 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.979035 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 06 12:19:06.979140 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.979060 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 06 12:19:06.993915 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:06.993893 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 06 12:19:07.072426 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.072400 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-5ddg8"] Sep 06 12:19:07.075639 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:19:07.075614 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod212e2510_1c27_4fdf_ad5f_51b8b334106a.slice/crio-8fa709c59d3d86a8ae135954c026ffd13e28da471dbfa49186a9b95af6f6cc3f WatchSource:0}: Error finding container 8fa709c59d3d86a8ae135954c026ffd13e28da471dbfa49186a9b95af6f6cc3f: Status 404 returned error can't find the container with id 8fa709c59d3d86a8ae135954c026ffd13e28da471dbfa49186a9b95af6f6cc3f Sep 06 12:19:07.105259 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.105233 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/79460c57-2888-4499-937f-cbc5752f4956-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.105363 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.105282 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/79460c57-2888-4499-937f-cbc5752f4956-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.105363 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.105312 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/79460c57-2888-4499-937f-cbc5752f4956-config-volume\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.105449 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.105405 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/79460c57-2888-4499-937f-cbc5752f4956-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.105486 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.105448 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79460c57-2888-4499-937f-cbc5752f4956-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.105486 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.105471 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79460c57-2888-4499-937f-cbc5752f4956-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.105558 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.105508 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79460c57-2888-4499-937f-cbc5752f4956-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.105600 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.105559 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79460c57-2888-4499-937f-cbc5752f4956-config-out\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.105600 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.105583 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/79460c57-2888-4499-937f-cbc5752f4956-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.105673 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.105611 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79460c57-2888-4499-937f-cbc5752f4956-web-config\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.105673 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.105637 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79460c57-2888-4499-937f-cbc5752f4956-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.105673 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.105668 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbxhr\" (UniqueName: \"kubernetes.io/projected/79460c57-2888-4499-937f-cbc5752f4956-kube-api-access-fbxhr\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.105801 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.105693 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79460c57-2888-4499-937f-cbc5752f4956-tls-assets\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.134592 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.134573 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-bf74d"] Sep 06 12:19:07.136376 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:19:07.136351 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe7870d2_35ee_43d8_9b02_74ad8ee286b1.slice/crio-fb2d8f13d03a26dc46ffa40fb345e12cd929a4dba40c60694ea8c39e78667f4f WatchSource:0}: Error finding container fb2d8f13d03a26dc46ffa40fb345e12cd929a4dba40c60694ea8c39e78667f4f: Status 404 returned error can't find the container with id fb2d8f13d03a26dc46ffa40fb345e12cd929a4dba40c60694ea8c39e78667f4f Sep 06 12:19:07.206844 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.206821 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79460c57-2888-4499-937f-cbc5752f4956-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.206937 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.206875 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79460c57-2888-4499-937f-cbc5752f4956-config-out\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.206937 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.206906 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/79460c57-2888-4499-937f-cbc5752f4956-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.207044 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.206956 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79460c57-2888-4499-937f-cbc5752f4956-web-config\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.207116 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.207092 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79460c57-2888-4499-937f-cbc5752f4956-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.207173 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.207138 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbxhr\" (UniqueName: \"kubernetes.io/projected/79460c57-2888-4499-937f-cbc5752f4956-kube-api-access-fbxhr\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.207237 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.207180 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79460c57-2888-4499-937f-cbc5752f4956-tls-assets\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.207237 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.207231 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/79460c57-2888-4499-937f-cbc5752f4956-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.207499 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.207270 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/79460c57-2888-4499-937f-cbc5752f4956-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.207499 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.207298 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/79460c57-2888-4499-937f-cbc5752f4956-config-volume\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.207499 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.207342 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/79460c57-2888-4499-937f-cbc5752f4956-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.207499 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.207373 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79460c57-2888-4499-937f-cbc5752f4956-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.207499 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.207400 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79460c57-2888-4499-937f-cbc5752f4956-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.207499 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.207407 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/79460c57-2888-4499-937f-cbc5752f4956-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.208192 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.207971 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79460c57-2888-4499-937f-cbc5752f4956-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.208792 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.208575 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79460c57-2888-4499-937f-cbc5752f4956-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.210684 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.210228 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79460c57-2888-4499-937f-cbc5752f4956-web-config\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.210684 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.210638 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79460c57-2888-4499-937f-cbc5752f4956-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.210904 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.210690 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79460c57-2888-4499-937f-cbc5752f4956-config-out\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.211873 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.211810 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/79460c57-2888-4499-937f-cbc5752f4956-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.212126 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.212107 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/79460c57-2888-4499-937f-cbc5752f4956-config-volume\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.212211 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.212130 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79460c57-2888-4499-937f-cbc5752f4956-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.212211 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.212162 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79460c57-2888-4499-937f-cbc5752f4956-tls-assets\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.212563 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.212540 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/79460c57-2888-4499-937f-cbc5752f4956-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.213253 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.213231 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/79460c57-2888-4499-937f-cbc5752f4956-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.214375 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.214357 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbxhr\" (UniqueName: \"kubernetes.io/projected/79460c57-2888-4499-937f-cbc5752f4956-kube-api-access-fbxhr\") pod \"alertmanager-main-0\" (UID: \"79460c57-2888-4499-937f-cbc5752f4956\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.287138 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.287118 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 06 12:19:07.563346 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.563309 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 06 12:19:07.568250 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:19:07.568227 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79460c57_2888_4499_937f_cbc5752f4956.slice/crio-460b18c0a5b3af89d75d8f3721e1197aebcb31623fbc77cb6442c420b89742c8 WatchSource:0}: Error finding container 460b18c0a5b3af89d75d8f3721e1197aebcb31623fbc77cb6442c420b89742c8: Status 404 returned error can't find the container with id 460b18c0a5b3af89d75d8f3721e1197aebcb31623fbc77cb6442c420b89742c8 Sep 06 12:19:07.899102 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.897271 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-8675944c9b-bnsxn"] Sep 06 12:19:07.900988 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.900963 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8675944c9b-bnsxn" Sep 06 12:19:07.903709 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.903513 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 06 12:19:07.903709 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.903532 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 06 12:19:07.903917 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.903766 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 06 12:19:07.903917 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.903802 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 06 12:19:07.903917 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.903874 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-hxcwb\"" Sep 06 12:19:07.904111 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.903962 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7f47m4s13apsi\"" Sep 06 12:19:07.904111 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.904073 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 06 12:19:07.912445 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.912367 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8675944c9b-bnsxn"] Sep 06 12:19:07.967238 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.967212 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79460c57-2888-4499-937f-cbc5752f4956","Type":"ContainerStarted","Data":"460b18c0a5b3af89d75d8f3721e1197aebcb31623fbc77cb6442c420b89742c8"} Sep 06 12:19:07.968534 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.968477 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-bf74d" event={"ID":"be7870d2-35ee-43d8-9b02-74ad8ee286b1","Type":"ContainerStarted","Data":"fb2d8f13d03a26dc46ffa40fb345e12cd929a4dba40c60694ea8c39e78667f4f"} Sep 06 12:19:07.970452 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.970429 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5ddg8" event={"ID":"212e2510-1c27-4fdf-ad5f-51b8b334106a","Type":"ContainerStarted","Data":"a3dcf1a8937c6c74aac5a8fa5350bb79810df362e0af0301fcdee42626f89189"} Sep 06 12:19:07.970529 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.970486 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5ddg8" event={"ID":"212e2510-1c27-4fdf-ad5f-51b8b334106a","Type":"ContainerStarted","Data":"7c7318c89d6ec0532a4b94a9a5671d61d2e5be8cb5283a5076a49e41f0e87c36"} Sep 06 12:19:07.970529 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:07.970517 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5ddg8" event={"ID":"212e2510-1c27-4fdf-ad5f-51b8b334106a","Type":"ContainerStarted","Data":"8fa709c59d3d86a8ae135954c026ffd13e28da471dbfa49186a9b95af6f6cc3f"} Sep 06 12:19:08.014049 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:08.014023 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/09447812-284c-43c7-b928-f9afdb434f1f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8675944c9b-bnsxn\" (UID: \"09447812-284c-43c7-b928-f9afdb434f1f\") " pod="openshift-monitoring/thanos-querier-8675944c9b-bnsxn" Sep 06 12:19:08.014160 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:08.014084 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/09447812-284c-43c7-b928-f9afdb434f1f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8675944c9b-bnsxn\" (UID: \"09447812-284c-43c7-b928-f9afdb434f1f\") " pod="openshift-monitoring/thanos-querier-8675944c9b-bnsxn" Sep 06 12:19:08.014160 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:08.014124 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/09447812-284c-43c7-b928-f9afdb434f1f-secret-grpc-tls\") pod \"thanos-querier-8675944c9b-bnsxn\" (UID: \"09447812-284c-43c7-b928-f9afdb434f1f\") " pod="openshift-monitoring/thanos-querier-8675944c9b-bnsxn" Sep 06 12:19:08.014243 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:08.014182 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09447812-284c-43c7-b928-f9afdb434f1f-metrics-client-ca\") pod \"thanos-querier-8675944c9b-bnsxn\" (UID: \"09447812-284c-43c7-b928-f9afdb434f1f\") " pod="openshift-monitoring/thanos-querier-8675944c9b-bnsxn" Sep 06 12:19:08.014243 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:08.014208 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ssn9\" (UniqueName: \"kubernetes.io/projected/09447812-284c-43c7-b928-f9afdb434f1f-kube-api-access-5ssn9\") pod \"thanos-querier-8675944c9b-bnsxn\" (UID: \"09447812-284c-43c7-b928-f9afdb434f1f\") " pod="openshift-monitoring/thanos-querier-8675944c9b-bnsxn" Sep 06 12:19:08.014318 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:08.014243 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/09447812-284c-43c7-b928-f9afdb434f1f-secret-thanos-querier-tls\") pod \"thanos-querier-8675944c9b-bnsxn\" (UID: \"09447812-284c-43c7-b928-f9afdb434f1f\") " pod="openshift-monitoring/thanos-querier-8675944c9b-bnsxn" Sep 06 12:19:08.014318 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:08.014283 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/09447812-284c-43c7-b928-f9afdb434f1f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8675944c9b-bnsxn\" (UID: \"09447812-284c-43c7-b928-f9afdb434f1f\") " pod="openshift-monitoring/thanos-querier-8675944c9b-bnsxn" Sep 06 12:19:08.014402 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:08.014339 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/09447812-284c-43c7-b928-f9afdb434f1f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8675944c9b-bnsxn\" (UID: \"09447812-284c-43c7-b928-f9afdb434f1f\") " pod="openshift-monitoring/thanos-querier-8675944c9b-bnsxn" Sep 06 12:19:08.115342 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:08.115311 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/09447812-284c-43c7-b928-f9afdb434f1f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8675944c9b-bnsxn\" (UID: \"09447812-284c-43c7-b928-f9afdb434f1f\") " pod="openshift-monitoring/thanos-querier-8675944c9b-bnsxn" Sep 06 12:19:08.115464 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:08.115352 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/09447812-284c-43c7-b928-f9afdb434f1f-secret-grpc-tls\") pod \"thanos-querier-8675944c9b-bnsxn\" (UID: \"09447812-284c-43c7-b928-f9afdb434f1f\") " pod="openshift-monitoring/thanos-querier-8675944c9b-bnsxn" Sep 06 12:19:08.115464 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:08.115404 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09447812-284c-43c7-b928-f9afdb434f1f-metrics-client-ca\") pod \"thanos-querier-8675944c9b-bnsxn\" (UID: \"09447812-284c-43c7-b928-f9afdb434f1f\") " pod="openshift-monitoring/thanos-querier-8675944c9b-bnsxn" Sep 06 12:19:08.115464 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:08.115421 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ssn9\" (UniqueName: \"kubernetes.io/projected/09447812-284c-43c7-b928-f9afdb434f1f-kube-api-access-5ssn9\") pod \"thanos-querier-8675944c9b-bnsxn\" (UID: \"09447812-284c-43c7-b928-f9afdb434f1f\") " pod="openshift-monitoring/thanos-querier-8675944c9b-bnsxn" Sep 06 12:19:08.115464 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:08.115455 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/09447812-284c-43c7-b928-f9afdb434f1f-secret-thanos-querier-tls\") pod \"thanos-querier-8675944c9b-bnsxn\" (UID: \"09447812-284c-43c7-b928-f9afdb434f1f\") " pod="openshift-monitoring/thanos-querier-8675944c9b-bnsxn" Sep 06 12:19:08.115602 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:08.115494 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/09447812-284c-43c7-b928-f9afdb434f1f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8675944c9b-bnsxn\" (UID: \"09447812-284c-43c7-b928-f9afdb434f1f\") " pod="openshift-monitoring/thanos-querier-8675944c9b-bnsxn" Sep 06 12:19:08.115602 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:08.115537 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/09447812-284c-43c7-b928-f9afdb434f1f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8675944c9b-bnsxn\" (UID: \"09447812-284c-43c7-b928-f9afdb434f1f\") " pod="openshift-monitoring/thanos-querier-8675944c9b-bnsxn" Sep 06 12:19:08.115602 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:08.115591 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/09447812-284c-43c7-b928-f9afdb434f1f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8675944c9b-bnsxn\" (UID: \"09447812-284c-43c7-b928-f9afdb434f1f\") " pod="openshift-monitoring/thanos-querier-8675944c9b-bnsxn" Sep 06 12:19:08.116552 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:08.116170 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09447812-284c-43c7-b928-f9afdb434f1f-metrics-client-ca\") pod \"thanos-querier-8675944c9b-bnsxn\" (UID: \"09447812-284c-43c7-b928-f9afdb434f1f\") " pod="openshift-monitoring/thanos-querier-8675944c9b-bnsxn" Sep 06 12:19:08.118611 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:08.118588 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/09447812-284c-43c7-b928-f9afdb434f1f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8675944c9b-bnsxn\" (UID: \"09447812-284c-43c7-b928-f9afdb434f1f\") " pod="openshift-monitoring/thanos-querier-8675944c9b-bnsxn" Sep 06 12:19:08.118706 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:08.118667 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/09447812-284c-43c7-b928-f9afdb434f1f-secret-thanos-querier-tls\") pod \"thanos-querier-8675944c9b-bnsxn\" (UID: \"09447812-284c-43c7-b928-f9afdb434f1f\") " pod="openshift-monitoring/thanos-querier-8675944c9b-bnsxn" Sep 06 12:19:08.118779 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:08.118704 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/09447812-284c-43c7-b928-f9afdb434f1f-secret-grpc-tls\") pod \"thanos-querier-8675944c9b-bnsxn\" (UID: \"09447812-284c-43c7-b928-f9afdb434f1f\") " pod="openshift-monitoring/thanos-querier-8675944c9b-bnsxn" Sep 06 12:19:08.119071 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:08.119049 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/09447812-284c-43c7-b928-f9afdb434f1f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8675944c9b-bnsxn\" (UID: \"09447812-284c-43c7-b928-f9afdb434f1f\") " pod="openshift-monitoring/thanos-querier-8675944c9b-bnsxn" Sep 06 12:19:08.119151 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:08.119090 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/09447812-284c-43c7-b928-f9afdb434f1f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8675944c9b-bnsxn\" (UID: \"09447812-284c-43c7-b928-f9afdb434f1f\") " pod="openshift-monitoring/thanos-querier-8675944c9b-bnsxn" Sep 06 12:19:08.119151 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:08.119101 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/09447812-284c-43c7-b928-f9afdb434f1f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8675944c9b-bnsxn\" (UID: \"09447812-284c-43c7-b928-f9afdb434f1f\") " pod="openshift-monitoring/thanos-querier-8675944c9b-bnsxn" Sep 06 12:19:08.123456 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:08.123438 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ssn9\" (UniqueName: \"kubernetes.io/projected/09447812-284c-43c7-b928-f9afdb434f1f-kube-api-access-5ssn9\") pod \"thanos-querier-8675944c9b-bnsxn\" (UID: \"09447812-284c-43c7-b928-f9afdb434f1f\") " pod="openshift-monitoring/thanos-querier-8675944c9b-bnsxn" Sep 06 12:19:08.213572 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:08.213552 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8675944c9b-bnsxn" Sep 06 12:19:08.530574 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:08.530500 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8675944c9b-bnsxn"] Sep 06 12:19:08.734114 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:19:08.734075 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09447812_284c_43c7_b928_f9afdb434f1f.slice/crio-91ce911844491f15db869cbf625defb84f7858ec5054ca7569e62a71c63f57c6 WatchSource:0}: Error finding container 91ce911844491f15db869cbf625defb84f7858ec5054ca7569e62a71c63f57c6: Status 404 returned error can't find the container with id 91ce911844491f15db869cbf625defb84f7858ec5054ca7569e62a71c63f57c6 Sep 06 12:19:08.977408 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:08.977374 2576 generic.go:356] "Generic (PLEG): container finished" podID="529c654b-f937-4604-8271-488f66663e84" containerID="ffe970a92d1720156a77e877f3580bd74fa0bf7a02af6bf3e8f64847cb783ee6" exitCode=0 Sep 06 12:19:08.977877 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:08.977471 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n825g" event={"ID":"529c654b-f937-4604-8271-488f66663e84","Type":"ContainerDied","Data":"ffe970a92d1720156a77e877f3580bd74fa0bf7a02af6bf3e8f64847cb783ee6"} Sep 06 12:19:08.980103 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:08.980062 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5ddg8" event={"ID":"212e2510-1c27-4fdf-ad5f-51b8b334106a","Type":"ContainerStarted","Data":"ca693cbdd6be841a7f675991f8772b33ee3afa49fb87ef326d9dd52a351939fe"} Sep 06 12:19:08.982546 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:08.982522 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8675944c9b-bnsxn" event={"ID":"09447812-284c-43c7-b928-f9afdb434f1f","Type":"ContainerStarted","Data":"91ce911844491f15db869cbf625defb84f7858ec5054ca7569e62a71c63f57c6"} Sep 06 12:19:09.008389 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:09.008348 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5ddg8" podStartSLOduration=3.000061959 podStartE2EDuration="4.008334379s" totalImagesPullingTime="1.00827242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:19:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:19:09.007977491 +0000 UTC m=+64.863735401" watchObservedRunningTime="2026-09-06 12:19:09.008334379 +0000 UTC m=+64.864092276" Sep 06 12:19:09.989684 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:09.989652 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n825g" event={"ID":"529c654b-f937-4604-8271-488f66663e84","Type":"ContainerStarted","Data":"63c2708858f77236b9241bd1169df2d0848228a53b0846692337154292d9ec43"} Sep 06 12:19:09.990145 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:09.989745 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n825g" event={"ID":"529c654b-f937-4604-8271-488f66663e84","Type":"ContainerStarted","Data":"d25edc8539093b19332bc35bf04ec36842233fbb657c1f3adbea0348caf63e8e"} Sep 06 12:19:09.991480 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:09.991453 2576 generic.go:356] "Generic (PLEG): container finished" podID="79460c57-2888-4499-937f-cbc5752f4956" containerID="a574464f6c01c000c2bfc4f4b5ddfcce0cc657b693abf40f9ecda701f8aaa0d2" exitCode=0 Sep 06 12:19:09.991563 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:09.991535 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79460c57-2888-4499-937f-cbc5752f4956","Type":"ContainerDied","Data":"a574464f6c01c000c2bfc4f4b5ddfcce0cc657b693abf40f9ecda701f8aaa0d2"} Sep 06 12:19:09.994022 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:09.993997 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-bf74d" event={"ID":"be7870d2-35ee-43d8-9b02-74ad8ee286b1","Type":"ContainerStarted","Data":"95cdba155d177ce58608582c1311397cd97f37f55c6aff5328c180966c28c825"} Sep 06 12:19:09.994126 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:09.994046 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-bf74d" event={"ID":"be7870d2-35ee-43d8-9b02-74ad8ee286b1","Type":"ContainerStarted","Data":"5ffe9f57e48cb882cfcea85c31fc87cbb76058ddbd14b654492b46324f742add"} Sep 06 12:19:09.994126 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:09.994073 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-bf74d" event={"ID":"be7870d2-35ee-43d8-9b02-74ad8ee286b1","Type":"ContainerStarted","Data":"de07e84707699b596588f67c0a35b980f352caa02568197922e81b6bfc4f54a9"} Sep 06 12:19:10.007773 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.007722 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-n825g" podStartSLOduration=3.613403478 podStartE2EDuration="5.007710477s" totalImagesPullingTime="1.313262321s" totalInitContainerRuntime="81.044678ms" isStatefulPod=true podCreationTimestamp="2026-09-06 12:19:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:19:10.006583349 +0000 UTC m=+65.862341257" watchObservedRunningTime="2026-09-06 12:19:10.007710477 +0000 UTC m=+65.863468364" Sep 06 12:19:10.023290 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.023248 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-bf74d" podStartSLOduration=3.09064745 podStartE2EDuration="5.023233434s" totalImagesPullingTime="1.932585984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:19:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:19:10.022342692 +0000 UTC m=+65.878100580" watchObservedRunningTime="2026-09-06 12:19:10.023233434 +0000 UTC m=+65.878991320" Sep 06 12:19:10.193775 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.193718 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-757d878946-7fxrv"] Sep 06 12:19:10.196383 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.196363 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-757d878946-7fxrv" Sep 06 12:19:10.199294 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.199269 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 06 12:19:10.199394 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.199320 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 06 12:19:10.199394 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.199366 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-8590tc4rl544d\"" Sep 06 12:19:10.199394 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.199373 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 06 12:19:10.199564 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.199320 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 06 12:19:10.199686 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.199671 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-kkwqk\"" Sep 06 12:19:10.206076 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.206058 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-757d878946-7fxrv"] Sep 06 12:19:10.341130 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.341054 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa177905-aa73-46c3-8f2b-7a2a76c69607-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-757d878946-7fxrv\" (UID: \"aa177905-aa73-46c3-8f2b-7a2a76c69607\") " pod="openshift-monitoring/metrics-server-757d878946-7fxrv" Sep 06 12:19:10.341277 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.341136 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/aa177905-aa73-46c3-8f2b-7a2a76c69607-client-ca-bundle\") pod \"metrics-server-757d878946-7fxrv\" (UID: \"aa177905-aa73-46c3-8f2b-7a2a76c69607\") " pod="openshift-monitoring/metrics-server-757d878946-7fxrv" Sep 06 12:19:10.341277 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.341192 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/aa177905-aa73-46c3-8f2b-7a2a76c69607-audit-log\") pod \"metrics-server-757d878946-7fxrv\" (UID: \"aa177905-aa73-46c3-8f2b-7a2a76c69607\") " pod="openshift-monitoring/metrics-server-757d878946-7fxrv" Sep 06 12:19:10.341277 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.341219 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/aa177905-aa73-46c3-8f2b-7a2a76c69607-metrics-server-audit-profiles\") pod \"metrics-server-757d878946-7fxrv\" (UID: \"aa177905-aa73-46c3-8f2b-7a2a76c69607\") " pod="openshift-monitoring/metrics-server-757d878946-7fxrv" Sep 06 12:19:10.341277 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.341252 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e095072c-bd72-49b4-8cba-b49c94e40bf9-original-pull-secret\") pod \"global-pull-secret-syncer-m6lx6\" (UID: \"e095072c-bd72-49b4-8cba-b49c94e40bf9\") " pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:19:10.341475 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.341344 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/aa177905-aa73-46c3-8f2b-7a2a76c69607-secret-metrics-server-client-certs\") pod \"metrics-server-757d878946-7fxrv\" (UID: \"aa177905-aa73-46c3-8f2b-7a2a76c69607\") " pod="openshift-monitoring/metrics-server-757d878946-7fxrv" Sep 06 12:19:10.341475 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.341399 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kc6b\" (UniqueName: \"kubernetes.io/projected/aa177905-aa73-46c3-8f2b-7a2a76c69607-kube-api-access-5kc6b\") pod \"metrics-server-757d878946-7fxrv\" (UID: \"aa177905-aa73-46c3-8f2b-7a2a76c69607\") " pod="openshift-monitoring/metrics-server-757d878946-7fxrv" Sep 06 12:19:10.341475 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.341425 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/aa177905-aa73-46c3-8f2b-7a2a76c69607-secret-metrics-server-tls\") pod \"metrics-server-757d878946-7fxrv\" (UID: \"aa177905-aa73-46c3-8f2b-7a2a76c69607\") " pod="openshift-monitoring/metrics-server-757d878946-7fxrv" Sep 06 12:19:10.343645 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.343623 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 06 12:19:10.354825 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.354807 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e095072c-bd72-49b4-8cba-b49c94e40bf9-original-pull-secret\") pod \"global-pull-secret-syncer-m6lx6\" (UID: \"e095072c-bd72-49b4-8cba-b49c94e40bf9\") " pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:19:10.360192 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.359642 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6lx6" Sep 06 12:19:10.435482 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.435457 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8454d65567-kn6mk" Sep 06 12:19:10.435593 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.435511 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8454d65567-kn6mk" Sep 06 12:19:10.441108 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.441089 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8454d65567-kn6mk" Sep 06 12:19:10.442543 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.442522 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa177905-aa73-46c3-8f2b-7a2a76c69607-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-757d878946-7fxrv\" (UID: \"aa177905-aa73-46c3-8f2b-7a2a76c69607\") " pod="openshift-monitoring/metrics-server-757d878946-7fxrv" Sep 06 12:19:10.442640 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.442561 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/aa177905-aa73-46c3-8f2b-7a2a76c69607-client-ca-bundle\") pod \"metrics-server-757d878946-7fxrv\" (UID: \"aa177905-aa73-46c3-8f2b-7a2a76c69607\") " pod="openshift-monitoring/metrics-server-757d878946-7fxrv" Sep 06 12:19:10.442640 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.442611 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/aa177905-aa73-46c3-8f2b-7a2a76c69607-audit-log\") pod \"metrics-server-757d878946-7fxrv\" (UID: \"aa177905-aa73-46c3-8f2b-7a2a76c69607\") " pod="openshift-monitoring/metrics-server-757d878946-7fxrv" Sep 06 12:19:10.442744 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.442638 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/aa177905-aa73-46c3-8f2b-7a2a76c69607-metrics-server-audit-profiles\") pod \"metrics-server-757d878946-7fxrv\" (UID: \"aa177905-aa73-46c3-8f2b-7a2a76c69607\") " pod="openshift-monitoring/metrics-server-757d878946-7fxrv" Sep 06 12:19:10.442744 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.442688 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p48n7\" (UniqueName: \"kubernetes.io/projected/24ab4b74-f789-4642-adb7-a66bb6b75bb2-kube-api-access-p48n7\") pod \"network-check-target-gpxpp\" (UID: \"24ab4b74-f789-4642-adb7-a66bb6b75bb2\") " pod="openshift-network-diagnostics/network-check-target-gpxpp" Sep 06 12:19:10.442744 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.442715 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/aa177905-aa73-46c3-8f2b-7a2a76c69607-secret-metrics-server-client-certs\") pod \"metrics-server-757d878946-7fxrv\" (UID: \"aa177905-aa73-46c3-8f2b-7a2a76c69607\") " pod="openshift-monitoring/metrics-server-757d878946-7fxrv" Sep 06 12:19:10.442910 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.442774 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kc6b\" (UniqueName: \"kubernetes.io/projected/aa177905-aa73-46c3-8f2b-7a2a76c69607-kube-api-access-5kc6b\") pod \"metrics-server-757d878946-7fxrv\" (UID: \"aa177905-aa73-46c3-8f2b-7a2a76c69607\") " pod="openshift-monitoring/metrics-server-757d878946-7fxrv" Sep 06 12:19:10.442910 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.442805 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/aa177905-aa73-46c3-8f2b-7a2a76c69607-secret-metrics-server-tls\") pod \"metrics-server-757d878946-7fxrv\" (UID: \"aa177905-aa73-46c3-8f2b-7a2a76c69607\") " pod="openshift-monitoring/metrics-server-757d878946-7fxrv" Sep 06 12:19:10.442910 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.442840 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8d626d1-3961-441f-829f-2c8a637e63f7-metrics-certs\") pod \"network-metrics-daemon-wlhb5\" (UID: \"e8d626d1-3961-441f-829f-2c8a637e63f7\") " pod="openshift-multus/network-metrics-daemon-wlhb5" Sep 06 12:19:10.443299 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.443256 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/aa177905-aa73-46c3-8f2b-7a2a76c69607-audit-log\") pod \"metrics-server-757d878946-7fxrv\" (UID: \"aa177905-aa73-46c3-8f2b-7a2a76c69607\") " pod="openshift-monitoring/metrics-server-757d878946-7fxrv" Sep 06 12:19:10.443299 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.443264 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa177905-aa73-46c3-8f2b-7a2a76c69607-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-757d878946-7fxrv\" (UID: \"aa177905-aa73-46c3-8f2b-7a2a76c69607\") " pod="openshift-monitoring/metrics-server-757d878946-7fxrv" Sep 06 12:19:10.444316 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.444295 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/aa177905-aa73-46c3-8f2b-7a2a76c69607-metrics-server-audit-profiles\") pod \"metrics-server-757d878946-7fxrv\" (UID: \"aa177905-aa73-46c3-8f2b-7a2a76c69607\") " pod="openshift-monitoring/metrics-server-757d878946-7fxrv" Sep 06 12:19:10.445495 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.445471 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/aa177905-aa73-46c3-8f2b-7a2a76c69607-client-ca-bundle\") pod \"metrics-server-757d878946-7fxrv\" (UID: \"aa177905-aa73-46c3-8f2b-7a2a76c69607\") " pod="openshift-monitoring/metrics-server-757d878946-7fxrv" Sep 06 12:19:10.445681 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.445665 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 06 12:19:10.445737 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.445723 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 06 12:19:10.446008 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.445983 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/aa177905-aa73-46c3-8f2b-7a2a76c69607-secret-metrics-server-client-certs\") pod \"metrics-server-757d878946-7fxrv\" (UID: \"aa177905-aa73-46c3-8f2b-7a2a76c69607\") " pod="openshift-monitoring/metrics-server-757d878946-7fxrv" Sep 06 12:19:10.447662 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.447609 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/aa177905-aa73-46c3-8f2b-7a2a76c69607-secret-metrics-server-tls\") pod \"metrics-server-757d878946-7fxrv\" (UID: \"aa177905-aa73-46c3-8f2b-7a2a76c69607\") " pod="openshift-monitoring/metrics-server-757d878946-7fxrv" Sep 06 12:19:10.450851 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.450834 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kc6b\" (UniqueName: \"kubernetes.io/projected/aa177905-aa73-46c3-8f2b-7a2a76c69607-kube-api-access-5kc6b\") pod \"metrics-server-757d878946-7fxrv\" (UID: \"aa177905-aa73-46c3-8f2b-7a2a76c69607\") " pod="openshift-monitoring/metrics-server-757d878946-7fxrv" Sep 06 12:19:10.455699 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.455675 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 06 12:19:10.455870 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.455848 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8d626d1-3961-441f-829f-2c8a637e63f7-metrics-certs\") pod \"network-metrics-daemon-wlhb5\" (UID: \"e8d626d1-3961-441f-829f-2c8a637e63f7\") " pod="openshift-multus/network-metrics-daemon-wlhb5" Sep 06 12:19:10.466453 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.466431 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p48n7\" (UniqueName: \"kubernetes.io/projected/24ab4b74-f789-4642-adb7-a66bb6b75bb2-kube-api-access-p48n7\") pod \"network-check-target-gpxpp\" (UID: \"24ab4b74-f789-4642-adb7-a66bb6b75bb2\") " pod="openshift-network-diagnostics/network-check-target-gpxpp" Sep 06 12:19:10.507793 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.507767 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-757d878946-7fxrv" Sep 06 12:19:10.560072 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.560043 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qrr2p\"" Sep 06 12:19:10.568107 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.567963 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlhb5" Sep 06 12:19:10.671571 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.668483 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vrsdl\"" Sep 06 12:19:10.676796 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.676777 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxpp" Sep 06 12:19:10.692404 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.692382 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-w9hp5"] Sep 06 12:19:10.703939 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.703659 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55ff767878-tg6ft"] Sep 06 12:19:10.703939 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.703705 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-w9hp5"] Sep 06 12:19:10.704746 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.704158 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-w9hp5" Sep 06 12:19:10.706721 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.706343 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 06 12:19:10.706721 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.706584 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-sqxr7\"" Sep 06 12:19:10.733110 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.733061 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7dc4c856bf-2pjhb"] Sep 06 12:19:10.748779 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.748714 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dc4c856bf-2pjhb"] Sep 06 12:19:10.748871 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.748852 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dc4c856bf-2pjhb" Sep 06 12:19:10.795938 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.795908 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-757d878946-7fxrv"] Sep 06 12:19:10.797747 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.797725 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m6lx6"] Sep 06 12:19:10.800668 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:19:10.800647 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa177905_aa73_46c3_8f2b_7a2a76c69607.slice/crio-b64e91eacee8a2ca81d140d97605578fddd254112a87cc587627f92f5d5da9e1 WatchSource:0}: Error finding container b64e91eacee8a2ca81d140d97605578fddd254112a87cc587627f92f5d5da9e1: Status 404 returned error can't find the container with id b64e91eacee8a2ca81d140d97605578fddd254112a87cc587627f92f5d5da9e1 Sep 06 12:19:10.802296 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:19:10.802272 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode095072c_bd72_49b4_8cba_b49c94e40bf9.slice/crio-292b8cdbcaccf9d5a64649b4180e015715ce0c1d3adfc900089bdba9f787ff21 WatchSource:0}: Error finding container 292b8cdbcaccf9d5a64649b4180e015715ce0c1d3adfc900089bdba9f787ff21: Status 404 returned error can't find the container with id 292b8cdbcaccf9d5a64649b4180e015715ce0c1d3adfc900089bdba9f787ff21 Sep 06 12:19:10.847186 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.847111 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znv7t\" (UniqueName: \"kubernetes.io/projected/393ca7da-6bc7-4f06-9191-d9dee8f5623b-kube-api-access-znv7t\") pod \"console-7dc4c856bf-2pjhb\" (UID: \"393ca7da-6bc7-4f06-9191-d9dee8f5623b\") " pod="openshift-console/console-7dc4c856bf-2pjhb" Sep 06 12:19:10.847186 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.847163 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/393ca7da-6bc7-4f06-9191-d9dee8f5623b-console-oauth-config\") pod \"console-7dc4c856bf-2pjhb\" (UID: \"393ca7da-6bc7-4f06-9191-d9dee8f5623b\") " pod="openshift-console/console-7dc4c856bf-2pjhb" Sep 06 12:19:10.847329 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.847192 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/393ca7da-6bc7-4f06-9191-d9dee8f5623b-trusted-ca-bundle\") pod \"console-7dc4c856bf-2pjhb\" (UID: \"393ca7da-6bc7-4f06-9191-d9dee8f5623b\") " pod="openshift-console/console-7dc4c856bf-2pjhb" Sep 06 12:19:10.847496 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.847473 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/393ca7da-6bc7-4f06-9191-d9dee8f5623b-console-serving-cert\") pod \"console-7dc4c856bf-2pjhb\" (UID: \"393ca7da-6bc7-4f06-9191-d9dee8f5623b\") " pod="openshift-console/console-7dc4c856bf-2pjhb" Sep 06 12:19:10.847570 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.847511 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b88e1bb2-b111-43b7-82a1-a528302c18b9-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-w9hp5\" (UID: \"b88e1bb2-b111-43b7-82a1-a528302c18b9\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-w9hp5" Sep 06 12:19:10.847631 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.847581 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/393ca7da-6bc7-4f06-9191-d9dee8f5623b-service-ca\") pod \"console-7dc4c856bf-2pjhb\" (UID: \"393ca7da-6bc7-4f06-9191-d9dee8f5623b\") " pod="openshift-console/console-7dc4c856bf-2pjhb" Sep 06 12:19:10.847681 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.847637 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/393ca7da-6bc7-4f06-9191-d9dee8f5623b-oauth-serving-cert\") pod \"console-7dc4c856bf-2pjhb\" (UID: \"393ca7da-6bc7-4f06-9191-d9dee8f5623b\") " pod="openshift-console/console-7dc4c856bf-2pjhb" Sep 06 12:19:10.847681 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.847671 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtdbx\" (UniqueName: \"kubernetes.io/projected/b88e1bb2-b111-43b7-82a1-a528302c18b9-kube-api-access-qtdbx\") pod \"monitoring-plugin-85bbbbbf4-w9hp5\" (UID: \"b88e1bb2-b111-43b7-82a1-a528302c18b9\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-w9hp5" Sep 06 12:19:10.847790 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.847769 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/393ca7da-6bc7-4f06-9191-d9dee8f5623b-console-config\") pod \"console-7dc4c856bf-2pjhb\" (UID: \"393ca7da-6bc7-4f06-9191-d9dee8f5623b\") " pod="openshift-console/console-7dc4c856bf-2pjhb" Sep 06 12:19:10.887202 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.887172 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wlhb5"] Sep 06 12:19:10.893957 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:19:10.893936 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8d626d1_3961_441f_829f_2c8a637e63f7.slice/crio-e6b7acbce3bc8432be1354077c3f3b565b8b5672aae18538669b3f46754e40e1 WatchSource:0}: Error finding container e6b7acbce3bc8432be1354077c3f3b565b8b5672aae18538669b3f46754e40e1: Status 404 returned error can't find the container with id e6b7acbce3bc8432be1354077c3f3b565b8b5672aae18538669b3f46754e40e1 Sep 06 12:19:10.949008 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.948979 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znv7t\" (UniqueName: \"kubernetes.io/projected/393ca7da-6bc7-4f06-9191-d9dee8f5623b-kube-api-access-znv7t\") pod \"console-7dc4c856bf-2pjhb\" (UID: \"393ca7da-6bc7-4f06-9191-d9dee8f5623b\") " pod="openshift-console/console-7dc4c856bf-2pjhb" Sep 06 12:19:10.949119 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.949033 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/393ca7da-6bc7-4f06-9191-d9dee8f5623b-console-oauth-config\") pod \"console-7dc4c856bf-2pjhb\" (UID: \"393ca7da-6bc7-4f06-9191-d9dee8f5623b\") " pod="openshift-console/console-7dc4c856bf-2pjhb" Sep 06 12:19:10.949119 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.949059 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/393ca7da-6bc7-4f06-9191-d9dee8f5623b-trusted-ca-bundle\") pod \"console-7dc4c856bf-2pjhb\" (UID: \"393ca7da-6bc7-4f06-9191-d9dee8f5623b\") " pod="openshift-console/console-7dc4c856bf-2pjhb" Sep 06 12:19:10.949119 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.949104 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/393ca7da-6bc7-4f06-9191-d9dee8f5623b-console-serving-cert\") pod \"console-7dc4c856bf-2pjhb\" (UID: \"393ca7da-6bc7-4f06-9191-d9dee8f5623b\") " pod="openshift-console/console-7dc4c856bf-2pjhb" Sep 06 12:19:10.949287 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.949134 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b88e1bb2-b111-43b7-82a1-a528302c18b9-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-w9hp5\" (UID: \"b88e1bb2-b111-43b7-82a1-a528302c18b9\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-w9hp5" Sep 06 12:19:10.949287 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.949183 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/393ca7da-6bc7-4f06-9191-d9dee8f5623b-service-ca\") pod \"console-7dc4c856bf-2pjhb\" (UID: \"393ca7da-6bc7-4f06-9191-d9dee8f5623b\") " pod="openshift-console/console-7dc4c856bf-2pjhb" Sep 06 12:19:10.949287 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.949212 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/393ca7da-6bc7-4f06-9191-d9dee8f5623b-oauth-serving-cert\") pod \"console-7dc4c856bf-2pjhb\" (UID: \"393ca7da-6bc7-4f06-9191-d9dee8f5623b\") " pod="openshift-console/console-7dc4c856bf-2pjhb" Sep 06 12:19:10.949287 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.949245 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtdbx\" (UniqueName: \"kubernetes.io/projected/b88e1bb2-b111-43b7-82a1-a528302c18b9-kube-api-access-qtdbx\") pod \"monitoring-plugin-85bbbbbf4-w9hp5\" (UID: \"b88e1bb2-b111-43b7-82a1-a528302c18b9\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-w9hp5" Sep 06 12:19:10.949475 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.949292 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/393ca7da-6bc7-4f06-9191-d9dee8f5623b-console-config\") pod \"console-7dc4c856bf-2pjhb\" (UID: \"393ca7da-6bc7-4f06-9191-d9dee8f5623b\") " pod="openshift-console/console-7dc4c856bf-2pjhb" Sep 06 12:19:10.950085 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.950053 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/393ca7da-6bc7-4f06-9191-d9dee8f5623b-console-config\") pod \"console-7dc4c856bf-2pjhb\" (UID: \"393ca7da-6bc7-4f06-9191-d9dee8f5623b\") " pod="openshift-console/console-7dc4c856bf-2pjhb" Sep 06 12:19:10.950321 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.950273 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/393ca7da-6bc7-4f06-9191-d9dee8f5623b-trusted-ca-bundle\") pod \"console-7dc4c856bf-2pjhb\" (UID: \"393ca7da-6bc7-4f06-9191-d9dee8f5623b\") " pod="openshift-console/console-7dc4c856bf-2pjhb" Sep 06 12:19:10.950411 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.950325 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/393ca7da-6bc7-4f06-9191-d9dee8f5623b-oauth-serving-cert\") pod \"console-7dc4c856bf-2pjhb\" (UID: \"393ca7da-6bc7-4f06-9191-d9dee8f5623b\") " pod="openshift-console/console-7dc4c856bf-2pjhb" Sep 06 12:19:10.950659 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.950574 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/393ca7da-6bc7-4f06-9191-d9dee8f5623b-service-ca\") pod \"console-7dc4c856bf-2pjhb\" (UID: \"393ca7da-6bc7-4f06-9191-d9dee8f5623b\") " pod="openshift-console/console-7dc4c856bf-2pjhb" Sep 06 12:19:10.952169 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.952132 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/393ca7da-6bc7-4f06-9191-d9dee8f5623b-console-oauth-config\") pod \"console-7dc4c856bf-2pjhb\" (UID: \"393ca7da-6bc7-4f06-9191-d9dee8f5623b\") " pod="openshift-console/console-7dc4c856bf-2pjhb" Sep 06 12:19:10.952578 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.952448 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b88e1bb2-b111-43b7-82a1-a528302c18b9-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-w9hp5\" (UID: \"b88e1bb2-b111-43b7-82a1-a528302c18b9\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-w9hp5" Sep 06 12:19:10.953388 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.953371 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/393ca7da-6bc7-4f06-9191-d9dee8f5623b-console-serving-cert\") pod \"console-7dc4c856bf-2pjhb\" (UID: \"393ca7da-6bc7-4f06-9191-d9dee8f5623b\") " pod="openshift-console/console-7dc4c856bf-2pjhb" Sep 06 12:19:10.957042 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.957020 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtdbx\" (UniqueName: \"kubernetes.io/projected/b88e1bb2-b111-43b7-82a1-a528302c18b9-kube-api-access-qtdbx\") pod \"monitoring-plugin-85bbbbbf4-w9hp5\" (UID: \"b88e1bb2-b111-43b7-82a1-a528302c18b9\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-w9hp5" Sep 06 12:19:10.957235 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.957219 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znv7t\" (UniqueName: \"kubernetes.io/projected/393ca7da-6bc7-4f06-9191-d9dee8f5623b-kube-api-access-znv7t\") pod \"console-7dc4c856bf-2pjhb\" (UID: \"393ca7da-6bc7-4f06-9191-d9dee8f5623b\") " pod="openshift-console/console-7dc4c856bf-2pjhb" Sep 06 12:19:10.962553 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.962532 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gpxpp"] Sep 06 12:19:10.966177 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:19:10.966156 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ab4b74_f789_4642_adb7_a66bb6b75bb2.slice/crio-94c0d126321921f56db86ce65b76161839a8d9573b2964a3cf6351efb0d4d25f WatchSource:0}: Error finding container 94c0d126321921f56db86ce65b76161839a8d9573b2964a3cf6351efb0d4d25f: Status 404 returned error can't find the container with id 94c0d126321921f56db86ce65b76161839a8d9573b2964a3cf6351efb0d4d25f Sep 06 12:19:10.999745 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:10.999643 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-757d878946-7fxrv" event={"ID":"aa177905-aa73-46c3-8f2b-7a2a76c69607","Type":"ContainerStarted","Data":"b64e91eacee8a2ca81d140d97605578fddd254112a87cc587627f92f5d5da9e1"} Sep 06 12:19:11.001150 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:11.001110 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wlhb5" event={"ID":"e8d626d1-3961-441f-829f-2c8a637e63f7","Type":"ContainerStarted","Data":"e6b7acbce3bc8432be1354077c3f3b565b8b5672aae18538669b3f46754e40e1"} Sep 06 12:19:11.002582 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:11.002482 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m6lx6" event={"ID":"e095072c-bd72-49b4-8cba-b49c94e40bf9","Type":"ContainerStarted","Data":"292b8cdbcaccf9d5a64649b4180e015715ce0c1d3adfc900089bdba9f787ff21"} Sep 06 12:19:11.005171 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:11.005138 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8675944c9b-bnsxn" event={"ID":"09447812-284c-43c7-b928-f9afdb434f1f","Type":"ContainerStarted","Data":"2beb7851f3e656e46c9eb670f2cf5c7c0d1a913982103446e1ce7b3922ea5254"} Sep 06 12:19:11.005242 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:11.005203 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8675944c9b-bnsxn" event={"ID":"09447812-284c-43c7-b928-f9afdb434f1f","Type":"ContainerStarted","Data":"2e2a83a009e5c1636bd76d8c101364f8a17791eb013b5e981dc87d0afdbb0cb3"} Sep 06 12:19:11.005242 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:11.005235 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8675944c9b-bnsxn" event={"ID":"09447812-284c-43c7-b928-f9afdb434f1f","Type":"ContainerStarted","Data":"331dab824ea7e7ef64b7888cfe67caca1b7b978d22ee868f22c989cf784033da"} Sep 06 12:19:11.006537 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:11.006493 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gpxpp" event={"ID":"24ab4b74-f789-4642-adb7-a66bb6b75bb2","Type":"ContainerStarted","Data":"94c0d126321921f56db86ce65b76161839a8d9573b2964a3cf6351efb0d4d25f"} Sep 06 12:19:11.011150 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:11.011100 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8454d65567-kn6mk" Sep 06 12:19:11.016700 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:11.016684 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-w9hp5" Sep 06 12:19:11.077057 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:11.076458 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dc4c856bf-2pjhb" Sep 06 12:19:11.310211 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:11.310138 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-w9hp5"] Sep 06 12:19:11.313573 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:19:11.313547 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb88e1bb2_b111_43b7_82a1_a528302c18b9.slice/crio-acb3f9d692b43d68fb975b3eb64602bee7ac76c20deeb7759163add06fe4fed1 WatchSource:0}: Error finding container acb3f9d692b43d68fb975b3eb64602bee7ac76c20deeb7759163add06fe4fed1: Status 404 returned error can't find the container with id acb3f9d692b43d68fb975b3eb64602bee7ac76c20deeb7759163add06fe4fed1 Sep 06 12:19:11.387799 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:11.387661 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dc4c856bf-2pjhb"] Sep 06 12:19:12.016375 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.016331 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-w9hp5" event={"ID":"b88e1bb2-b111-43b7-82a1-a528302c18b9","Type":"ContainerStarted","Data":"acb3f9d692b43d68fb975b3eb64602bee7ac76c20deeb7759163add06fe4fed1"} Sep 06 12:19:12.115259 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:19:12.114932 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod393ca7da_6bc7_4f06_9191_d9dee8f5623b.slice/crio-28509715a972d82609b845b62343078f0be8d9b44c761ec591192ad0f9333945 WatchSource:0}: Error finding container 28509715a972d82609b845b62343078f0be8d9b44c761ec591192ad0f9333945: Status 404 returned error can't find the container with id 28509715a972d82609b845b62343078f0be8d9b44c761ec591192ad0f9333945 Sep 06 12:19:12.172851 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.172307 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 06 12:19:12.191946 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.191917 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 06 12:19:12.192102 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.192079 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.199681 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.194859 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 06 12:19:12.199681 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.195145 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 06 12:19:12.199681 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.195379 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 06 12:19:12.199681 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.195610 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 06 12:19:12.199681 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.196178 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 06 12:19:12.199681 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.196942 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 06 12:19:12.199681 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.197876 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 06 12:19:12.199681 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.198451 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 06 12:19:12.199681 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.198605 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rgzp8\"" Sep 06 12:19:12.199681 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.198936 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 06 12:19:12.199681 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.199687 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 06 12:19:12.203457 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.203191 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-41l60lbr484bb\"" Sep 06 12:19:12.206789 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.204504 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 06 12:19:12.206789 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.206525 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 06 12:19:12.207062 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.207029 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 06 12:19:12.209251 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.209230 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 06 12:19:12.264522 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.264467 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7afd0d05-3e0b-493f-bba1-999be7877cf5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.264522 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.264516 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7afd0d05-3e0b-493f-bba1-999be7877cf5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.264719 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.264580 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7afd0d05-3e0b-493f-bba1-999be7877cf5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.264719 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.264644 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7afd0d05-3e0b-493f-bba1-999be7877cf5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.264719 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.264670 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7afd0d05-3e0b-493f-bba1-999be7877cf5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.264719 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.264708 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7afd0d05-3e0b-493f-bba1-999be7877cf5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.264953 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.264773 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7afd0d05-3e0b-493f-bba1-999be7877cf5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.264953 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.264819 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7afd0d05-3e0b-493f-bba1-999be7877cf5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.264953 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.264849 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7afd0d05-3e0b-493f-bba1-999be7877cf5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.264953 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.264874 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7afd0d05-3e0b-493f-bba1-999be7877cf5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.264953 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.264925 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7afd0d05-3e0b-493f-bba1-999be7877cf5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.265187 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.264966 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7afd0d05-3e0b-493f-bba1-999be7877cf5-config-out\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.265187 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.264991 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7afd0d05-3e0b-493f-bba1-999be7877cf5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.265187 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.265073 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7afd0d05-3e0b-493f-bba1-999be7877cf5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.265187 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.265111 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7afd0d05-3e0b-493f-bba1-999be7877cf5-web-config\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.265377 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.265175 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7afd0d05-3e0b-493f-bba1-999be7877cf5-config\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.265377 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.265249 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7afd0d05-3e0b-493f-bba1-999be7877cf5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.265377 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.265302 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7afd0d05-3e0b-493f-bba1-999be7877cf5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.265377 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.265342 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwfhz\" (UniqueName: \"kubernetes.io/projected/7afd0d05-3e0b-493f-bba1-999be7877cf5-kube-api-access-kwfhz\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.265377 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.265370 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7afd0d05-3e0b-493f-bba1-999be7877cf5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.366357 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.366268 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwfhz\" (UniqueName: \"kubernetes.io/projected/7afd0d05-3e0b-493f-bba1-999be7877cf5-kube-api-access-kwfhz\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.366357 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.366318 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7afd0d05-3e0b-493f-bba1-999be7877cf5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.366357 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.366351 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7afd0d05-3e0b-493f-bba1-999be7877cf5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.366615 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.366382 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7afd0d05-3e0b-493f-bba1-999be7877cf5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.366615 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.366409 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7afd0d05-3e0b-493f-bba1-999be7877cf5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.366615 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.366452 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7afd0d05-3e0b-493f-bba1-999be7877cf5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.366615 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.366475 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7afd0d05-3e0b-493f-bba1-999be7877cf5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.366615 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.366511 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7afd0d05-3e0b-493f-bba1-999be7877cf5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.366615 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.366540 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7afd0d05-3e0b-493f-bba1-999be7877cf5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.366615 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.366573 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7afd0d05-3e0b-493f-bba1-999be7877cf5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.366615 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.366601 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7afd0d05-3e0b-493f-bba1-999be7877cf5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.367259 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.366625 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7afd0d05-3e0b-493f-bba1-999be7877cf5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.367259 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.366653 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7afd0d05-3e0b-493f-bba1-999be7877cf5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.367259 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.366686 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7afd0d05-3e0b-493f-bba1-999be7877cf5-config-out\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.367259 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.366709 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7afd0d05-3e0b-493f-bba1-999be7877cf5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.367259 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.366772 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7afd0d05-3e0b-493f-bba1-999be7877cf5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.367259 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.366806 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7afd0d05-3e0b-493f-bba1-999be7877cf5-web-config\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.367259 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.366844 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7afd0d05-3e0b-493f-bba1-999be7877cf5-config\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.367259 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.366873 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7afd0d05-3e0b-493f-bba1-999be7877cf5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.367259 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.366897 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7afd0d05-3e0b-493f-bba1-999be7877cf5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.367930 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.367904 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7afd0d05-3e0b-493f-bba1-999be7877cf5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.368374 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.368186 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7afd0d05-3e0b-493f-bba1-999be7877cf5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.368482 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.368413 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7afd0d05-3e0b-493f-bba1-999be7877cf5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.370097 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.369235 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7afd0d05-3e0b-493f-bba1-999be7877cf5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.370097 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.369246 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7afd0d05-3e0b-493f-bba1-999be7877cf5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.376608 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.372224 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7afd0d05-3e0b-493f-bba1-999be7877cf5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.376608 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.374036 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7afd0d05-3e0b-493f-bba1-999be7877cf5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.378593 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.377607 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7afd0d05-3e0b-493f-bba1-999be7877cf5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.378593 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.378061 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7afd0d05-3e0b-493f-bba1-999be7877cf5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.378593 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.378173 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7afd0d05-3e0b-493f-bba1-999be7877cf5-config-out\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.378593 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.378286 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7afd0d05-3e0b-493f-bba1-999be7877cf5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.378593 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.378461 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7afd0d05-3e0b-493f-bba1-999be7877cf5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.379507 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.379259 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7afd0d05-3e0b-493f-bba1-999be7877cf5-config\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.379507 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.379413 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7afd0d05-3e0b-493f-bba1-999be7877cf5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.379507 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.379425 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7afd0d05-3e0b-493f-bba1-999be7877cf5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.379507 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.379506 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7afd0d05-3e0b-493f-bba1-999be7877cf5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.380205 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.380010 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7afd0d05-3e0b-493f-bba1-999be7877cf5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.380299 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.380199 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7afd0d05-3e0b-493f-bba1-999be7877cf5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.380299 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.380271 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwfhz\" (UniqueName: \"kubernetes.io/projected/7afd0d05-3e0b-493f-bba1-999be7877cf5-kube-api-access-kwfhz\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.381570 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.381388 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7afd0d05-3e0b-493f-bba1-999be7877cf5-web-config\") pod \"prometheus-k8s-0\" (UID: \"7afd0d05-3e0b-493f-bba1-999be7877cf5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:12.519741 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:12.519456 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:13.025374 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:13.025326 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dc4c856bf-2pjhb" event={"ID":"393ca7da-6bc7-4f06-9191-d9dee8f5623b","Type":"ContainerStarted","Data":"ef2f6d9dd6210987e923fdae132141a92698a4d1fad0f31e7429f951c3341c5c"} Sep 06 12:19:13.025858 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:13.025403 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dc4c856bf-2pjhb" event={"ID":"393ca7da-6bc7-4f06-9191-d9dee8f5623b","Type":"ContainerStarted","Data":"28509715a972d82609b845b62343078f0be8d9b44c761ec591192ad0f9333945"} Sep 06 12:19:13.042116 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:13.042057 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7dc4c856bf-2pjhb" podStartSLOduration=3.042039521 podStartE2EDuration="3.042039521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:19:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:19:13.041637615 +0000 UTC m=+68.897395521" watchObservedRunningTime="2026-09-06 12:19:13.042039521 +0000 UTC m=+68.897797409" Sep 06 12:19:13.827021 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:13.826990 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 06 12:19:14.006306 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:19:14.006276 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7afd0d05_3e0b_493f_bba1_999be7877cf5.slice/crio-70582ae7599375540029a59f7f797d09847a87fc53e3e89b57fc1c8968980cfd WatchSource:0}: Error finding container 70582ae7599375540029a59f7f797d09847a87fc53e3e89b57fc1c8968980cfd: Status 404 returned error can't find the container with id 70582ae7599375540029a59f7f797d09847a87fc53e3e89b57fc1c8968980cfd Sep 06 12:19:14.031838 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:14.031800 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7afd0d05-3e0b-493f-bba1-999be7877cf5","Type":"ContainerStarted","Data":"70582ae7599375540029a59f7f797d09847a87fc53e3e89b57fc1c8968980cfd"} Sep 06 12:19:17.046307 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:17.046276 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wlhb5" event={"ID":"e8d626d1-3961-441f-829f-2c8a637e63f7","Type":"ContainerStarted","Data":"d9e14048f9b8bf27fc448b5e62de79ef1f4f18d566e64deb91af8a386f8fc9f7"} Sep 06 12:19:17.046697 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:17.046335 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wlhb5" event={"ID":"e8d626d1-3961-441f-829f-2c8a637e63f7","Type":"ContainerStarted","Data":"99a6e458c587b487eba40708ab44212330ac20c149d8c34418cad15fe5b8ce62"} Sep 06 12:19:17.047935 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:17.047907 2576 generic.go:356] "Generic (PLEG): container finished" podID="7afd0d05-3e0b-493f-bba1-999be7877cf5" containerID="57162ae022bc7c485f1979207ee3e6bee2295585c57aabc68bdcba93279d7b33" exitCode=0 Sep 06 12:19:17.048047 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:17.047955 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7afd0d05-3e0b-493f-bba1-999be7877cf5","Type":"ContainerDied","Data":"57162ae022bc7c485f1979207ee3e6bee2295585c57aabc68bdcba93279d7b33"} Sep 06 12:19:17.051311 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:17.051287 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8675944c9b-bnsxn" event={"ID":"09447812-284c-43c7-b928-f9afdb434f1f","Type":"ContainerStarted","Data":"989390cc4f69b15f3860519aca6791af314167664497dc55b02287f9f894329a"} Sep 06 12:19:17.051384 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:17.051354 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8675944c9b-bnsxn" event={"ID":"09447812-284c-43c7-b928-f9afdb434f1f","Type":"ContainerStarted","Data":"4272b1d6f47b2d6d0d97eaa60b68e02f70c30401e79cf763ab7fc3e2c11d2b20"} Sep 06 12:19:17.051430 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:17.051396 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8675944c9b-bnsxn" event={"ID":"09447812-284c-43c7-b928-f9afdb434f1f","Type":"ContainerStarted","Data":"fa18f9acddeae1c50572041f2d4f899c56b2babe753ca8d0a0432b7f7df523e2"} Sep 06 12:19:17.051466 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:17.051437 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-8675944c9b-bnsxn" Sep 06 12:19:17.053052 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:17.053031 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gpxpp" event={"ID":"24ab4b74-f789-4642-adb7-a66bb6b75bb2","Type":"ContainerStarted","Data":"fa7e585e254506fc70e1627701460be84bd0850629868b393e9811d64b53f385"} Sep 06 12:19:17.053230 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:17.053212 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gpxpp" Sep 06 12:19:17.056937 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:17.056916 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79460c57-2888-4499-937f-cbc5752f4956","Type":"ContainerStarted","Data":"19baffde4f51b918b1aa92326619d51e15b10027139c35d94223cbc2ad44ce3e"} Sep 06 12:19:17.057052 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:17.056975 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79460c57-2888-4499-937f-cbc5752f4956","Type":"ContainerStarted","Data":"69f188e7bf86e06a433f9b2c0eaecdaecaf726e49816c0cad730daf02530238e"} Sep 06 12:19:17.057052 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:17.057016 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79460c57-2888-4499-937f-cbc5752f4956","Type":"ContainerStarted","Data":"277e9d11349a1c6850efd5249babd0046f6098260436490e12916ed7f7a8781c"} Sep 06 12:19:17.057052 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:17.057051 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79460c57-2888-4499-937f-cbc5752f4956","Type":"ContainerStarted","Data":"022035a7bc154918b675add8b067229bbe0c7ceb57d997bd3121971737c2b8f6"} Sep 06 12:19:17.057208 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:17.057075 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79460c57-2888-4499-937f-cbc5752f4956","Type":"ContainerStarted","Data":"5ec0320be2b4cdbd3cd4ca2f0c3d093e11bb01a3ff7b97bed8d2af568caca879"} Sep 06 12:19:17.057208 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:17.057097 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79460c57-2888-4499-937f-cbc5752f4956","Type":"ContainerStarted","Data":"2472f5f58241a83991a88ffd7cb02300c2a5a2f368bcdb95603dbe1e13f0f4cc"} Sep 06 12:19:17.057775 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:17.057734 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-8675944c9b-bnsxn" Sep 06 12:19:17.058637 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:17.058616 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m6lx6" event={"ID":"e095072c-bd72-49b4-8cba-b49c94e40bf9","Type":"ContainerStarted","Data":"424b972f42aba97e4c828ea78ee0dbb87b0f810bee0e342d5b61b5bfeb6d586a"} Sep 06 12:19:17.060528 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:17.060080 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-757d878946-7fxrv" event={"ID":"aa177905-aa73-46c3-8f2b-7a2a76c69607","Type":"ContainerStarted","Data":"6cceb8752b05a3112d53e914708af1f89b63d0a46f0c86b094d45b03f141a1d7"} Sep 06 12:19:17.062173 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:17.062154 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-w9hp5" event={"ID":"b88e1bb2-b111-43b7-82a1-a528302c18b9","Type":"ContainerStarted","Data":"36a6ffe8a9f46e98e473f5e714f7cf40f07675cff522006a64da25cbe5ab0c0b"} Sep 06 12:19:17.062271 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:17.062183 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wlhb5" podStartSLOduration=70.408636039 podStartE2EDuration="1m13.062165224s" totalImagesPullingTime="2.653529185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:18:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:19:17.06056081 +0000 UTC m=+72.916318698" watchObservedRunningTime="2026-09-06 12:19:17.062165224 +0000 UTC m=+72.917923112" Sep 06 12:19:17.062341 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:17.062327 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-w9hp5" Sep 06 12:19:17.066990 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:17.066973 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-w9hp5" Sep 06 12:19:17.080329 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:17.080295 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-757d878946-7fxrv" podStartSLOduration=4.334905942 podStartE2EDuration="7.080283105s" totalImagesPullingTime="2.745377163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:19:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:19:17.078937465 +0000 UTC m=+72.934695386" watchObservedRunningTime="2026-09-06 12:19:17.080283105 +0000 UTC m=+72.936040991" Sep 06 12:19:17.111928 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:17.111893 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=6.057306102 podStartE2EDuration="11.111881005s" totalImagesPullingTime="5.035199134s" totalInitContainerRuntime="19.375769ms" isStatefulPod=false podCreationTimestamp="2026-09-06 12:19:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:19:17.110814542 +0000 UTC m=+72.966572449" watchObservedRunningTime="2026-09-06 12:19:17.111881005 +0000 UTC m=+72.967638904" Sep 06 12:19:17.135597 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:17.135551 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-8675944c9b-bnsxn" podStartSLOduration=2.750293623 podStartE2EDuration="10.135534356s" totalImagesPullingTime="7.385240733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:19:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:19:17.133243571 +0000 UTC m=+72.989001507" watchObservedRunningTime="2026-09-06 12:19:17.135534356 +0000 UTC m=+72.991292243" Sep 06 12:19:17.177719 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:17.177678 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-m6lx6" podStartSLOduration=67.469725114 podStartE2EDuration="1m13.177664628s" totalImagesPullingTime="5.707939514s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 12:18:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:19:17.176717383 +0000 UTC m=+73.032475288" watchObservedRunningTime="2026-09-06 12:19:17.177664628 +0000 UTC m=+73.033422547" Sep 06 12:19:17.191495 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:17.191450 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gpxpp" podStartSLOduration=67.648280308 podStartE2EDuration="1m13.191436215s" totalImagesPullingTime="5.543155907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:18:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:19:17.191436121 +0000 UTC m=+73.047194027" watchObservedRunningTime="2026-09-06 12:19:17.191436215 +0000 UTC m=+73.047194102" Sep 06 12:19:17.207154 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:17.207088 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-w9hp5" podStartSLOduration=2.025430726 podStartE2EDuration="7.207072488s" totalImagesPullingTime="5.181641762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:19:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:19:17.205036025 +0000 UTC m=+73.060793931" watchObservedRunningTime="2026-09-06 12:19:17.207072488 +0000 UTC m=+73.062830375" Sep 06 12:19:20.081995 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:20.081964 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7afd0d05-3e0b-493f-bba1-999be7877cf5","Type":"ContainerStarted","Data":"932df69cb99f411538eeda49d85d7c83cec44272a0f0089557f049a392ede039"} Sep 06 12:19:21.076773 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:21.076705 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7dc4c856bf-2pjhb" Sep 06 12:19:21.076945 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:21.076796 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7dc4c856bf-2pjhb" Sep 06 12:19:21.081693 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:21.081664 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7dc4c856bf-2pjhb" Sep 06 12:19:21.089710 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:21.089679 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7afd0d05-3e0b-493f-bba1-999be7877cf5","Type":"ContainerStarted","Data":"62c1f05d8038870afcb905d96cd12fe56e95cd42a391517f230c149db16b1aad"} Sep 06 12:19:21.090016 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:21.089775 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7afd0d05-3e0b-493f-bba1-999be7877cf5","Type":"ContainerStarted","Data":"0a737dd4bbb2bc559e7b7c688662dbd2629934edd34e02eb76209e0504b1b8e3"} Sep 06 12:19:21.090016 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:21.089809 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7afd0d05-3e0b-493f-bba1-999be7877cf5","Type":"ContainerStarted","Data":"b6afbec33acb22f354d5637f882ea2059ed34009d0951d90dc8a366e37df3085"} Sep 06 12:19:21.090016 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:21.089836 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7afd0d05-3e0b-493f-bba1-999be7877cf5","Type":"ContainerStarted","Data":"836dbafc161fcda654b9039f37f59e8fc9b3c31d546e0715fbd540f15961e980"} Sep 06 12:19:21.090016 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:21.089863 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7afd0d05-3e0b-493f-bba1-999be7877cf5","Type":"ContainerStarted","Data":"54e02871a063cea4b7095c21540edcf57805077ecccbd28831834cb1b7401721"} Sep 06 12:19:21.093672 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:21.093653 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7dc4c856bf-2pjhb" Sep 06 12:19:21.138537 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:21.138503 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8454d65567-kn6mk"] Sep 06 12:19:21.139025 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:21.138976 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=6.151149901 podStartE2EDuration="9.138957292s" totalImagesPullingTime="2.965763985s" totalInitContainerRuntime="22.043406ms" isStatefulPod=false podCreationTimestamp="2026-09-06 12:19:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:19:21.135951525 +0000 UTC m=+76.991709439" watchObservedRunningTime="2026-09-06 12:19:21.138957292 +0000 UTC m=+76.994715181" Sep 06 12:19:22.520591 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:22.520542 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:19:30.508678 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:30.508546 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-757d878946-7fxrv" Sep 06 12:19:30.508678 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:30.508623 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-757d878946-7fxrv" Sep 06 12:19:35.733118 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:35.733049 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-55ff767878-tg6ft" podUID="b1b8df13-45dd-4577-af01-5dda0cfab94f" containerName="console" containerID="cri-o://25e6a11faa86b34996d9bb1062ca245b81a77ebb3208663d4373a9889ec3943d" gracePeriod=15 Sep 06 12:19:35.966390 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:35.966367 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55ff767878-tg6ft" Sep 06 12:19:36.068025 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:36.067968 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b1b8df13-45dd-4577-af01-5dda0cfab94f-oauth-serving-cert\") pod \"b1b8df13-45dd-4577-af01-5dda0cfab94f\" (UID: \"b1b8df13-45dd-4577-af01-5dda0cfab94f\") " Sep 06 12:19:36.068025 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:36.068015 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b1b8df13-45dd-4577-af01-5dda0cfab94f-console-config\") pod \"b1b8df13-45dd-4577-af01-5dda0cfab94f\" (UID: \"b1b8df13-45dd-4577-af01-5dda0cfab94f\") " Sep 06 12:19:36.068188 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:36.068042 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b1b8df13-45dd-4577-af01-5dda0cfab94f-console-oauth-config\") pod \"b1b8df13-45dd-4577-af01-5dda0cfab94f\" (UID: \"b1b8df13-45dd-4577-af01-5dda0cfab94f\") " Sep 06 12:19:36.068188 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:36.068064 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v25km\" (UniqueName: \"kubernetes.io/projected/b1b8df13-45dd-4577-af01-5dda0cfab94f-kube-api-access-v25km\") pod \"b1b8df13-45dd-4577-af01-5dda0cfab94f\" (UID: \"b1b8df13-45dd-4577-af01-5dda0cfab94f\") " Sep 06 12:19:36.068188 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:36.068082 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1b8df13-45dd-4577-af01-5dda0cfab94f-console-serving-cert\") pod \"b1b8df13-45dd-4577-af01-5dda0cfab94f\" (UID: \"b1b8df13-45dd-4577-af01-5dda0cfab94f\") " Sep 06 12:19:36.068188 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:36.068111 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b8df13-45dd-4577-af01-5dda0cfab94f-service-ca\") pod \"b1b8df13-45dd-4577-af01-5dda0cfab94f\" (UID: \"b1b8df13-45dd-4577-af01-5dda0cfab94f\") " Sep 06 12:19:36.068422 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:36.068393 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1b8df13-45dd-4577-af01-5dda0cfab94f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b1b8df13-45dd-4577-af01-5dda0cfab94f" (UID: "b1b8df13-45dd-4577-af01-5dda0cfab94f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:19:36.068531 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:36.068498 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1b8df13-45dd-4577-af01-5dda0cfab94f-console-config" (OuterVolumeSpecName: "console-config") pod "b1b8df13-45dd-4577-af01-5dda0cfab94f" (UID: "b1b8df13-45dd-4577-af01-5dda0cfab94f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:19:36.068627 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:36.068567 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1b8df13-45dd-4577-af01-5dda0cfab94f-service-ca" (OuterVolumeSpecName: "service-ca") pod "b1b8df13-45dd-4577-af01-5dda0cfab94f" (UID: "b1b8df13-45dd-4577-af01-5dda0cfab94f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:19:36.070463 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:36.070436 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1b8df13-45dd-4577-af01-5dda0cfab94f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b1b8df13-45dd-4577-af01-5dda0cfab94f" (UID: "b1b8df13-45dd-4577-af01-5dda0cfab94f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 12:19:36.070900 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:36.070877 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b8df13-45dd-4577-af01-5dda0cfab94f-kube-api-access-v25km" (OuterVolumeSpecName: "kube-api-access-v25km") pod "b1b8df13-45dd-4577-af01-5dda0cfab94f" (UID: "b1b8df13-45dd-4577-af01-5dda0cfab94f"). InnerVolumeSpecName "kube-api-access-v25km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:19:36.070996 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:36.070955 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1b8df13-45dd-4577-af01-5dda0cfab94f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b1b8df13-45dd-4577-af01-5dda0cfab94f" (UID: "b1b8df13-45dd-4577-af01-5dda0cfab94f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 12:19:36.169244 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:36.169217 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b1b8df13-45dd-4577-af01-5dda0cfab94f-console-oauth-config\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:19:36.169244 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:36.169244 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v25km\" (UniqueName: \"kubernetes.io/projected/b1b8df13-45dd-4577-af01-5dda0cfab94f-kube-api-access-v25km\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:19:36.169394 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:36.169258 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1b8df13-45dd-4577-af01-5dda0cfab94f-console-serving-cert\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:19:36.169394 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:36.169270 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b8df13-45dd-4577-af01-5dda0cfab94f-service-ca\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:19:36.169394 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:36.169283 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b1b8df13-45dd-4577-af01-5dda0cfab94f-oauth-serving-cert\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:19:36.169394 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:36.169294 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b1b8df13-45dd-4577-af01-5dda0cfab94f-console-config\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:19:36.169868 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:36.169846 2576 generic.go:356] "Generic (PLEG): container finished" podID="b1b8df13-45dd-4577-af01-5dda0cfab94f" containerID="25e6a11faa86b34996d9bb1062ca245b81a77ebb3208663d4373a9889ec3943d" exitCode=0 Sep 06 12:19:36.170009 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:36.169900 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55ff767878-tg6ft" Sep 06 12:19:36.170009 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:36.169922 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55ff767878-tg6ft" event={"ID":"b1b8df13-45dd-4577-af01-5dda0cfab94f","Type":"ContainerDied","Data":"25e6a11faa86b34996d9bb1062ca245b81a77ebb3208663d4373a9889ec3943d"} Sep 06 12:19:36.170009 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:36.169997 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55ff767878-tg6ft" event={"ID":"b1b8df13-45dd-4577-af01-5dda0cfab94f","Type":"ContainerDied","Data":"73ba8971546d57373b95580858385875e38f3dc7aa4763a30dba404de2eb997c"} Sep 06 12:19:36.170128 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:36.170020 2576 scope.go:117] "RemoveContainer" containerID="25e6a11faa86b34996d9bb1062ca245b81a77ebb3208663d4373a9889ec3943d" Sep 06 12:19:36.178880 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:36.178863 2576 scope.go:117] "RemoveContainer" containerID="25e6a11faa86b34996d9bb1062ca245b81a77ebb3208663d4373a9889ec3943d" Sep 06 12:19:36.179160 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:19:36.179140 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25e6a11faa86b34996d9bb1062ca245b81a77ebb3208663d4373a9889ec3943d\": container with ID starting with 25e6a11faa86b34996d9bb1062ca245b81a77ebb3208663d4373a9889ec3943d not found: ID does not exist" containerID="25e6a11faa86b34996d9bb1062ca245b81a77ebb3208663d4373a9889ec3943d" Sep 06 12:19:36.179204 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:36.179168 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25e6a11faa86b34996d9bb1062ca245b81a77ebb3208663d4373a9889ec3943d"} err="failed to get container status \"25e6a11faa86b34996d9bb1062ca245b81a77ebb3208663d4373a9889ec3943d\": rpc error: code = NotFound desc = could not find container \"25e6a11faa86b34996d9bb1062ca245b81a77ebb3208663d4373a9889ec3943d\": container with ID starting with 25e6a11faa86b34996d9bb1062ca245b81a77ebb3208663d4373a9889ec3943d not found: ID does not exist" Sep 06 12:19:36.189354 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:36.189332 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55ff767878-tg6ft"] Sep 06 12:19:36.192676 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:36.192655 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55ff767878-tg6ft"] Sep 06 12:19:36.652366 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:36.652337 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1b8df13-45dd-4577-af01-5dda0cfab94f" path="/var/lib/kubelet/pods/b1b8df13-45dd-4577-af01-5dda0cfab94f/volumes" Sep 06 12:19:46.159185 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:46.159133 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8454d65567-kn6mk" podUID="bcae8b57-e511-448f-975c-08772a4af285" containerName="console" containerID="cri-o://1e7e9edff93c1f210ae9d66f3564a1c2971de569e3a1e7e34eeb699b38bd594a" gracePeriod=15 Sep 06 12:19:46.425641 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:46.425589 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8454d65567-kn6mk" Sep 06 12:19:46.439256 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:46.439233 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bcae8b57-e511-448f-975c-08772a4af285-oauth-serving-cert\") pod \"bcae8b57-e511-448f-975c-08772a4af285\" (UID: \"bcae8b57-e511-448f-975c-08772a4af285\") " Sep 06 12:19:46.439355 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:46.439278 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bcae8b57-e511-448f-975c-08772a4af285-console-serving-cert\") pod \"bcae8b57-e511-448f-975c-08772a4af285\" (UID: \"bcae8b57-e511-448f-975c-08772a4af285\") " Sep 06 12:19:46.439355 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:46.439302 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bcae8b57-e511-448f-975c-08772a4af285-console-oauth-config\") pod \"bcae8b57-e511-448f-975c-08772a4af285\" (UID: \"bcae8b57-e511-448f-975c-08772a4af285\") " Sep 06 12:19:46.439355 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:46.439339 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bcae8b57-e511-448f-975c-08772a4af285-console-config\") pod \"bcae8b57-e511-448f-975c-08772a4af285\" (UID: \"bcae8b57-e511-448f-975c-08772a4af285\") " Sep 06 12:19:46.439515 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:46.439363 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7tw5x\" (UniqueName: \"kubernetes.io/projected/bcae8b57-e511-448f-975c-08772a4af285-kube-api-access-7tw5x\") pod \"bcae8b57-e511-448f-975c-08772a4af285\" (UID: \"bcae8b57-e511-448f-975c-08772a4af285\") " Sep 06 12:19:46.439515 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:46.439384 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bcae8b57-e511-448f-975c-08772a4af285-service-ca\") pod \"bcae8b57-e511-448f-975c-08772a4af285\" (UID: \"bcae8b57-e511-448f-975c-08772a4af285\") " Sep 06 12:19:46.439515 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:46.439444 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcae8b57-e511-448f-975c-08772a4af285-trusted-ca-bundle\") pod \"bcae8b57-e511-448f-975c-08772a4af285\" (UID: \"bcae8b57-e511-448f-975c-08772a4af285\") " Sep 06 12:19:46.439710 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:46.439653 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcae8b57-e511-448f-975c-08772a4af285-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bcae8b57-e511-448f-975c-08772a4af285" (UID: "bcae8b57-e511-448f-975c-08772a4af285"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:19:46.439783 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:46.439745 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcae8b57-e511-448f-975c-08772a4af285-console-config" (OuterVolumeSpecName: "console-config") pod "bcae8b57-e511-448f-975c-08772a4af285" (UID: "bcae8b57-e511-448f-975c-08772a4af285"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:19:46.440008 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:46.439980 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcae8b57-e511-448f-975c-08772a4af285-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "bcae8b57-e511-448f-975c-08772a4af285" (UID: "bcae8b57-e511-448f-975c-08772a4af285"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:19:46.440281 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:46.440058 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcae8b57-e511-448f-975c-08772a4af285-service-ca" (OuterVolumeSpecName: "service-ca") pod "bcae8b57-e511-448f-975c-08772a4af285" (UID: "bcae8b57-e511-448f-975c-08772a4af285"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:19:46.442321 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:46.442292 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcae8b57-e511-448f-975c-08772a4af285-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bcae8b57-e511-448f-975c-08772a4af285" (UID: "bcae8b57-e511-448f-975c-08772a4af285"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 12:19:46.442556 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:46.442535 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcae8b57-e511-448f-975c-08772a4af285-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bcae8b57-e511-448f-975c-08772a4af285" (UID: "bcae8b57-e511-448f-975c-08772a4af285"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 12:19:46.442656 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:46.442629 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcae8b57-e511-448f-975c-08772a4af285-kube-api-access-7tw5x" (OuterVolumeSpecName: "kube-api-access-7tw5x") pod "bcae8b57-e511-448f-975c-08772a4af285" (UID: "bcae8b57-e511-448f-975c-08772a4af285"). InnerVolumeSpecName "kube-api-access-7tw5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:19:46.540300 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:46.540276 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bcae8b57-e511-448f-975c-08772a4af285-oauth-serving-cert\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:19:46.540300 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:46.540299 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bcae8b57-e511-448f-975c-08772a4af285-console-serving-cert\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:19:46.540462 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:46.540309 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bcae8b57-e511-448f-975c-08772a4af285-console-oauth-config\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:19:46.540462 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:46.540317 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bcae8b57-e511-448f-975c-08772a4af285-console-config\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:19:46.540462 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:46.540325 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tw5x\" (UniqueName: \"kubernetes.io/projected/bcae8b57-e511-448f-975c-08772a4af285-kube-api-access-7tw5x\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:19:46.540462 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:46.540333 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bcae8b57-e511-448f-975c-08772a4af285-service-ca\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:19:46.540462 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:46.540342 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcae8b57-e511-448f-975c-08772a4af285-trusted-ca-bundle\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:19:47.234465 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:47.234423 2576 generic.go:356] "Generic (PLEG): container finished" podID="bcae8b57-e511-448f-975c-08772a4af285" containerID="1e7e9edff93c1f210ae9d66f3564a1c2971de569e3a1e7e34eeb699b38bd594a" exitCode=0 Sep 06 12:19:47.234946 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:47.234469 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8454d65567-kn6mk" Sep 06 12:19:47.234946 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:47.234472 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8454d65567-kn6mk" event={"ID":"bcae8b57-e511-448f-975c-08772a4af285","Type":"ContainerDied","Data":"1e7e9edff93c1f210ae9d66f3564a1c2971de569e3a1e7e34eeb699b38bd594a"} Sep 06 12:19:47.234946 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:47.234595 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8454d65567-kn6mk" event={"ID":"bcae8b57-e511-448f-975c-08772a4af285","Type":"ContainerDied","Data":"b7be584a3b985072a51b723a4c17951286d5851c17448cc0f3eedd6196e3319a"} Sep 06 12:19:47.234946 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:47.234622 2576 scope.go:117] "RemoveContainer" containerID="1e7e9edff93c1f210ae9d66f3564a1c2971de569e3a1e7e34eeb699b38bd594a" Sep 06 12:19:47.246969 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:47.246949 2576 scope.go:117] "RemoveContainer" containerID="1e7e9edff93c1f210ae9d66f3564a1c2971de569e3a1e7e34eeb699b38bd594a" Sep 06 12:19:47.247555 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:19:47.247530 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e7e9edff93c1f210ae9d66f3564a1c2971de569e3a1e7e34eeb699b38bd594a\": container with ID starting with 1e7e9edff93c1f210ae9d66f3564a1c2971de569e3a1e7e34eeb699b38bd594a not found: ID does not exist" containerID="1e7e9edff93c1f210ae9d66f3564a1c2971de569e3a1e7e34eeb699b38bd594a" Sep 06 12:19:47.247651 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:47.247565 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e7e9edff93c1f210ae9d66f3564a1c2971de569e3a1e7e34eeb699b38bd594a"} err="failed to get container status \"1e7e9edff93c1f210ae9d66f3564a1c2971de569e3a1e7e34eeb699b38bd594a\": rpc error: code = NotFound desc = could not find container \"1e7e9edff93c1f210ae9d66f3564a1c2971de569e3a1e7e34eeb699b38bd594a\": container with ID starting with 1e7e9edff93c1f210ae9d66f3564a1c2971de569e3a1e7e34eeb699b38bd594a not found: ID does not exist" Sep 06 12:19:47.248996 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:47.248977 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8454d65567-kn6mk"] Sep 06 12:19:47.252005 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:47.251988 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8454d65567-kn6mk"] Sep 06 12:19:48.073904 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:48.073878 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gpxpp" Sep 06 12:19:48.652169 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:48.652124 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcae8b57-e511-448f-975c-08772a4af285" path="/var/lib/kubelet/pods/bcae8b57-e511-448f-975c-08772a4af285/volumes" Sep 06 12:19:50.514571 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:50.514537 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-757d878946-7fxrv" Sep 06 12:19:50.518548 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:19:50.518522 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-757d878946-7fxrv" Sep 06 12:20:12.520563 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:12.520518 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:20:12.542995 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:12.542964 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:20:13.387803 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:13.387775 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 06 12:20:34.635587 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:34.635545 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d86f95d48-4cbrm"] Sep 06 12:20:34.636170 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:34.635972 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1b8df13-45dd-4577-af01-5dda0cfab94f" containerName="console" Sep 06 12:20:34.636170 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:34.635990 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1b8df13-45dd-4577-af01-5dda0cfab94f" containerName="console" Sep 06 12:20:34.636170 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:34.636011 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcae8b57-e511-448f-975c-08772a4af285" containerName="console" Sep 06 12:20:34.636170 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:34.636021 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcae8b57-e511-448f-975c-08772a4af285" containerName="console" Sep 06 12:20:34.636170 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:34.636124 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcae8b57-e511-448f-975c-08772a4af285" containerName="console" Sep 06 12:20:34.636170 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:34.636140 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1b8df13-45dd-4577-af01-5dda0cfab94f" containerName="console" Sep 06 12:20:34.638407 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:34.638385 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d86f95d48-4cbrm" Sep 06 12:20:34.652893 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:34.652868 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d86f95d48-4cbrm"] Sep 06 12:20:34.809909 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:34.809873 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f25b4f1d-727c-42e7-963c-5669283addb5-service-ca\") pod \"console-6d86f95d48-4cbrm\" (UID: \"f25b4f1d-727c-42e7-963c-5669283addb5\") " pod="openshift-console/console-6d86f95d48-4cbrm" Sep 06 12:20:34.810051 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:34.809914 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f25b4f1d-727c-42e7-963c-5669283addb5-console-oauth-config\") pod \"console-6d86f95d48-4cbrm\" (UID: \"f25b4f1d-727c-42e7-963c-5669283addb5\") " pod="openshift-console/console-6d86f95d48-4cbrm" Sep 06 12:20:34.810051 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:34.809992 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f25b4f1d-727c-42e7-963c-5669283addb5-console-config\") pod \"console-6d86f95d48-4cbrm\" (UID: \"f25b4f1d-727c-42e7-963c-5669283addb5\") " pod="openshift-console/console-6d86f95d48-4cbrm" Sep 06 12:20:34.810051 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:34.810034 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f25b4f1d-727c-42e7-963c-5669283addb5-trusted-ca-bundle\") pod \"console-6d86f95d48-4cbrm\" (UID: \"f25b4f1d-727c-42e7-963c-5669283addb5\") " pod="openshift-console/console-6d86f95d48-4cbrm" Sep 06 12:20:34.810230 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:34.810055 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f25b4f1d-727c-42e7-963c-5669283addb5-oauth-serving-cert\") pod \"console-6d86f95d48-4cbrm\" (UID: \"f25b4f1d-727c-42e7-963c-5669283addb5\") " pod="openshift-console/console-6d86f95d48-4cbrm" Sep 06 12:20:34.810230 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:34.810155 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f25b4f1d-727c-42e7-963c-5669283addb5-console-serving-cert\") pod \"console-6d86f95d48-4cbrm\" (UID: \"f25b4f1d-727c-42e7-963c-5669283addb5\") " pod="openshift-console/console-6d86f95d48-4cbrm" Sep 06 12:20:34.810230 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:34.810210 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7g8j\" (UniqueName: \"kubernetes.io/projected/f25b4f1d-727c-42e7-963c-5669283addb5-kube-api-access-m7g8j\") pod \"console-6d86f95d48-4cbrm\" (UID: \"f25b4f1d-727c-42e7-963c-5669283addb5\") " pod="openshift-console/console-6d86f95d48-4cbrm" Sep 06 12:20:34.911321 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:34.911297 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f25b4f1d-727c-42e7-963c-5669283addb5-console-serving-cert\") pod \"console-6d86f95d48-4cbrm\" (UID: \"f25b4f1d-727c-42e7-963c-5669283addb5\") " pod="openshift-console/console-6d86f95d48-4cbrm" Sep 06 12:20:34.911429 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:34.911330 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7g8j\" (UniqueName: \"kubernetes.io/projected/f25b4f1d-727c-42e7-963c-5669283addb5-kube-api-access-m7g8j\") pod \"console-6d86f95d48-4cbrm\" (UID: \"f25b4f1d-727c-42e7-963c-5669283addb5\") " pod="openshift-console/console-6d86f95d48-4cbrm" Sep 06 12:20:34.911429 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:34.911367 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f25b4f1d-727c-42e7-963c-5669283addb5-service-ca\") pod \"console-6d86f95d48-4cbrm\" (UID: \"f25b4f1d-727c-42e7-963c-5669283addb5\") " pod="openshift-console/console-6d86f95d48-4cbrm" Sep 06 12:20:34.911429 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:34.911383 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f25b4f1d-727c-42e7-963c-5669283addb5-console-oauth-config\") pod \"console-6d86f95d48-4cbrm\" (UID: \"f25b4f1d-727c-42e7-963c-5669283addb5\") " pod="openshift-console/console-6d86f95d48-4cbrm" Sep 06 12:20:34.911429 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:34.911406 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f25b4f1d-727c-42e7-963c-5669283addb5-console-config\") pod \"console-6d86f95d48-4cbrm\" (UID: \"f25b4f1d-727c-42e7-963c-5669283addb5\") " pod="openshift-console/console-6d86f95d48-4cbrm" Sep 06 12:20:34.911630 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:34.911501 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f25b4f1d-727c-42e7-963c-5669283addb5-trusted-ca-bundle\") pod \"console-6d86f95d48-4cbrm\" (UID: \"f25b4f1d-727c-42e7-963c-5669283addb5\") " pod="openshift-console/console-6d86f95d48-4cbrm" Sep 06 12:20:34.911630 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:34.911540 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f25b4f1d-727c-42e7-963c-5669283addb5-oauth-serving-cert\") pod \"console-6d86f95d48-4cbrm\" (UID: \"f25b4f1d-727c-42e7-963c-5669283addb5\") " pod="openshift-console/console-6d86f95d48-4cbrm" Sep 06 12:20:34.912240 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:34.912158 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f25b4f1d-727c-42e7-963c-5669283addb5-service-ca\") pod \"console-6d86f95d48-4cbrm\" (UID: \"f25b4f1d-727c-42e7-963c-5669283addb5\") " pod="openshift-console/console-6d86f95d48-4cbrm" Sep 06 12:20:34.912240 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:34.912194 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f25b4f1d-727c-42e7-963c-5669283addb5-console-config\") pod \"console-6d86f95d48-4cbrm\" (UID: \"f25b4f1d-727c-42e7-963c-5669283addb5\") " pod="openshift-console/console-6d86f95d48-4cbrm" Sep 06 12:20:34.912240 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:34.912211 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f25b4f1d-727c-42e7-963c-5669283addb5-oauth-serving-cert\") pod \"console-6d86f95d48-4cbrm\" (UID: \"f25b4f1d-727c-42e7-963c-5669283addb5\") " pod="openshift-console/console-6d86f95d48-4cbrm" Sep 06 12:20:34.912485 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:34.912357 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f25b4f1d-727c-42e7-963c-5669283addb5-trusted-ca-bundle\") pod \"console-6d86f95d48-4cbrm\" (UID: \"f25b4f1d-727c-42e7-963c-5669283addb5\") " pod="openshift-console/console-6d86f95d48-4cbrm" Sep 06 12:20:34.913998 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:34.913973 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f25b4f1d-727c-42e7-963c-5669283addb5-console-oauth-config\") pod \"console-6d86f95d48-4cbrm\" (UID: \"f25b4f1d-727c-42e7-963c-5669283addb5\") " pod="openshift-console/console-6d86f95d48-4cbrm" Sep 06 12:20:34.914196 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:34.914176 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f25b4f1d-727c-42e7-963c-5669283addb5-console-serving-cert\") pod \"console-6d86f95d48-4cbrm\" (UID: \"f25b4f1d-727c-42e7-963c-5669283addb5\") " pod="openshift-console/console-6d86f95d48-4cbrm" Sep 06 12:20:34.918601 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:34.918585 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7g8j\" (UniqueName: \"kubernetes.io/projected/f25b4f1d-727c-42e7-963c-5669283addb5-kube-api-access-m7g8j\") pod \"console-6d86f95d48-4cbrm\" (UID: \"f25b4f1d-727c-42e7-963c-5669283addb5\") " pod="openshift-console/console-6d86f95d48-4cbrm" Sep 06 12:20:34.949225 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:34.949205 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d86f95d48-4cbrm" Sep 06 12:20:35.219559 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:35.219464 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d86f95d48-4cbrm"] Sep 06 12:20:35.223008 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:20:35.222976 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf25b4f1d_727c_42e7_963c_5669283addb5.slice/crio-43665e19598ebe646597d80c2f71605d0e2133a9d09c7279d65fa7d61c6e2308 WatchSource:0}: Error finding container 43665e19598ebe646597d80c2f71605d0e2133a9d09c7279d65fa7d61c6e2308: Status 404 returned error can't find the container with id 43665e19598ebe646597d80c2f71605d0e2133a9d09c7279d65fa7d61c6e2308 Sep 06 12:20:35.465803 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:35.465744 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d86f95d48-4cbrm" event={"ID":"f25b4f1d-727c-42e7-963c-5669283addb5","Type":"ContainerStarted","Data":"ab3b3803c0f2e6d3021cde049753fba4848c094406709713c16fafe7f8866d55"} Sep 06 12:20:35.465987 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:35.465831 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d86f95d48-4cbrm" event={"ID":"f25b4f1d-727c-42e7-963c-5669283addb5","Type":"ContainerStarted","Data":"43665e19598ebe646597d80c2f71605d0e2133a9d09c7279d65fa7d61c6e2308"} Sep 06 12:20:35.480914 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:35.480814 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d86f95d48-4cbrm" podStartSLOduration=1.480793375 podStartE2EDuration="1.480793375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:20:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:20:35.480501052 +0000 UTC m=+151.336258945" watchObservedRunningTime="2026-09-06 12:20:35.480793375 +0000 UTC m=+151.336551264" Sep 06 12:20:42.154052 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:42.154011 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d86f95d48-4cbrm"] Sep 06 12:20:42.188837 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:42.188806 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-df67f944b-ttjpg"] Sep 06 12:20:42.191032 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:42.191017 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-df67f944b-ttjpg" Sep 06 12:20:42.200320 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:42.200297 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-df67f944b-ttjpg"] Sep 06 12:20:42.369123 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:42.369085 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r8wp\" (UniqueName: \"kubernetes.io/projected/17ee86bd-4799-47e1-a2d6-268938f99941-kube-api-access-8r8wp\") pod \"console-df67f944b-ttjpg\" (UID: \"17ee86bd-4799-47e1-a2d6-268938f99941\") " pod="openshift-console/console-df67f944b-ttjpg" Sep 06 12:20:42.369288 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:42.369134 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17ee86bd-4799-47e1-a2d6-268938f99941-console-oauth-config\") pod \"console-df67f944b-ttjpg\" (UID: \"17ee86bd-4799-47e1-a2d6-268938f99941\") " pod="openshift-console/console-df67f944b-ttjpg" Sep 06 12:20:42.369288 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:42.369161 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17ee86bd-4799-47e1-a2d6-268938f99941-oauth-serving-cert\") pod \"console-df67f944b-ttjpg\" (UID: \"17ee86bd-4799-47e1-a2d6-268938f99941\") " pod="openshift-console/console-df67f944b-ttjpg" Sep 06 12:20:42.369288 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:42.369187 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17ee86bd-4799-47e1-a2d6-268938f99941-console-serving-cert\") pod \"console-df67f944b-ttjpg\" (UID: \"17ee86bd-4799-47e1-a2d6-268938f99941\") " pod="openshift-console/console-df67f944b-ttjpg" Sep 06 12:20:42.369288 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:42.369254 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17ee86bd-4799-47e1-a2d6-268938f99941-console-config\") pod \"console-df67f944b-ttjpg\" (UID: \"17ee86bd-4799-47e1-a2d6-268938f99941\") " pod="openshift-console/console-df67f944b-ttjpg" Sep 06 12:20:42.369288 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:42.369276 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17ee86bd-4799-47e1-a2d6-268938f99941-trusted-ca-bundle\") pod \"console-df67f944b-ttjpg\" (UID: \"17ee86bd-4799-47e1-a2d6-268938f99941\") " pod="openshift-console/console-df67f944b-ttjpg" Sep 06 12:20:42.369452 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:42.369296 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17ee86bd-4799-47e1-a2d6-268938f99941-service-ca\") pod \"console-df67f944b-ttjpg\" (UID: \"17ee86bd-4799-47e1-a2d6-268938f99941\") " pod="openshift-console/console-df67f944b-ttjpg" Sep 06 12:20:42.470599 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:42.470525 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17ee86bd-4799-47e1-a2d6-268938f99941-console-config\") pod \"console-df67f944b-ttjpg\" (UID: \"17ee86bd-4799-47e1-a2d6-268938f99941\") " pod="openshift-console/console-df67f944b-ttjpg" Sep 06 12:20:42.470599 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:42.470564 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17ee86bd-4799-47e1-a2d6-268938f99941-trusted-ca-bundle\") pod \"console-df67f944b-ttjpg\" (UID: \"17ee86bd-4799-47e1-a2d6-268938f99941\") " pod="openshift-console/console-df67f944b-ttjpg" Sep 06 12:20:42.470599 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:42.470587 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17ee86bd-4799-47e1-a2d6-268938f99941-service-ca\") pod \"console-df67f944b-ttjpg\" (UID: \"17ee86bd-4799-47e1-a2d6-268938f99941\") " pod="openshift-console/console-df67f944b-ttjpg" Sep 06 12:20:42.470889 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:42.470631 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r8wp\" (UniqueName: \"kubernetes.io/projected/17ee86bd-4799-47e1-a2d6-268938f99941-kube-api-access-8r8wp\") pod \"console-df67f944b-ttjpg\" (UID: \"17ee86bd-4799-47e1-a2d6-268938f99941\") " pod="openshift-console/console-df67f944b-ttjpg" Sep 06 12:20:42.470889 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:42.470654 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17ee86bd-4799-47e1-a2d6-268938f99941-console-oauth-config\") pod \"console-df67f944b-ttjpg\" (UID: \"17ee86bd-4799-47e1-a2d6-268938f99941\") " pod="openshift-console/console-df67f944b-ttjpg" Sep 06 12:20:42.470889 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:42.470732 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17ee86bd-4799-47e1-a2d6-268938f99941-oauth-serving-cert\") pod \"console-df67f944b-ttjpg\" (UID: \"17ee86bd-4799-47e1-a2d6-268938f99941\") " pod="openshift-console/console-df67f944b-ttjpg" Sep 06 12:20:42.470889 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:42.470797 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17ee86bd-4799-47e1-a2d6-268938f99941-console-serving-cert\") pod \"console-df67f944b-ttjpg\" (UID: \"17ee86bd-4799-47e1-a2d6-268938f99941\") " pod="openshift-console/console-df67f944b-ttjpg" Sep 06 12:20:42.471426 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:42.471394 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17ee86bd-4799-47e1-a2d6-268938f99941-console-config\") pod \"console-df67f944b-ttjpg\" (UID: \"17ee86bd-4799-47e1-a2d6-268938f99941\") " pod="openshift-console/console-df67f944b-ttjpg" Sep 06 12:20:42.471557 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:42.471448 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17ee86bd-4799-47e1-a2d6-268938f99941-service-ca\") pod \"console-df67f944b-ttjpg\" (UID: \"17ee86bd-4799-47e1-a2d6-268938f99941\") " pod="openshift-console/console-df67f944b-ttjpg" Sep 06 12:20:42.471557 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:42.471540 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17ee86bd-4799-47e1-a2d6-268938f99941-trusted-ca-bundle\") pod \"console-df67f944b-ttjpg\" (UID: \"17ee86bd-4799-47e1-a2d6-268938f99941\") " pod="openshift-console/console-df67f944b-ttjpg" Sep 06 12:20:42.471636 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:42.471538 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17ee86bd-4799-47e1-a2d6-268938f99941-oauth-serving-cert\") pod \"console-df67f944b-ttjpg\" (UID: \"17ee86bd-4799-47e1-a2d6-268938f99941\") " pod="openshift-console/console-df67f944b-ttjpg" Sep 06 12:20:42.473461 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:42.473441 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17ee86bd-4799-47e1-a2d6-268938f99941-console-serving-cert\") pod \"console-df67f944b-ttjpg\" (UID: \"17ee86bd-4799-47e1-a2d6-268938f99941\") " pod="openshift-console/console-df67f944b-ttjpg" Sep 06 12:20:42.473667 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:42.473641 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17ee86bd-4799-47e1-a2d6-268938f99941-console-oauth-config\") pod \"console-df67f944b-ttjpg\" (UID: \"17ee86bd-4799-47e1-a2d6-268938f99941\") " pod="openshift-console/console-df67f944b-ttjpg" Sep 06 12:20:42.478122 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:42.478094 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r8wp\" (UniqueName: \"kubernetes.io/projected/17ee86bd-4799-47e1-a2d6-268938f99941-kube-api-access-8r8wp\") pod \"console-df67f944b-ttjpg\" (UID: \"17ee86bd-4799-47e1-a2d6-268938f99941\") " pod="openshift-console/console-df67f944b-ttjpg" Sep 06 12:20:42.501674 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:42.501647 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-df67f944b-ttjpg" Sep 06 12:20:42.770267 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:42.770211 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-df67f944b-ttjpg"] Sep 06 12:20:42.772499 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:20:42.772474 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17ee86bd_4799_47e1_a2d6_268938f99941.slice/crio-1ff5117ac356ebb32dec1448f60e22ddcd2c29171e2e8615fd1359d42f23eb04 WatchSource:0}: Error finding container 1ff5117ac356ebb32dec1448f60e22ddcd2c29171e2e8615fd1359d42f23eb04: Status 404 returned error can't find the container with id 1ff5117ac356ebb32dec1448f60e22ddcd2c29171e2e8615fd1359d42f23eb04 Sep 06 12:20:43.501927 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:43.501883 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-df67f944b-ttjpg" event={"ID":"17ee86bd-4799-47e1-a2d6-268938f99941","Type":"ContainerStarted","Data":"939f49be8d5815103fd25b2dec9f476fc5a765f0a29be4995d354559f8e25547"} Sep 06 12:20:43.502380 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:43.501952 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-df67f944b-ttjpg" event={"ID":"17ee86bd-4799-47e1-a2d6-268938f99941","Type":"ContainerStarted","Data":"1ff5117ac356ebb32dec1448f60e22ddcd2c29171e2e8615fd1359d42f23eb04"} Sep 06 12:20:43.517777 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:43.517712 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-df67f944b-ttjpg" podStartSLOduration=1.517696672 podStartE2EDuration="1.517696672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:20:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:20:43.516694074 +0000 UTC m=+159.372451985" watchObservedRunningTime="2026-09-06 12:20:43.517696672 +0000 UTC m=+159.373454559" Sep 06 12:20:44.949565 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:44.949527 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d86f95d48-4cbrm" Sep 06 12:20:52.502807 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:52.502768 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-df67f944b-ttjpg" Sep 06 12:20:52.503302 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:52.502835 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-df67f944b-ttjpg" Sep 06 12:20:52.507899 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:52.507874 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-df67f944b-ttjpg" Sep 06 12:20:52.546426 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:52.546394 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-df67f944b-ttjpg" Sep 06 12:20:52.592831 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:20:52.592809 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dc4c856bf-2pjhb"] Sep 06 12:21:07.174388 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:07.174330 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6d86f95d48-4cbrm" podUID="f25b4f1d-727c-42e7-963c-5669283addb5" containerName="console" containerID="cri-o://ab3b3803c0f2e6d3021cde049753fba4848c094406709713c16fafe7f8866d55" gracePeriod=15 Sep 06 12:21:07.423575 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:07.423553 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d86f95d48-4cbrm" Sep 06 12:21:07.452949 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:07.452886 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7g8j\" (UniqueName: \"kubernetes.io/projected/f25b4f1d-727c-42e7-963c-5669283addb5-kube-api-access-m7g8j\") pod \"f25b4f1d-727c-42e7-963c-5669283addb5\" (UID: \"f25b4f1d-727c-42e7-963c-5669283addb5\") " Sep 06 12:21:07.452949 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:07.452946 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f25b4f1d-727c-42e7-963c-5669283addb5-service-ca\") pod \"f25b4f1d-727c-42e7-963c-5669283addb5\" (UID: \"f25b4f1d-727c-42e7-963c-5669283addb5\") " Sep 06 12:21:07.453125 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:07.452971 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f25b4f1d-727c-42e7-963c-5669283addb5-console-config\") pod \"f25b4f1d-727c-42e7-963c-5669283addb5\" (UID: \"f25b4f1d-727c-42e7-963c-5669283addb5\") " Sep 06 12:21:07.453125 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:07.453023 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f25b4f1d-727c-42e7-963c-5669283addb5-console-serving-cert\") pod \"f25b4f1d-727c-42e7-963c-5669283addb5\" (UID: \"f25b4f1d-727c-42e7-963c-5669283addb5\") " Sep 06 12:21:07.453125 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:07.453055 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f25b4f1d-727c-42e7-963c-5669283addb5-oauth-serving-cert\") pod \"f25b4f1d-727c-42e7-963c-5669283addb5\" (UID: \"f25b4f1d-727c-42e7-963c-5669283addb5\") " Sep 06 12:21:07.453125 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:07.453089 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f25b4f1d-727c-42e7-963c-5669283addb5-console-oauth-config\") pod \"f25b4f1d-727c-42e7-963c-5669283addb5\" (UID: \"f25b4f1d-727c-42e7-963c-5669283addb5\") " Sep 06 12:21:07.453324 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:07.453139 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f25b4f1d-727c-42e7-963c-5669283addb5-trusted-ca-bundle\") pod \"f25b4f1d-727c-42e7-963c-5669283addb5\" (UID: \"f25b4f1d-727c-42e7-963c-5669283addb5\") " Sep 06 12:21:07.453476 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:07.453387 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f25b4f1d-727c-42e7-963c-5669283addb5-service-ca" (OuterVolumeSpecName: "service-ca") pod "f25b4f1d-727c-42e7-963c-5669283addb5" (UID: "f25b4f1d-727c-42e7-963c-5669283addb5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:21:07.453579 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:07.453554 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f25b4f1d-727c-42e7-963c-5669283addb5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f25b4f1d-727c-42e7-963c-5669283addb5" (UID: "f25b4f1d-727c-42e7-963c-5669283addb5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:21:07.453828 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:07.453656 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f25b4f1d-727c-42e7-963c-5669283addb5-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f25b4f1d-727c-42e7-963c-5669283addb5" (UID: "f25b4f1d-727c-42e7-963c-5669283addb5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:21:07.453973 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:07.453952 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f25b4f1d-727c-42e7-963c-5669283addb5-console-config" (OuterVolumeSpecName: "console-config") pod "f25b4f1d-727c-42e7-963c-5669283addb5" (UID: "f25b4f1d-727c-42e7-963c-5669283addb5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:21:07.455304 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:07.455283 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f25b4f1d-727c-42e7-963c-5669283addb5-kube-api-access-m7g8j" (OuterVolumeSpecName: "kube-api-access-m7g8j") pod "f25b4f1d-727c-42e7-963c-5669283addb5" (UID: "f25b4f1d-727c-42e7-963c-5669283addb5"). InnerVolumeSpecName "kube-api-access-m7g8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:21:07.455490 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:07.455476 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f25b4f1d-727c-42e7-963c-5669283addb5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f25b4f1d-727c-42e7-963c-5669283addb5" (UID: "f25b4f1d-727c-42e7-963c-5669283addb5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 12:21:07.455943 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:07.455908 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f25b4f1d-727c-42e7-963c-5669283addb5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f25b4f1d-727c-42e7-963c-5669283addb5" (UID: "f25b4f1d-727c-42e7-963c-5669283addb5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 12:21:07.553983 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:07.553954 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f25b4f1d-727c-42e7-963c-5669283addb5-console-oauth-config\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:21:07.553983 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:07.553979 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f25b4f1d-727c-42e7-963c-5669283addb5-trusted-ca-bundle\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:21:07.554140 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:07.553992 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7g8j\" (UniqueName: \"kubernetes.io/projected/f25b4f1d-727c-42e7-963c-5669283addb5-kube-api-access-m7g8j\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:21:07.554140 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:07.554005 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f25b4f1d-727c-42e7-963c-5669283addb5-service-ca\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:21:07.554140 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:07.554016 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f25b4f1d-727c-42e7-963c-5669283addb5-console-config\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:21:07.554140 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:07.554027 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f25b4f1d-727c-42e7-963c-5669283addb5-console-serving-cert\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:21:07.554140 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:07.554038 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f25b4f1d-727c-42e7-963c-5669283addb5-oauth-serving-cert\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:21:07.604708 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:07.604680 2576 generic.go:356] "Generic (PLEG): container finished" podID="f25b4f1d-727c-42e7-963c-5669283addb5" containerID="ab3b3803c0f2e6d3021cde049753fba4848c094406709713c16fafe7f8866d55" exitCode=0 Sep 06 12:21:07.604842 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:07.604733 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d86f95d48-4cbrm" Sep 06 12:21:07.604842 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:07.604794 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d86f95d48-4cbrm" event={"ID":"f25b4f1d-727c-42e7-963c-5669283addb5","Type":"ContainerDied","Data":"ab3b3803c0f2e6d3021cde049753fba4848c094406709713c16fafe7f8866d55"} Sep 06 12:21:07.604921 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:07.604851 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d86f95d48-4cbrm" event={"ID":"f25b4f1d-727c-42e7-963c-5669283addb5","Type":"ContainerDied","Data":"43665e19598ebe646597d80c2f71605d0e2133a9d09c7279d65fa7d61c6e2308"} Sep 06 12:21:07.604921 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:07.604872 2576 scope.go:117] "RemoveContainer" containerID="ab3b3803c0f2e6d3021cde049753fba4848c094406709713c16fafe7f8866d55" Sep 06 12:21:07.614008 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:07.613989 2576 scope.go:117] "RemoveContainer" containerID="ab3b3803c0f2e6d3021cde049753fba4848c094406709713c16fafe7f8866d55" Sep 06 12:21:07.614289 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:21:07.614272 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab3b3803c0f2e6d3021cde049753fba4848c094406709713c16fafe7f8866d55\": container with ID starting with ab3b3803c0f2e6d3021cde049753fba4848c094406709713c16fafe7f8866d55 not found: ID does not exist" containerID="ab3b3803c0f2e6d3021cde049753fba4848c094406709713c16fafe7f8866d55" Sep 06 12:21:07.614329 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:07.614298 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab3b3803c0f2e6d3021cde049753fba4848c094406709713c16fafe7f8866d55"} err="failed to get container status \"ab3b3803c0f2e6d3021cde049753fba4848c094406709713c16fafe7f8866d55\": rpc error: code = NotFound desc = could not find container \"ab3b3803c0f2e6d3021cde049753fba4848c094406709713c16fafe7f8866d55\": container with ID starting with ab3b3803c0f2e6d3021cde049753fba4848c094406709713c16fafe7f8866d55 not found: ID does not exist" Sep 06 12:21:07.625121 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:07.625095 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d86f95d48-4cbrm"] Sep 06 12:21:07.628775 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:07.628737 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d86f95d48-4cbrm"] Sep 06 12:21:08.651820 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:08.651787 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f25b4f1d-727c-42e7-963c-5669283addb5" path="/var/lib/kubelet/pods/f25b4f1d-727c-42e7-963c-5669283addb5/volumes" Sep 06 12:21:17.618954 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:17.618911 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7dc4c856bf-2pjhb" podUID="393ca7da-6bc7-4f06-9191-d9dee8f5623b" containerName="console" containerID="cri-o://ef2f6d9dd6210987e923fdae132141a92698a4d1fad0f31e7429f951c3341c5c" gracePeriod=15 Sep 06 12:21:17.850920 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:17.850898 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dc4c856bf-2pjhb" Sep 06 12:21:17.929599 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:17.929563 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/393ca7da-6bc7-4f06-9191-d9dee8f5623b-trusted-ca-bundle\") pod \"393ca7da-6bc7-4f06-9191-d9dee8f5623b\" (UID: \"393ca7da-6bc7-4f06-9191-d9dee8f5623b\") " Sep 06 12:21:17.929741 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:17.929633 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/393ca7da-6bc7-4f06-9191-d9dee8f5623b-console-oauth-config\") pod \"393ca7da-6bc7-4f06-9191-d9dee8f5623b\" (UID: \"393ca7da-6bc7-4f06-9191-d9dee8f5623b\") " Sep 06 12:21:17.929741 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:17.929657 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znv7t\" (UniqueName: \"kubernetes.io/projected/393ca7da-6bc7-4f06-9191-d9dee8f5623b-kube-api-access-znv7t\") pod \"393ca7da-6bc7-4f06-9191-d9dee8f5623b\" (UID: \"393ca7da-6bc7-4f06-9191-d9dee8f5623b\") " Sep 06 12:21:17.929741 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:17.929686 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/393ca7da-6bc7-4f06-9191-d9dee8f5623b-console-serving-cert\") pod \"393ca7da-6bc7-4f06-9191-d9dee8f5623b\" (UID: \"393ca7da-6bc7-4f06-9191-d9dee8f5623b\") " Sep 06 12:21:17.929741 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:17.929701 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/393ca7da-6bc7-4f06-9191-d9dee8f5623b-oauth-serving-cert\") pod \"393ca7da-6bc7-4f06-9191-d9dee8f5623b\" (UID: \"393ca7da-6bc7-4f06-9191-d9dee8f5623b\") " Sep 06 12:21:17.929741 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:17.929741 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/393ca7da-6bc7-4f06-9191-d9dee8f5623b-service-ca\") pod \"393ca7da-6bc7-4f06-9191-d9dee8f5623b\" (UID: \"393ca7da-6bc7-4f06-9191-d9dee8f5623b\") " Sep 06 12:21:17.930035 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:17.929832 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/393ca7da-6bc7-4f06-9191-d9dee8f5623b-console-config\") pod \"393ca7da-6bc7-4f06-9191-d9dee8f5623b\" (UID: \"393ca7da-6bc7-4f06-9191-d9dee8f5623b\") " Sep 06 12:21:17.930035 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:17.929951 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/393ca7da-6bc7-4f06-9191-d9dee8f5623b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "393ca7da-6bc7-4f06-9191-d9dee8f5623b" (UID: "393ca7da-6bc7-4f06-9191-d9dee8f5623b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:21:17.930136 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:17.930106 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/393ca7da-6bc7-4f06-9191-d9dee8f5623b-trusted-ca-bundle\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:21:17.930198 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:17.930140 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/393ca7da-6bc7-4f06-9191-d9dee8f5623b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "393ca7da-6bc7-4f06-9191-d9dee8f5623b" (UID: "393ca7da-6bc7-4f06-9191-d9dee8f5623b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:21:17.930328 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:17.930306 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/393ca7da-6bc7-4f06-9191-d9dee8f5623b-service-ca" (OuterVolumeSpecName: "service-ca") pod "393ca7da-6bc7-4f06-9191-d9dee8f5623b" (UID: "393ca7da-6bc7-4f06-9191-d9dee8f5623b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:21:17.930393 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:17.930362 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/393ca7da-6bc7-4f06-9191-d9dee8f5623b-console-config" (OuterVolumeSpecName: "console-config") pod "393ca7da-6bc7-4f06-9191-d9dee8f5623b" (UID: "393ca7da-6bc7-4f06-9191-d9dee8f5623b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:21:17.932003 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:17.931982 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/393ca7da-6bc7-4f06-9191-d9dee8f5623b-kube-api-access-znv7t" (OuterVolumeSpecName: "kube-api-access-znv7t") pod "393ca7da-6bc7-4f06-9191-d9dee8f5623b" (UID: "393ca7da-6bc7-4f06-9191-d9dee8f5623b"). InnerVolumeSpecName "kube-api-access-znv7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:21:17.932421 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:17.932402 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/393ca7da-6bc7-4f06-9191-d9dee8f5623b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "393ca7da-6bc7-4f06-9191-d9dee8f5623b" (UID: "393ca7da-6bc7-4f06-9191-d9dee8f5623b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 12:21:17.932533 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:17.932473 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/393ca7da-6bc7-4f06-9191-d9dee8f5623b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "393ca7da-6bc7-4f06-9191-d9dee8f5623b" (UID: "393ca7da-6bc7-4f06-9191-d9dee8f5623b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 12:21:18.030741 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:18.030712 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/393ca7da-6bc7-4f06-9191-d9dee8f5623b-console-config\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:21:18.030741 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:18.030741 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/393ca7da-6bc7-4f06-9191-d9dee8f5623b-console-oauth-config\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:21:18.030883 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:18.030772 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znv7t\" (UniqueName: \"kubernetes.io/projected/393ca7da-6bc7-4f06-9191-d9dee8f5623b-kube-api-access-znv7t\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:21:18.030883 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:18.030782 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/393ca7da-6bc7-4f06-9191-d9dee8f5623b-console-serving-cert\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:21:18.030883 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:18.030790 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/393ca7da-6bc7-4f06-9191-d9dee8f5623b-oauth-serving-cert\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:21:18.030883 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:18.030798 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/393ca7da-6bc7-4f06-9191-d9dee8f5623b-service-ca\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:21:18.655148 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:18.655101 2576 generic.go:356] "Generic (PLEG): container finished" podID="393ca7da-6bc7-4f06-9191-d9dee8f5623b" containerID="ef2f6d9dd6210987e923fdae132141a92698a4d1fad0f31e7429f951c3341c5c" exitCode=0 Sep 06 12:21:18.655148 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:18.655142 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dc4c856bf-2pjhb" Sep 06 12:21:18.655657 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:18.655188 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dc4c856bf-2pjhb" event={"ID":"393ca7da-6bc7-4f06-9191-d9dee8f5623b","Type":"ContainerDied","Data":"ef2f6d9dd6210987e923fdae132141a92698a4d1fad0f31e7429f951c3341c5c"} Sep 06 12:21:18.655657 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:18.655251 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dc4c856bf-2pjhb" event={"ID":"393ca7da-6bc7-4f06-9191-d9dee8f5623b","Type":"ContainerDied","Data":"28509715a972d82609b845b62343078f0be8d9b44c761ec591192ad0f9333945"} Sep 06 12:21:18.655657 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:18.655285 2576 scope.go:117] "RemoveContainer" containerID="ef2f6d9dd6210987e923fdae132141a92698a4d1fad0f31e7429f951c3341c5c" Sep 06 12:21:18.665209 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:18.665193 2576 scope.go:117] "RemoveContainer" containerID="ef2f6d9dd6210987e923fdae132141a92698a4d1fad0f31e7429f951c3341c5c" Sep 06 12:21:18.665515 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:21:18.665492 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef2f6d9dd6210987e923fdae132141a92698a4d1fad0f31e7429f951c3341c5c\": container with ID starting with ef2f6d9dd6210987e923fdae132141a92698a4d1fad0f31e7429f951c3341c5c not found: ID does not exist" containerID="ef2f6d9dd6210987e923fdae132141a92698a4d1fad0f31e7429f951c3341c5c" Sep 06 12:21:18.665585 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:18.665522 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef2f6d9dd6210987e923fdae132141a92698a4d1fad0f31e7429f951c3341c5c"} err="failed to get container status \"ef2f6d9dd6210987e923fdae132141a92698a4d1fad0f31e7429f951c3341c5c\": rpc error: code = NotFound desc = could not find container \"ef2f6d9dd6210987e923fdae132141a92698a4d1fad0f31e7429f951c3341c5c\": container with ID starting with ef2f6d9dd6210987e923fdae132141a92698a4d1fad0f31e7429f951c3341c5c not found: ID does not exist" Sep 06 12:21:18.677190 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:18.677164 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dc4c856bf-2pjhb"] Sep 06 12:21:18.682413 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:18.682393 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7dc4c856bf-2pjhb"] Sep 06 12:21:20.653097 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:21:20.653064 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="393ca7da-6bc7-4f06-9191-d9dee8f5623b" path="/var/lib/kubelet/pods/393ca7da-6bc7-4f06-9191-d9dee8f5623b/volumes" Sep 06 12:22:10.220441 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:10.220402 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-f58nh"] Sep 06 12:22:10.220962 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:10.220906 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f25b4f1d-727c-42e7-963c-5669283addb5" containerName="console" Sep 06 12:22:10.220962 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:10.220927 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f25b4f1d-727c-42e7-963c-5669283addb5" containerName="console" Sep 06 12:22:10.220962 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:10.220954 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="393ca7da-6bc7-4f06-9191-d9dee8f5623b" containerName="console" Sep 06 12:22:10.220962 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:10.220962 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="393ca7da-6bc7-4f06-9191-d9dee8f5623b" containerName="console" Sep 06 12:22:10.221170 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:10.221045 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="393ca7da-6bc7-4f06-9191-d9dee8f5623b" containerName="console" Sep 06 12:22:10.221170 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:10.221064 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f25b4f1d-727c-42e7-963c-5669283addb5" containerName="console" Sep 06 12:22:10.225606 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:10.225585 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-f58nh" Sep 06 12:22:10.227946 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:10.227917 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 06 12:22:10.228061 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:10.227971 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-k5vjs\"" Sep 06 12:22:10.228637 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:10.228616 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 06 12:22:10.232725 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:10.232415 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-f58nh"] Sep 06 12:22:10.313516 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:10.313489 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d79bf6d5-546b-4fa8-a20c-1ff8511950a4-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-f58nh\" (UID: \"d79bf6d5-546b-4fa8-a20c-1ff8511950a4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-f58nh" Sep 06 12:22:10.313636 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:10.313521 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52svs\" (UniqueName: \"kubernetes.io/projected/d79bf6d5-546b-4fa8-a20c-1ff8511950a4-kube-api-access-52svs\") pod \"cert-manager-cainjector-559cdd8c87-f58nh\" (UID: \"d79bf6d5-546b-4fa8-a20c-1ff8511950a4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-f58nh" Sep 06 12:22:10.413915 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:10.413893 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d79bf6d5-546b-4fa8-a20c-1ff8511950a4-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-f58nh\" (UID: \"d79bf6d5-546b-4fa8-a20c-1ff8511950a4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-f58nh" Sep 06 12:22:10.414032 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:10.413926 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52svs\" (UniqueName: \"kubernetes.io/projected/d79bf6d5-546b-4fa8-a20c-1ff8511950a4-kube-api-access-52svs\") pod \"cert-manager-cainjector-559cdd8c87-f58nh\" (UID: \"d79bf6d5-546b-4fa8-a20c-1ff8511950a4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-f58nh" Sep 06 12:22:10.421327 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:10.421304 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d79bf6d5-546b-4fa8-a20c-1ff8511950a4-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-f58nh\" (UID: \"d79bf6d5-546b-4fa8-a20c-1ff8511950a4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-f58nh" Sep 06 12:22:10.421558 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:10.421535 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52svs\" (UniqueName: \"kubernetes.io/projected/d79bf6d5-546b-4fa8-a20c-1ff8511950a4-kube-api-access-52svs\") pod \"cert-manager-cainjector-559cdd8c87-f58nh\" (UID: \"d79bf6d5-546b-4fa8-a20c-1ff8511950a4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-f58nh" Sep 06 12:22:10.539884 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:10.539827 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-f58nh" Sep 06 12:22:10.808000 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:10.807828 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-f58nh"] Sep 06 12:22:10.810685 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:22:10.810654 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd79bf6d5_546b_4fa8_a20c_1ff8511950a4.slice/crio-0236d58633086b5ac9a67c8570e08c4ac60ad158dfaddfe6a6cf507c099f1e06 WatchSource:0}: Error finding container 0236d58633086b5ac9a67c8570e08c4ac60ad158dfaddfe6a6cf507c099f1e06: Status 404 returned error can't find the container with id 0236d58633086b5ac9a67c8570e08c4ac60ad158dfaddfe6a6cf507c099f1e06 Sep 06 12:22:10.873283 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:10.873260 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-f58nh" event={"ID":"d79bf6d5-546b-4fa8-a20c-1ff8511950a4","Type":"ContainerStarted","Data":"0236d58633086b5ac9a67c8570e08c4ac60ad158dfaddfe6a6cf507c099f1e06"} Sep 06 12:22:13.887771 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:13.887721 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-f58nh" event={"ID":"d79bf6d5-546b-4fa8-a20c-1ff8511950a4","Type":"ContainerStarted","Data":"987493aaa725091b40f2dfa2fe7b2c8636a9cebabdb60db670322151ccd9ee29"} Sep 06 12:22:13.902324 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:13.902283 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-f58nh" podStartSLOduration=0.927152774 podStartE2EDuration="3.902268992s" totalImagesPullingTime="2.975116218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:22:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:22:13.900782498 +0000 UTC m=+249.756540387" watchObservedRunningTime="2026-09-06 12:22:13.902268992 +0000 UTC m=+249.758026878" Sep 06 12:22:28.645241 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:28.645154 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-6vhzp"] Sep 06 12:22:28.648684 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:28.648657 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-6vhzp" Sep 06 12:22:28.650779 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:28.650744 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-r5qbl\"" Sep 06 12:22:28.657930 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:28.657906 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-6vhzp"] Sep 06 12:22:28.748203 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:28.748177 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp4mx\" (UniqueName: \"kubernetes.io/projected/49fa6b82-9d4a-49da-a2cb-4dfa281ed892-kube-api-access-cp4mx\") pod \"cert-manager-55b4d954d5-6vhzp\" (UID: \"49fa6b82-9d4a-49da-a2cb-4dfa281ed892\") " pod="cert-manager/cert-manager-55b4d954d5-6vhzp" Sep 06 12:22:28.748309 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:28.748241 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49fa6b82-9d4a-49da-a2cb-4dfa281ed892-bound-sa-token\") pod \"cert-manager-55b4d954d5-6vhzp\" (UID: \"49fa6b82-9d4a-49da-a2cb-4dfa281ed892\") " pod="cert-manager/cert-manager-55b4d954d5-6vhzp" Sep 06 12:22:28.848955 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:28.848926 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp4mx\" (UniqueName: \"kubernetes.io/projected/49fa6b82-9d4a-49da-a2cb-4dfa281ed892-kube-api-access-cp4mx\") pod \"cert-manager-55b4d954d5-6vhzp\" (UID: \"49fa6b82-9d4a-49da-a2cb-4dfa281ed892\") " pod="cert-manager/cert-manager-55b4d954d5-6vhzp" Sep 06 12:22:28.849051 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:28.848969 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49fa6b82-9d4a-49da-a2cb-4dfa281ed892-bound-sa-token\") pod \"cert-manager-55b4d954d5-6vhzp\" (UID: \"49fa6b82-9d4a-49da-a2cb-4dfa281ed892\") " pod="cert-manager/cert-manager-55b4d954d5-6vhzp" Sep 06 12:22:28.857103 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:28.857075 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49fa6b82-9d4a-49da-a2cb-4dfa281ed892-bound-sa-token\") pod \"cert-manager-55b4d954d5-6vhzp\" (UID: \"49fa6b82-9d4a-49da-a2cb-4dfa281ed892\") " pod="cert-manager/cert-manager-55b4d954d5-6vhzp" Sep 06 12:22:28.857236 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:28.857216 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp4mx\" (UniqueName: \"kubernetes.io/projected/49fa6b82-9d4a-49da-a2cb-4dfa281ed892-kube-api-access-cp4mx\") pod \"cert-manager-55b4d954d5-6vhzp\" (UID: \"49fa6b82-9d4a-49da-a2cb-4dfa281ed892\") " pod="cert-manager/cert-manager-55b4d954d5-6vhzp" Sep 06 12:22:28.958392 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:28.958326 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-6vhzp" Sep 06 12:22:29.226247 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:29.226168 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-6vhzp"] Sep 06 12:22:29.228892 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:22:29.228864 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49fa6b82_9d4a_49da_a2cb_4dfa281ed892.slice/crio-86bd8e47eedde1c278d497b8a5cdd05a9727526764ee6e922a95348989d3d1ec WatchSource:0}: Error finding container 86bd8e47eedde1c278d497b8a5cdd05a9727526764ee6e922a95348989d3d1ec: Status 404 returned error can't find the container with id 86bd8e47eedde1c278d497b8a5cdd05a9727526764ee6e922a95348989d3d1ec Sep 06 12:22:29.956375 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:29.956329 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-6vhzp" event={"ID":"49fa6b82-9d4a-49da-a2cb-4dfa281ed892","Type":"ContainerStarted","Data":"095218e86e66481f3f2b32a9b3cc45956e4ee315dd4cfdf2f8c0f143bd4b8f77"} Sep 06 12:22:29.956375 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:29.956370 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-6vhzp" event={"ID":"49fa6b82-9d4a-49da-a2cb-4dfa281ed892","Type":"ContainerStarted","Data":"86bd8e47eedde1c278d497b8a5cdd05a9727526764ee6e922a95348989d3d1ec"} Sep 06 12:22:29.970930 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:29.970881 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-6vhzp" podStartSLOduration=1.9708656740000001 podStartE2EDuration="1.970865674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:22:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:22:29.969408453 +0000 UTC m=+265.825166339" watchObservedRunningTime="2026-09-06 12:22:29.970865674 +0000 UTC m=+265.826623562" Sep 06 12:22:55.441454 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:55.441423 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7bf7c58dfb-wgl45"] Sep 06 12:22:55.444827 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:55.444803 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7bf7c58dfb-wgl45" Sep 06 12:22:55.447672 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:55.447648 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 06 12:22:55.449103 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:55.449079 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 06 12:22:55.449226 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:55.449136 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-lvcwj\"" Sep 06 12:22:55.449226 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:55.449136 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 06 12:22:55.449226 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:55.449143 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 06 12:22:55.465368 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:55.465346 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7bf7c58dfb-wgl45"] Sep 06 12:22:55.549496 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:55.549464 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce2e1a8c-9c75-42bb-ae98-04c1e57f5206-tmp\") pod \"opendatahub-operator-controller-manager-7bf7c58dfb-wgl45\" (UID: \"ce2e1a8c-9c75-42bb-ae98-04c1e57f5206\") " pod="opendatahub/opendatahub-operator-controller-manager-7bf7c58dfb-wgl45" Sep 06 12:22:55.549639 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:55.549508 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ce2e1a8c-9c75-42bb-ae98-04c1e57f5206-manifests\") pod \"opendatahub-operator-controller-manager-7bf7c58dfb-wgl45\" (UID: \"ce2e1a8c-9c75-42bb-ae98-04c1e57f5206\") " pod="opendatahub/opendatahub-operator-controller-manager-7bf7c58dfb-wgl45" Sep 06 12:22:55.549639 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:55.549531 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce2e1a8c-9c75-42bb-ae98-04c1e57f5206-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bf7c58dfb-wgl45\" (UID: \"ce2e1a8c-9c75-42bb-ae98-04c1e57f5206\") " pod="opendatahub/opendatahub-operator-controller-manager-7bf7c58dfb-wgl45" Sep 06 12:22:55.549639 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:55.549553 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ce2e1a8c-9c75-42bb-ae98-04c1e57f5206-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bf7c58dfb-wgl45\" (UID: \"ce2e1a8c-9c75-42bb-ae98-04c1e57f5206\") " pod="opendatahub/opendatahub-operator-controller-manager-7bf7c58dfb-wgl45" Sep 06 12:22:55.549639 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:55.549603 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc8x5\" (UniqueName: \"kubernetes.io/projected/ce2e1a8c-9c75-42bb-ae98-04c1e57f5206-kube-api-access-rc8x5\") pod \"opendatahub-operator-controller-manager-7bf7c58dfb-wgl45\" (UID: \"ce2e1a8c-9c75-42bb-ae98-04c1e57f5206\") " pod="opendatahub/opendatahub-operator-controller-manager-7bf7c58dfb-wgl45" Sep 06 12:22:55.650861 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:55.650833 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce2e1a8c-9c75-42bb-ae98-04c1e57f5206-tmp\") pod \"opendatahub-operator-controller-manager-7bf7c58dfb-wgl45\" (UID: \"ce2e1a8c-9c75-42bb-ae98-04c1e57f5206\") " pod="opendatahub/opendatahub-operator-controller-manager-7bf7c58dfb-wgl45" Sep 06 12:22:55.650980 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:55.650871 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ce2e1a8c-9c75-42bb-ae98-04c1e57f5206-manifests\") pod \"opendatahub-operator-controller-manager-7bf7c58dfb-wgl45\" (UID: \"ce2e1a8c-9c75-42bb-ae98-04c1e57f5206\") " pod="opendatahub/opendatahub-operator-controller-manager-7bf7c58dfb-wgl45" Sep 06 12:22:55.650980 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:55.650894 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce2e1a8c-9c75-42bb-ae98-04c1e57f5206-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bf7c58dfb-wgl45\" (UID: \"ce2e1a8c-9c75-42bb-ae98-04c1e57f5206\") " pod="opendatahub/opendatahub-operator-controller-manager-7bf7c58dfb-wgl45" Sep 06 12:22:55.650980 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:55.650957 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ce2e1a8c-9c75-42bb-ae98-04c1e57f5206-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bf7c58dfb-wgl45\" (UID: \"ce2e1a8c-9c75-42bb-ae98-04c1e57f5206\") " pod="opendatahub/opendatahub-operator-controller-manager-7bf7c58dfb-wgl45" Sep 06 12:22:55.651145 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:55.650988 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc8x5\" (UniqueName: \"kubernetes.io/projected/ce2e1a8c-9c75-42bb-ae98-04c1e57f5206-kube-api-access-rc8x5\") pod \"opendatahub-operator-controller-manager-7bf7c58dfb-wgl45\" (UID: \"ce2e1a8c-9c75-42bb-ae98-04c1e57f5206\") " pod="opendatahub/opendatahub-operator-controller-manager-7bf7c58dfb-wgl45" Sep 06 12:22:55.651286 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:55.651266 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ce2e1a8c-9c75-42bb-ae98-04c1e57f5206-manifests\") pod \"opendatahub-operator-controller-manager-7bf7c58dfb-wgl45\" (UID: \"ce2e1a8c-9c75-42bb-ae98-04c1e57f5206\") " pod="opendatahub/opendatahub-operator-controller-manager-7bf7c58dfb-wgl45" Sep 06 12:22:55.653359 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:55.653333 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce2e1a8c-9c75-42bb-ae98-04c1e57f5206-tmp\") pod \"opendatahub-operator-controller-manager-7bf7c58dfb-wgl45\" (UID: \"ce2e1a8c-9c75-42bb-ae98-04c1e57f5206\") " pod="opendatahub/opendatahub-operator-controller-manager-7bf7c58dfb-wgl45" Sep 06 12:22:55.653434 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:55.653370 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce2e1a8c-9c75-42bb-ae98-04c1e57f5206-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bf7c58dfb-wgl45\" (UID: \"ce2e1a8c-9c75-42bb-ae98-04c1e57f5206\") " pod="opendatahub/opendatahub-operator-controller-manager-7bf7c58dfb-wgl45" Sep 06 12:22:55.653558 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:55.653539 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ce2e1a8c-9c75-42bb-ae98-04c1e57f5206-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bf7c58dfb-wgl45\" (UID: \"ce2e1a8c-9c75-42bb-ae98-04c1e57f5206\") " pod="opendatahub/opendatahub-operator-controller-manager-7bf7c58dfb-wgl45" Sep 06 12:22:55.664922 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:55.664900 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc8x5\" (UniqueName: \"kubernetes.io/projected/ce2e1a8c-9c75-42bb-ae98-04c1e57f5206-kube-api-access-rc8x5\") pod \"opendatahub-operator-controller-manager-7bf7c58dfb-wgl45\" (UID: \"ce2e1a8c-9c75-42bb-ae98-04c1e57f5206\") " pod="opendatahub/opendatahub-operator-controller-manager-7bf7c58dfb-wgl45" Sep 06 12:22:55.757420 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:55.757346 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7bf7c58dfb-wgl45" Sep 06 12:22:56.030777 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:56.030689 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7bf7c58dfb-wgl45"] Sep 06 12:22:56.034362 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:22:56.034339 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce2e1a8c_9c75_42bb_ae98_04c1e57f5206.slice/crio-b4db968c93c36fdb6f7ffb79bee7fc5ee04d3f61e4f665375d939bf7e07ec5e2 WatchSource:0}: Error finding container b4db968c93c36fdb6f7ffb79bee7fc5ee04d3f61e4f665375d939bf7e07ec5e2: Status 404 returned error can't find the container with id b4db968c93c36fdb6f7ffb79bee7fc5ee04d3f61e4f665375d939bf7e07ec5e2 Sep 06 12:22:56.077713 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:56.077678 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7bf7c58dfb-wgl45" event={"ID":"ce2e1a8c-9c75-42bb-ae98-04c1e57f5206","Type":"ContainerStarted","Data":"b4db968c93c36fdb6f7ffb79bee7fc5ee04d3f61e4f665375d939bf7e07ec5e2"} Sep 06 12:22:59.095125 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:22:59.095089 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7bf7c58dfb-wgl45" event={"ID":"ce2e1a8c-9c75-42bb-ae98-04c1e57f5206","Type":"ContainerStarted","Data":"539a6ccd61952ace79c9a2b061ca29a19a6ec337e2b3fe24429275239cf305b3"} Sep 06 12:23:01.106116 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:01.106083 2576 generic.go:356] "Generic (PLEG): container finished" podID="ce2e1a8c-9c75-42bb-ae98-04c1e57f5206" containerID="539a6ccd61952ace79c9a2b061ca29a19a6ec337e2b3fe24429275239cf305b3" exitCode=0 Sep 06 12:23:01.106693 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:01.106165 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7bf7c58dfb-wgl45" event={"ID":"ce2e1a8c-9c75-42bb-ae98-04c1e57f5206","Type":"ContainerDied","Data":"539a6ccd61952ace79c9a2b061ca29a19a6ec337e2b3fe24429275239cf305b3"} Sep 06 12:23:02.113707 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:02.113678 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7bf7c58dfb-wgl45" event={"ID":"ce2e1a8c-9c75-42bb-ae98-04c1e57f5206","Type":"ContainerStarted","Data":"1037f92226da1a2f2bef55be43f006f11bb176dacb3f32dce0e9e0274c1e6099"} Sep 06 12:23:02.114086 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:02.113905 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7bf7c58dfb-wgl45" Sep 06 12:23:02.135192 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:02.135145 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7bf7c58dfb-wgl45" podStartSLOduration=2.789461405 podStartE2EDuration="7.135130372s" totalImagesPullingTime="3.309829174s" totalInitContainerRuntime="1.035839793s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:22:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:23:02.132692153 +0000 UTC m=+297.988450039" watchObservedRunningTime="2026-09-06 12:23:02.135130372 +0000 UTC m=+297.990888322" Sep 06 12:23:04.548347 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:04.548321 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98vz4_4103c09f-dd43-419d-92e5-2374af5b5e01/ovn-acl-logging/0.log" Sep 06 12:23:04.550646 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:04.550623 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98vz4_4103c09f-dd43-419d-92e5-2374af5b5e01/ovn-acl-logging/0.log" Sep 06 12:23:04.551308 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:04.551285 2576 kubelet.go:1648] "Image garbage collection succeeded" Sep 06 12:23:13.120388 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:13.120358 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7bf7c58dfb-wgl45" Sep 06 12:23:25.732447 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:25.732407 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-zw6hf"] Sep 06 12:23:25.735845 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:25.735824 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-zw6hf" Sep 06 12:23:25.738201 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:25.738179 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 06 12:23:25.738201 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:25.738190 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-whz2d\"" Sep 06 12:23:25.744400 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:25.744379 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-zw6hf"] Sep 06 12:23:25.889405 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:25.889375 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a537ef0-6b94-407f-b443-0dc16b4aa810-cert\") pod \"odh-model-controller-7998c67dc4-zw6hf\" (UID: \"4a537ef0-6b94-407f-b443-0dc16b4aa810\") " pod="opendatahub/odh-model-controller-7998c67dc4-zw6hf" Sep 06 12:23:25.889551 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:25.889411 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfqtl\" (UniqueName: \"kubernetes.io/projected/4a537ef0-6b94-407f-b443-0dc16b4aa810-kube-api-access-pfqtl\") pod \"odh-model-controller-7998c67dc4-zw6hf\" (UID: \"4a537ef0-6b94-407f-b443-0dc16b4aa810\") " pod="opendatahub/odh-model-controller-7998c67dc4-zw6hf" Sep 06 12:23:25.990615 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:25.990547 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a537ef0-6b94-407f-b443-0dc16b4aa810-cert\") pod \"odh-model-controller-7998c67dc4-zw6hf\" (UID: \"4a537ef0-6b94-407f-b443-0dc16b4aa810\") " pod="opendatahub/odh-model-controller-7998c67dc4-zw6hf" Sep 06 12:23:25.990615 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:25.990581 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfqtl\" (UniqueName: \"kubernetes.io/projected/4a537ef0-6b94-407f-b443-0dc16b4aa810-kube-api-access-pfqtl\") pod \"odh-model-controller-7998c67dc4-zw6hf\" (UID: \"4a537ef0-6b94-407f-b443-0dc16b4aa810\") " pod="opendatahub/odh-model-controller-7998c67dc4-zw6hf" Sep 06 12:23:25.990813 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:23:25.990690 2576 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 06 12:23:25.990813 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:23:25.990782 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a537ef0-6b94-407f-b443-0dc16b4aa810-cert podName:4a537ef0-6b94-407f-b443-0dc16b4aa810 nodeName:}" failed. No retries permitted until 2026-09-06 12:23:26.490745305 +0000 UTC m=+322.346503170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/4a537ef0-6b94-407f-b443-0dc16b4aa810-cert") pod "odh-model-controller-7998c67dc4-zw6hf" (UID: "4a537ef0-6b94-407f-b443-0dc16b4aa810") : secret "odh-model-controller-webhook-cert" not found Sep 06 12:23:25.999068 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:25.999051 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfqtl\" (UniqueName: \"kubernetes.io/projected/4a537ef0-6b94-407f-b443-0dc16b4aa810-kube-api-access-pfqtl\") pod \"odh-model-controller-7998c67dc4-zw6hf\" (UID: \"4a537ef0-6b94-407f-b443-0dc16b4aa810\") " pod="opendatahub/odh-model-controller-7998c67dc4-zw6hf" Sep 06 12:23:26.494350 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:26.494313 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a537ef0-6b94-407f-b443-0dc16b4aa810-cert\") pod \"odh-model-controller-7998c67dc4-zw6hf\" (UID: \"4a537ef0-6b94-407f-b443-0dc16b4aa810\") " pod="opendatahub/odh-model-controller-7998c67dc4-zw6hf" Sep 06 12:23:26.496858 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:26.496833 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a537ef0-6b94-407f-b443-0dc16b4aa810-cert\") pod \"odh-model-controller-7998c67dc4-zw6hf\" (UID: \"4a537ef0-6b94-407f-b443-0dc16b4aa810\") " pod="opendatahub/odh-model-controller-7998c67dc4-zw6hf" Sep 06 12:23:26.646659 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:26.646624 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-zw6hf" Sep 06 12:23:26.913478 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:26.913455 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-zw6hf"] Sep 06 12:23:26.915626 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:23:26.915586 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a537ef0_6b94_407f_b443_0dc16b4aa810.slice/crio-a801974de7d9615c00897eb808fe6647ff18fa1243a98193207637a85407098e WatchSource:0}: Error finding container a801974de7d9615c00897eb808fe6647ff18fa1243a98193207637a85407098e: Status 404 returned error can't find the container with id a801974de7d9615c00897eb808fe6647ff18fa1243a98193207637a85407098e Sep 06 12:23:26.917011 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:26.916995 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 06 12:23:27.239377 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:27.239280 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-zw6hf" event={"ID":"4a537ef0-6b94-407f-b443-0dc16b4aa810","Type":"ContainerStarted","Data":"a801974de7d9615c00897eb808fe6647ff18fa1243a98193207637a85407098e"} Sep 06 12:23:30.257140 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:30.257056 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-zw6hf" event={"ID":"4a537ef0-6b94-407f-b443-0dc16b4aa810","Type":"ContainerStarted","Data":"27b0b40519493e87abc365200ce60811d2b711189906b54e96768e43a77a3135"} Sep 06 12:23:30.257477 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:30.257156 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-zw6hf" Sep 06 12:23:30.272157 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:30.272112 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-zw6hf" podStartSLOduration=2.273952696 podStartE2EDuration="5.272096376s" totalImagesPullingTime="2.99814368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:23:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:23:30.271039789 +0000 UTC m=+326.126797698" watchObservedRunningTime="2026-09-06 12:23:30.272096376 +0000 UTC m=+326.127854263" Sep 06 12:23:31.984970 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:31.984938 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-849pv"] Sep 06 12:23:31.988293 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:31.988272 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-849pv" Sep 06 12:23:31.990811 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:31.990730 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 06 12:23:31.990927 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:31.990837 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 06 12:23:31.991003 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:31.990990 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 06 12:23:31.991129 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:31.991107 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-zdlqb\"" Sep 06 12:23:31.991244 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:31.991143 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 06 12:23:31.991397 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:31.991382 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 06 12:23:31.991715 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:31.991696 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 06 12:23:32.000543 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:32.000524 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-849pv"] Sep 06 12:23:32.149491 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:32.149468 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a4f7e77d-d6df-44ea-b9d3-07fb828050f5-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-849pv\" (UID: \"a4f7e77d-d6df-44ea-b9d3-07fb828050f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-849pv" Sep 06 12:23:32.149620 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:32.149504 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a4f7e77d-d6df-44ea-b9d3-07fb828050f5-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-849pv\" (UID: \"a4f7e77d-d6df-44ea-b9d3-07fb828050f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-849pv" Sep 06 12:23:32.149620 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:32.149532 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a4f7e77d-d6df-44ea-b9d3-07fb828050f5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-849pv\" (UID: \"a4f7e77d-d6df-44ea-b9d3-07fb828050f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-849pv" Sep 06 12:23:32.149620 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:32.149551 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4f7e77d-d6df-44ea-b9d3-07fb828050f5-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-849pv\" (UID: \"a4f7e77d-d6df-44ea-b9d3-07fb828050f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-849pv" Sep 06 12:23:32.149620 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:32.149565 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a4f7e77d-d6df-44ea-b9d3-07fb828050f5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-849pv\" (UID: \"a4f7e77d-d6df-44ea-b9d3-07fb828050f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-849pv" Sep 06 12:23:32.149620 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:32.149585 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4f7e77d-d6df-44ea-b9d3-07fb828050f5-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-849pv\" (UID: \"a4f7e77d-d6df-44ea-b9d3-07fb828050f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-849pv" Sep 06 12:23:32.149620 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:32.149613 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjpp6\" (UniqueName: \"kubernetes.io/projected/a4f7e77d-d6df-44ea-b9d3-07fb828050f5-kube-api-access-qjpp6\") pod \"istiod-openshift-gateway-5677ccb49b-849pv\" (UID: \"a4f7e77d-d6df-44ea-b9d3-07fb828050f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-849pv" Sep 06 12:23:32.250698 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:32.250632 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a4f7e77d-d6df-44ea-b9d3-07fb828050f5-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-849pv\" (UID: \"a4f7e77d-d6df-44ea-b9d3-07fb828050f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-849pv" Sep 06 12:23:32.250698 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:32.250674 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a4f7e77d-d6df-44ea-b9d3-07fb828050f5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-849pv\" (UID: \"a4f7e77d-d6df-44ea-b9d3-07fb828050f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-849pv" Sep 06 12:23:32.250698 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:32.250693 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4f7e77d-d6df-44ea-b9d3-07fb828050f5-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-849pv\" (UID: \"a4f7e77d-d6df-44ea-b9d3-07fb828050f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-849pv" Sep 06 12:23:32.250938 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:32.250710 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a4f7e77d-d6df-44ea-b9d3-07fb828050f5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-849pv\" (UID: \"a4f7e77d-d6df-44ea-b9d3-07fb828050f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-849pv" Sep 06 12:23:32.250938 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:32.250730 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4f7e77d-d6df-44ea-b9d3-07fb828050f5-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-849pv\" (UID: \"a4f7e77d-d6df-44ea-b9d3-07fb828050f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-849pv" Sep 06 12:23:32.250938 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:32.250788 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjpp6\" (UniqueName: \"kubernetes.io/projected/a4f7e77d-d6df-44ea-b9d3-07fb828050f5-kube-api-access-qjpp6\") pod \"istiod-openshift-gateway-5677ccb49b-849pv\" (UID: \"a4f7e77d-d6df-44ea-b9d3-07fb828050f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-849pv" Sep 06 12:23:32.250938 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:32.250833 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a4f7e77d-d6df-44ea-b9d3-07fb828050f5-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-849pv\" (UID: \"a4f7e77d-d6df-44ea-b9d3-07fb828050f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-849pv" Sep 06 12:23:32.251776 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:32.251736 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a4f7e77d-d6df-44ea-b9d3-07fb828050f5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-849pv\" (UID: \"a4f7e77d-d6df-44ea-b9d3-07fb828050f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-849pv" Sep 06 12:23:32.253291 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:32.253269 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4f7e77d-d6df-44ea-b9d3-07fb828050f5-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-849pv\" (UID: \"a4f7e77d-d6df-44ea-b9d3-07fb828050f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-849pv" Sep 06 12:23:32.253465 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:32.253444 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a4f7e77d-d6df-44ea-b9d3-07fb828050f5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-849pv\" (UID: \"a4f7e77d-d6df-44ea-b9d3-07fb828050f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-849pv" Sep 06 12:23:32.253541 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:32.253525 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a4f7e77d-d6df-44ea-b9d3-07fb828050f5-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-849pv\" (UID: \"a4f7e77d-d6df-44ea-b9d3-07fb828050f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-849pv" Sep 06 12:23:32.253616 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:32.253597 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a4f7e77d-d6df-44ea-b9d3-07fb828050f5-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-849pv\" (UID: \"a4f7e77d-d6df-44ea-b9d3-07fb828050f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-849pv" Sep 06 12:23:32.259294 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:32.259267 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4f7e77d-d6df-44ea-b9d3-07fb828050f5-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-849pv\" (UID: \"a4f7e77d-d6df-44ea-b9d3-07fb828050f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-849pv" Sep 06 12:23:32.259368 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:32.259347 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjpp6\" (UniqueName: \"kubernetes.io/projected/a4f7e77d-d6df-44ea-b9d3-07fb828050f5-kube-api-access-qjpp6\") pod \"istiod-openshift-gateway-5677ccb49b-849pv\" (UID: \"a4f7e77d-d6df-44ea-b9d3-07fb828050f5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-849pv" Sep 06 12:23:32.300888 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:32.300866 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-849pv" Sep 06 12:23:32.579976 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:32.579928 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-849pv"] Sep 06 12:23:32.582159 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:23:32.582134 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4f7e77d_d6df_44ea_b9d3_07fb828050f5.slice/crio-58a84925a4d5d36d8b36eea6b2d73be44cdb3de5da501ef6e716fb1a8bc67349 WatchSource:0}: Error finding container 58a84925a4d5d36d8b36eea6b2d73be44cdb3de5da501ef6e716fb1a8bc67349: Status 404 returned error can't find the container with id 58a84925a4d5d36d8b36eea6b2d73be44cdb3de5da501ef6e716fb1a8bc67349 Sep 06 12:23:33.278600 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:33.278494 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-849pv" event={"ID":"a4f7e77d-d6df-44ea-b9d3-07fb828050f5","Type":"ContainerStarted","Data":"58a84925a4d5d36d8b36eea6b2d73be44cdb3de5da501ef6e716fb1a8bc67349"} Sep 06 12:23:33.406466 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:33.405549 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c968668cd-rkhkl"] Sep 06 12:23:33.409548 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:33.409442 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7c968668cd-rkhkl" Sep 06 12:23:33.414158 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:33.414126 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 06 12:23:33.414291 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:33.414221 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 06 12:23:33.414413 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:33.414395 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 06 12:23:33.414413 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:33.414407 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-7kd99\"" Sep 06 12:23:33.414731 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:33.414563 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 06 12:23:33.414731 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:33.414613 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 06 12:23:33.422170 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:33.422149 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c968668cd-rkhkl"] Sep 06 12:23:33.563303 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:33.563221 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dd4a9b00-86c1-4b71-ae8b-382b54be0583-metrics-cert\") pod \"lws-controller-manager-7c968668cd-rkhkl\" (UID: \"dd4a9b00-86c1-4b71-ae8b-382b54be0583\") " pod="openshift-lws-operator/lws-controller-manager-7c968668cd-rkhkl" Sep 06 12:23:33.563452 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:33.563370 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/dd4a9b00-86c1-4b71-ae8b-382b54be0583-manager-config\") pod \"lws-controller-manager-7c968668cd-rkhkl\" (UID: \"dd4a9b00-86c1-4b71-ae8b-382b54be0583\") " pod="openshift-lws-operator/lws-controller-manager-7c968668cd-rkhkl" Sep 06 12:23:33.563452 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:33.563418 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd4a9b00-86c1-4b71-ae8b-382b54be0583-cert\") pod \"lws-controller-manager-7c968668cd-rkhkl\" (UID: \"dd4a9b00-86c1-4b71-ae8b-382b54be0583\") " pod="openshift-lws-operator/lws-controller-manager-7c968668cd-rkhkl" Sep 06 12:23:33.563563 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:33.563524 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sf6f\" (UniqueName: \"kubernetes.io/projected/dd4a9b00-86c1-4b71-ae8b-382b54be0583-kube-api-access-4sf6f\") pod \"lws-controller-manager-7c968668cd-rkhkl\" (UID: \"dd4a9b00-86c1-4b71-ae8b-382b54be0583\") " pod="openshift-lws-operator/lws-controller-manager-7c968668cd-rkhkl" Sep 06 12:23:33.664952 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:33.664912 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd4a9b00-86c1-4b71-ae8b-382b54be0583-cert\") pod \"lws-controller-manager-7c968668cd-rkhkl\" (UID: \"dd4a9b00-86c1-4b71-ae8b-382b54be0583\") " pod="openshift-lws-operator/lws-controller-manager-7c968668cd-rkhkl" Sep 06 12:23:33.665116 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:33.664966 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sf6f\" (UniqueName: \"kubernetes.io/projected/dd4a9b00-86c1-4b71-ae8b-382b54be0583-kube-api-access-4sf6f\") pod \"lws-controller-manager-7c968668cd-rkhkl\" (UID: \"dd4a9b00-86c1-4b71-ae8b-382b54be0583\") " pod="openshift-lws-operator/lws-controller-manager-7c968668cd-rkhkl" Sep 06 12:23:33.665116 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:33.665043 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dd4a9b00-86c1-4b71-ae8b-382b54be0583-metrics-cert\") pod \"lws-controller-manager-7c968668cd-rkhkl\" (UID: \"dd4a9b00-86c1-4b71-ae8b-382b54be0583\") " pod="openshift-lws-operator/lws-controller-manager-7c968668cd-rkhkl" Sep 06 12:23:33.665248 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:33.665138 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/dd4a9b00-86c1-4b71-ae8b-382b54be0583-manager-config\") pod \"lws-controller-manager-7c968668cd-rkhkl\" (UID: \"dd4a9b00-86c1-4b71-ae8b-382b54be0583\") " pod="openshift-lws-operator/lws-controller-manager-7c968668cd-rkhkl" Sep 06 12:23:33.665904 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:33.665879 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/dd4a9b00-86c1-4b71-ae8b-382b54be0583-manager-config\") pod \"lws-controller-manager-7c968668cd-rkhkl\" (UID: \"dd4a9b00-86c1-4b71-ae8b-382b54be0583\") " pod="openshift-lws-operator/lws-controller-manager-7c968668cd-rkhkl" Sep 06 12:23:33.668705 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:33.668672 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dd4a9b00-86c1-4b71-ae8b-382b54be0583-metrics-cert\") pod \"lws-controller-manager-7c968668cd-rkhkl\" (UID: \"dd4a9b00-86c1-4b71-ae8b-382b54be0583\") " pod="openshift-lws-operator/lws-controller-manager-7c968668cd-rkhkl" Sep 06 12:23:33.669197 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:33.669169 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd4a9b00-86c1-4b71-ae8b-382b54be0583-cert\") pod \"lws-controller-manager-7c968668cd-rkhkl\" (UID: \"dd4a9b00-86c1-4b71-ae8b-382b54be0583\") " pod="openshift-lws-operator/lws-controller-manager-7c968668cd-rkhkl" Sep 06 12:23:33.676369 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:33.676343 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sf6f\" (UniqueName: \"kubernetes.io/projected/dd4a9b00-86c1-4b71-ae8b-382b54be0583-kube-api-access-4sf6f\") pod \"lws-controller-manager-7c968668cd-rkhkl\" (UID: \"dd4a9b00-86c1-4b71-ae8b-382b54be0583\") " pod="openshift-lws-operator/lws-controller-manager-7c968668cd-rkhkl" Sep 06 12:23:33.730737 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:33.730710 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7c968668cd-rkhkl" Sep 06 12:23:34.039564 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:34.039538 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c968668cd-rkhkl"] Sep 06 12:23:34.042000 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:23:34.041959 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd4a9b00_86c1_4b71_ae8b_382b54be0583.slice/crio-44471fa39f630830094b4666eb926b8d773aa06fbdfe380953a2d39ea421754c WatchSource:0}: Error finding container 44471fa39f630830094b4666eb926b8d773aa06fbdfe380953a2d39ea421754c: Status 404 returned error can't find the container with id 44471fa39f630830094b4666eb926b8d773aa06fbdfe380953a2d39ea421754c Sep 06 12:23:34.287320 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:34.287257 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c968668cd-rkhkl" event={"ID":"dd4a9b00-86c1-4b71-ae8b-382b54be0583","Type":"ContainerStarted","Data":"44471fa39f630830094b4666eb926b8d773aa06fbdfe380953a2d39ea421754c"} Sep 06 12:23:35.051928 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:35.051889 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 06 12:23:35.051997 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:35.051967 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 06 12:23:35.296088 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:35.296046 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-849pv" event={"ID":"a4f7e77d-d6df-44ea-b9d3-07fb828050f5","Type":"ContainerStarted","Data":"ced04e319752ffbf11aeb37fad9b8093b6d3331c06087ffea91238e28982b6a3"} Sep 06 12:23:35.296542 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:35.296316 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-849pv" Sep 06 12:23:35.328856 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:35.328730 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-849pv" podStartSLOduration=1.86145625 podStartE2EDuration="4.328709608s" totalImagesPullingTime="2.467253358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:23:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:23:35.326059649 +0000 UTC m=+331.181817576" watchObservedRunningTime="2026-09-06 12:23:35.328709608 +0000 UTC m=+331.184467500" Sep 06 12:23:36.304927 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:36.304897 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-849pv" Sep 06 12:23:37.308406 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:37.308321 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c968668cd-rkhkl" event={"ID":"dd4a9b00-86c1-4b71-ae8b-382b54be0583","Type":"ContainerStarted","Data":"917776f866088b2c4b8e5b09fbfa39a0579bbad3af4cc9ce114aaaa3de2bc833"} Sep 06 12:23:37.308406 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:37.308392 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7c968668cd-rkhkl" Sep 06 12:23:37.325988 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:37.325938 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7c968668cd-rkhkl" podStartSLOduration=1.358946624 podStartE2EDuration="4.325923127s" totalImagesPullingTime="2.966976503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:23:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:23:37.325045131 +0000 UTC m=+333.180803076" watchObservedRunningTime="2026-09-06 12:23:37.325923127 +0000 UTC m=+333.181681013" Sep 06 12:23:41.263619 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:41.263589 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-zw6hf" Sep 06 12:23:48.315965 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:48.315931 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7c968668cd-rkhkl" Sep 06 12:23:56.606818 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:56.606781 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-79f466df84-6mqf4"] Sep 06 12:23:56.610868 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:56.610837 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79f466df84-6mqf4" Sep 06 12:23:56.620565 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:56.620539 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79f466df84-6mqf4"] Sep 06 12:23:56.648637 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:56.648609 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b577512-6c05-41fd-a560-e3c966cec496-console-serving-cert\") pod \"console-79f466df84-6mqf4\" (UID: \"1b577512-6c05-41fd-a560-e3c966cec496\") " pod="openshift-console/console-79f466df84-6mqf4" Sep 06 12:23:56.648637 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:56.648638 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b577512-6c05-41fd-a560-e3c966cec496-trusted-ca-bundle\") pod \"console-79f466df84-6mqf4\" (UID: \"1b577512-6c05-41fd-a560-e3c966cec496\") " pod="openshift-console/console-79f466df84-6mqf4" Sep 06 12:23:56.648841 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:56.648662 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b577512-6c05-41fd-a560-e3c966cec496-console-config\") pod \"console-79f466df84-6mqf4\" (UID: \"1b577512-6c05-41fd-a560-e3c966cec496\") " pod="openshift-console/console-79f466df84-6mqf4" Sep 06 12:23:56.648841 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:56.648686 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b577512-6c05-41fd-a560-e3c966cec496-console-oauth-config\") pod \"console-79f466df84-6mqf4\" (UID: \"1b577512-6c05-41fd-a560-e3c966cec496\") " pod="openshift-console/console-79f466df84-6mqf4" Sep 06 12:23:56.648841 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:56.648710 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b577512-6c05-41fd-a560-e3c966cec496-oauth-serving-cert\") pod \"console-79f466df84-6mqf4\" (UID: \"1b577512-6c05-41fd-a560-e3c966cec496\") " pod="openshift-console/console-79f466df84-6mqf4" Sep 06 12:23:56.648841 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:56.648801 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b577512-6c05-41fd-a560-e3c966cec496-service-ca\") pod \"console-79f466df84-6mqf4\" (UID: \"1b577512-6c05-41fd-a560-e3c966cec496\") " pod="openshift-console/console-79f466df84-6mqf4" Sep 06 12:23:56.648841 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:56.648828 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m68nj\" (UniqueName: \"kubernetes.io/projected/1b577512-6c05-41fd-a560-e3c966cec496-kube-api-access-m68nj\") pod \"console-79f466df84-6mqf4\" (UID: \"1b577512-6c05-41fd-a560-e3c966cec496\") " pod="openshift-console/console-79f466df84-6mqf4" Sep 06 12:23:56.749559 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:56.749536 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b577512-6c05-41fd-a560-e3c966cec496-console-oauth-config\") pod \"console-79f466df84-6mqf4\" (UID: \"1b577512-6c05-41fd-a560-e3c966cec496\") " pod="openshift-console/console-79f466df84-6mqf4" Sep 06 12:23:56.749676 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:56.749564 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b577512-6c05-41fd-a560-e3c966cec496-oauth-serving-cert\") pod \"console-79f466df84-6mqf4\" (UID: \"1b577512-6c05-41fd-a560-e3c966cec496\") " pod="openshift-console/console-79f466df84-6mqf4" Sep 06 12:23:56.749676 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:56.749584 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b577512-6c05-41fd-a560-e3c966cec496-service-ca\") pod \"console-79f466df84-6mqf4\" (UID: \"1b577512-6c05-41fd-a560-e3c966cec496\") " pod="openshift-console/console-79f466df84-6mqf4" Sep 06 12:23:56.749676 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:56.749605 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m68nj\" (UniqueName: \"kubernetes.io/projected/1b577512-6c05-41fd-a560-e3c966cec496-kube-api-access-m68nj\") pod \"console-79f466df84-6mqf4\" (UID: \"1b577512-6c05-41fd-a560-e3c966cec496\") " pod="openshift-console/console-79f466df84-6mqf4" Sep 06 12:23:56.749857 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:56.749695 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b577512-6c05-41fd-a560-e3c966cec496-console-serving-cert\") pod \"console-79f466df84-6mqf4\" (UID: \"1b577512-6c05-41fd-a560-e3c966cec496\") " pod="openshift-console/console-79f466df84-6mqf4" Sep 06 12:23:56.749857 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:56.749819 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b577512-6c05-41fd-a560-e3c966cec496-trusted-ca-bundle\") pod \"console-79f466df84-6mqf4\" (UID: \"1b577512-6c05-41fd-a560-e3c966cec496\") " pod="openshift-console/console-79f466df84-6mqf4" Sep 06 12:23:56.749968 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:56.749870 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b577512-6c05-41fd-a560-e3c966cec496-console-config\") pod \"console-79f466df84-6mqf4\" (UID: \"1b577512-6c05-41fd-a560-e3c966cec496\") " pod="openshift-console/console-79f466df84-6mqf4" Sep 06 12:23:56.750424 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:56.750400 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b577512-6c05-41fd-a560-e3c966cec496-oauth-serving-cert\") pod \"console-79f466df84-6mqf4\" (UID: \"1b577512-6c05-41fd-a560-e3c966cec496\") " pod="openshift-console/console-79f466df84-6mqf4" Sep 06 12:23:56.750640 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:56.750615 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b577512-6c05-41fd-a560-e3c966cec496-service-ca\") pod \"console-79f466df84-6mqf4\" (UID: \"1b577512-6c05-41fd-a560-e3c966cec496\") " pod="openshift-console/console-79f466df84-6mqf4" Sep 06 12:23:56.750640 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:56.750628 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b577512-6c05-41fd-a560-e3c966cec496-trusted-ca-bundle\") pod \"console-79f466df84-6mqf4\" (UID: \"1b577512-6c05-41fd-a560-e3c966cec496\") " pod="openshift-console/console-79f466df84-6mqf4" Sep 06 12:23:56.750873 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:56.750853 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b577512-6c05-41fd-a560-e3c966cec496-console-config\") pod \"console-79f466df84-6mqf4\" (UID: \"1b577512-6c05-41fd-a560-e3c966cec496\") " pod="openshift-console/console-79f466df84-6mqf4" Sep 06 12:23:56.752274 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:56.752248 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b577512-6c05-41fd-a560-e3c966cec496-console-oauth-config\") pod \"console-79f466df84-6mqf4\" (UID: \"1b577512-6c05-41fd-a560-e3c966cec496\") " pod="openshift-console/console-79f466df84-6mqf4" Sep 06 12:23:56.752378 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:56.752357 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b577512-6c05-41fd-a560-e3c966cec496-console-serving-cert\") pod \"console-79f466df84-6mqf4\" (UID: \"1b577512-6c05-41fd-a560-e3c966cec496\") " pod="openshift-console/console-79f466df84-6mqf4" Sep 06 12:23:56.756804 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:56.756783 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m68nj\" (UniqueName: \"kubernetes.io/projected/1b577512-6c05-41fd-a560-e3c966cec496-kube-api-access-m68nj\") pod \"console-79f466df84-6mqf4\" (UID: \"1b577512-6c05-41fd-a560-e3c966cec496\") " pod="openshift-console/console-79f466df84-6mqf4" Sep 06 12:23:56.921781 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:56.921741 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79f466df84-6mqf4" Sep 06 12:23:57.194394 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:57.194366 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79f466df84-6mqf4"] Sep 06 12:23:57.197034 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:23:57.196993 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b577512_6c05_41fd_a560_e3c966cec496.slice/crio-ff57af55c9b127994c2921daaf24abce48a0ed14ef57048305ac5cfe7755a62e WatchSource:0}: Error finding container ff57af55c9b127994c2921daaf24abce48a0ed14ef57048305ac5cfe7755a62e: Status 404 returned error can't find the container with id ff57af55c9b127994c2921daaf24abce48a0ed14ef57048305ac5cfe7755a62e Sep 06 12:23:57.410196 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:57.410160 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79f466df84-6mqf4" event={"ID":"1b577512-6c05-41fd-a560-e3c966cec496","Type":"ContainerStarted","Data":"a03a8f3be316204987ab6b2e39842c99836af2fe941de07abca00beacda065a4"} Sep 06 12:23:57.410335 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:57.410213 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79f466df84-6mqf4" event={"ID":"1b577512-6c05-41fd-a560-e3c966cec496","Type":"ContainerStarted","Data":"ff57af55c9b127994c2921daaf24abce48a0ed14ef57048305ac5cfe7755a62e"} Sep 06 12:23:57.429949 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:23:57.429903 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-79f466df84-6mqf4" podStartSLOduration=1.429888099 podStartE2EDuration="1.429888099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:23:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:23:57.428916657 +0000 UTC m=+353.284674567" watchObservedRunningTime="2026-09-06 12:23:57.429888099 +0000 UTC m=+353.285645985" Sep 06 12:24:06.921896 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:06.921861 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79f466df84-6mqf4" Sep 06 12:24:06.922399 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:06.921909 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-79f466df84-6mqf4" Sep 06 12:24:06.926868 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:06.926846 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-79f466df84-6mqf4" Sep 06 12:24:07.465656 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:07.465626 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-79f466df84-6mqf4" Sep 06 12:24:07.514238 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:07.514208 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-df67f944b-ttjpg"] Sep 06 12:24:32.536162 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:32.536120 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-df67f944b-ttjpg" podUID="17ee86bd-4799-47e1-a2d6-268938f99941" containerName="console" containerID="cri-o://939f49be8d5815103fd25b2dec9f476fc5a765f0a29be4995d354559f8e25547" gracePeriod=15 Sep 06 12:24:32.543852 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:32.543821 2576 patch_prober.go:28] interesting pod/console-df67f944b-ttjpg container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.134.0.22:8443/health\": dial tcp 10.134.0.22:8443: connect: connection refused" start-of-body= Sep 06 12:24:32.543946 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:32.543873 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-df67f944b-ttjpg" podUID="17ee86bd-4799-47e1-a2d6-268938f99941" containerName="console" probeResult="failure" output="Get \"https://10.134.0.22:8443/health\": dial tcp 10.134.0.22:8443: connect: connection refused" Sep 06 12:24:32.778937 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:32.778912 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-df67f944b-ttjpg" Sep 06 12:24:32.946416 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:32.946388 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17ee86bd-4799-47e1-a2d6-268938f99941-console-oauth-config\") pod \"17ee86bd-4799-47e1-a2d6-268938f99941\" (UID: \"17ee86bd-4799-47e1-a2d6-268938f99941\") " Sep 06 12:24:32.946566 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:32.946427 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17ee86bd-4799-47e1-a2d6-268938f99941-console-serving-cert\") pod \"17ee86bd-4799-47e1-a2d6-268938f99941\" (UID: \"17ee86bd-4799-47e1-a2d6-268938f99941\") " Sep 06 12:24:32.946566 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:32.946471 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8r8wp\" (UniqueName: \"kubernetes.io/projected/17ee86bd-4799-47e1-a2d6-268938f99941-kube-api-access-8r8wp\") pod \"17ee86bd-4799-47e1-a2d6-268938f99941\" (UID: \"17ee86bd-4799-47e1-a2d6-268938f99941\") " Sep 06 12:24:32.946566 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:32.946486 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17ee86bd-4799-47e1-a2d6-268938f99941-oauth-serving-cert\") pod \"17ee86bd-4799-47e1-a2d6-268938f99941\" (UID: \"17ee86bd-4799-47e1-a2d6-268938f99941\") " Sep 06 12:24:32.946566 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:32.946515 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17ee86bd-4799-47e1-a2d6-268938f99941-console-config\") pod \"17ee86bd-4799-47e1-a2d6-268938f99941\" (UID: \"17ee86bd-4799-47e1-a2d6-268938f99941\") " Sep 06 12:24:32.946566 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:32.946541 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17ee86bd-4799-47e1-a2d6-268938f99941-trusted-ca-bundle\") pod \"17ee86bd-4799-47e1-a2d6-268938f99941\" (UID: \"17ee86bd-4799-47e1-a2d6-268938f99941\") " Sep 06 12:24:32.946848 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:32.946576 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17ee86bd-4799-47e1-a2d6-268938f99941-service-ca\") pod \"17ee86bd-4799-47e1-a2d6-268938f99941\" (UID: \"17ee86bd-4799-47e1-a2d6-268938f99941\") " Sep 06 12:24:32.946988 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:32.946950 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17ee86bd-4799-47e1-a2d6-268938f99941-console-config" (OuterVolumeSpecName: "console-config") pod "17ee86bd-4799-47e1-a2d6-268938f99941" (UID: "17ee86bd-4799-47e1-a2d6-268938f99941"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:24:32.947052 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:32.947025 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17ee86bd-4799-47e1-a2d6-268938f99941-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "17ee86bd-4799-47e1-a2d6-268938f99941" (UID: "17ee86bd-4799-47e1-a2d6-268938f99941"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:24:32.947095 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:32.946964 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17ee86bd-4799-47e1-a2d6-268938f99941-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "17ee86bd-4799-47e1-a2d6-268938f99941" (UID: "17ee86bd-4799-47e1-a2d6-268938f99941"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:24:32.947350 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:32.947295 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17ee86bd-4799-47e1-a2d6-268938f99941-service-ca" (OuterVolumeSpecName: "service-ca") pod "17ee86bd-4799-47e1-a2d6-268938f99941" (UID: "17ee86bd-4799-47e1-a2d6-268938f99941"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:24:32.948499 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:32.948469 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17ee86bd-4799-47e1-a2d6-268938f99941-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "17ee86bd-4799-47e1-a2d6-268938f99941" (UID: "17ee86bd-4799-47e1-a2d6-268938f99941"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 12:24:32.948645 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:32.948627 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17ee86bd-4799-47e1-a2d6-268938f99941-kube-api-access-8r8wp" (OuterVolumeSpecName: "kube-api-access-8r8wp") pod "17ee86bd-4799-47e1-a2d6-268938f99941" (UID: "17ee86bd-4799-47e1-a2d6-268938f99941"). InnerVolumeSpecName "kube-api-access-8r8wp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:24:32.948744 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:32.948722 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17ee86bd-4799-47e1-a2d6-268938f99941-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "17ee86bd-4799-47e1-a2d6-268938f99941" (UID: "17ee86bd-4799-47e1-a2d6-268938f99941"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 12:24:33.047990 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:33.047962 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17ee86bd-4799-47e1-a2d6-268938f99941-console-oauth-config\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:24:33.047990 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:33.047984 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17ee86bd-4799-47e1-a2d6-268938f99941-console-serving-cert\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:24:33.047990 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:33.047993 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r8wp\" (UniqueName: \"kubernetes.io/projected/17ee86bd-4799-47e1-a2d6-268938f99941-kube-api-access-8r8wp\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:24:33.048211 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:33.048002 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17ee86bd-4799-47e1-a2d6-268938f99941-oauth-serving-cert\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:24:33.048211 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:33.048010 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17ee86bd-4799-47e1-a2d6-268938f99941-console-config\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:24:33.048211 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:33.048018 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17ee86bd-4799-47e1-a2d6-268938f99941-trusted-ca-bundle\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:24:33.048211 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:33.048025 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17ee86bd-4799-47e1-a2d6-268938f99941-service-ca\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:24:33.587607 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:33.587574 2576 generic.go:356] "Generic (PLEG): container finished" podID="17ee86bd-4799-47e1-a2d6-268938f99941" containerID="939f49be8d5815103fd25b2dec9f476fc5a765f0a29be4995d354559f8e25547" exitCode=0 Sep 06 12:24:33.587987 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:33.587625 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-df67f944b-ttjpg" Sep 06 12:24:33.587987 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:33.587664 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-df67f944b-ttjpg" event={"ID":"17ee86bd-4799-47e1-a2d6-268938f99941","Type":"ContainerDied","Data":"939f49be8d5815103fd25b2dec9f476fc5a765f0a29be4995d354559f8e25547"} Sep 06 12:24:33.587987 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:33.587712 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-df67f944b-ttjpg" event={"ID":"17ee86bd-4799-47e1-a2d6-268938f99941","Type":"ContainerDied","Data":"1ff5117ac356ebb32dec1448f60e22ddcd2c29171e2e8615fd1359d42f23eb04"} Sep 06 12:24:33.587987 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:33.587731 2576 scope.go:117] "RemoveContainer" containerID="939f49be8d5815103fd25b2dec9f476fc5a765f0a29be4995d354559f8e25547" Sep 06 12:24:33.596845 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:33.596828 2576 scope.go:117] "RemoveContainer" containerID="939f49be8d5815103fd25b2dec9f476fc5a765f0a29be4995d354559f8e25547" Sep 06 12:24:33.597134 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:24:33.597117 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"939f49be8d5815103fd25b2dec9f476fc5a765f0a29be4995d354559f8e25547\": container with ID starting with 939f49be8d5815103fd25b2dec9f476fc5a765f0a29be4995d354559f8e25547 not found: ID does not exist" containerID="939f49be8d5815103fd25b2dec9f476fc5a765f0a29be4995d354559f8e25547" Sep 06 12:24:33.597177 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:33.597141 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"939f49be8d5815103fd25b2dec9f476fc5a765f0a29be4995d354559f8e25547"} err="failed to get container status \"939f49be8d5815103fd25b2dec9f476fc5a765f0a29be4995d354559f8e25547\": rpc error: code = NotFound desc = could not find container \"939f49be8d5815103fd25b2dec9f476fc5a765f0a29be4995d354559f8e25547\": container with ID starting with 939f49be8d5815103fd25b2dec9f476fc5a765f0a29be4995d354559f8e25547 not found: ID does not exist" Sep 06 12:24:33.608884 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:33.608859 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-df67f944b-ttjpg"] Sep 06 12:24:33.612823 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:33.612807 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-df67f944b-ttjpg"] Sep 06 12:24:34.652942 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:24:34.652908 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17ee86bd-4799-47e1-a2d6-268938f99941" path="/var/lib/kubelet/pods/17ee86bd-4799-47e1-a2d6-268938f99941/volumes" Sep 06 12:25:12.845690 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:12.845657 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ptcxv"] Sep 06 12:25:12.846091 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:12.846072 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17ee86bd-4799-47e1-a2d6-268938f99941" containerName="console" Sep 06 12:25:12.846091 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:12.846087 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="17ee86bd-4799-47e1-a2d6-268938f99941" containerName="console" Sep 06 12:25:12.846161 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:12.846146 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="17ee86bd-4799-47e1-a2d6-268938f99941" containerName="console" Sep 06 12:25:12.850650 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:12.850631 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ptcxv" Sep 06 12:25:12.853504 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:12.853483 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 06 12:25:12.853900 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:12.853881 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 06 12:25:12.853992 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:12.853891 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-x9427\"" Sep 06 12:25:12.869809 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:12.869788 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ptcxv"] Sep 06 12:25:12.950257 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:12.950224 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f8c8b41-cd2a-4f60-b4c3-0909152c6819-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-ptcxv\" (UID: \"1f8c8b41-cd2a-4f60-b4c3-0909152c6819\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ptcxv" Sep 06 12:25:12.950397 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:12.950312 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb76x\" (UniqueName: \"kubernetes.io/projected/1f8c8b41-cd2a-4f60-b4c3-0909152c6819-kube-api-access-hb76x\") pod \"kuadrant-operator-controller-manager-6bd956ccf-ptcxv\" (UID: \"1f8c8b41-cd2a-4f60-b4c3-0909152c6819\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ptcxv" Sep 06 12:25:13.050705 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:13.050673 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f8c8b41-cd2a-4f60-b4c3-0909152c6819-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-ptcxv\" (UID: \"1f8c8b41-cd2a-4f60-b4c3-0909152c6819\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ptcxv" Sep 06 12:25:13.050853 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:13.050731 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb76x\" (UniqueName: \"kubernetes.io/projected/1f8c8b41-cd2a-4f60-b4c3-0909152c6819-kube-api-access-hb76x\") pod \"kuadrant-operator-controller-manager-6bd956ccf-ptcxv\" (UID: \"1f8c8b41-cd2a-4f60-b4c3-0909152c6819\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ptcxv" Sep 06 12:25:13.051056 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:13.051037 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f8c8b41-cd2a-4f60-b4c3-0909152c6819-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-ptcxv\" (UID: \"1f8c8b41-cd2a-4f60-b4c3-0909152c6819\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ptcxv" Sep 06 12:25:13.059821 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:13.059798 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb76x\" (UniqueName: \"kubernetes.io/projected/1f8c8b41-cd2a-4f60-b4c3-0909152c6819-kube-api-access-hb76x\") pod \"kuadrant-operator-controller-manager-6bd956ccf-ptcxv\" (UID: \"1f8c8b41-cd2a-4f60-b4c3-0909152c6819\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ptcxv" Sep 06 12:25:13.164231 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:13.164208 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ptcxv" Sep 06 12:25:13.441800 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:13.441778 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ptcxv"] Sep 06 12:25:13.444171 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:25:13.444137 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f8c8b41_cd2a_4f60_b4c3_0909152c6819.slice/crio-ff724231cc2f2c0dd65fe1d58da1382ccd40d3d4de960ed710b5c0f3f2aac499 WatchSource:0}: Error finding container ff724231cc2f2c0dd65fe1d58da1382ccd40d3d4de960ed710b5c0f3f2aac499: Status 404 returned error can't find the container with id ff724231cc2f2c0dd65fe1d58da1382ccd40d3d4de960ed710b5c0f3f2aac499 Sep 06 12:25:13.782602 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:13.782526 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ptcxv" event={"ID":"1f8c8b41-cd2a-4f60-b4c3-0909152c6819","Type":"ContainerStarted","Data":"ff724231cc2f2c0dd65fe1d58da1382ccd40d3d4de960ed710b5c0f3f2aac499"} Sep 06 12:25:17.808448 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:17.808365 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ptcxv" event={"ID":"1f8c8b41-cd2a-4f60-b4c3-0909152c6819","Type":"ContainerStarted","Data":"540a6a8b51817b6d40e528614d44f476a8129ee21d53bc9d1967c86ffea78d0f"} Sep 06 12:25:17.808830 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:17.808502 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ptcxv" Sep 06 12:25:17.829080 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:17.829031 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ptcxv" podStartSLOduration=1.84836849 podStartE2EDuration="5.829017455s" totalImagesPullingTime="3.980648965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:25:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:25:17.826348837 +0000 UTC m=+433.682106739" watchObservedRunningTime="2026-09-06 12:25:17.829017455 +0000 UTC m=+433.684775342" Sep 06 12:25:28.816175 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:28.816092 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ptcxv" Sep 06 12:25:41.046538 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:41.046505 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ptcxv"] Sep 06 12:25:41.047107 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:41.046695 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ptcxv" podUID="1f8c8b41-cd2a-4f60-b4c3-0909152c6819" containerName="manager" containerID="cri-o://540a6a8b51817b6d40e528614d44f476a8129ee21d53bc9d1967c86ffea78d0f" gracePeriod=10 Sep 06 12:25:41.287957 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:41.287932 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ptcxv" Sep 06 12:25:41.385936 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:41.385876 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hb76x\" (UniqueName: \"kubernetes.io/projected/1f8c8b41-cd2a-4f60-b4c3-0909152c6819-kube-api-access-hb76x\") pod \"1f8c8b41-cd2a-4f60-b4c3-0909152c6819\" (UID: \"1f8c8b41-cd2a-4f60-b4c3-0909152c6819\") " Sep 06 12:25:41.386056 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:41.385978 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f8c8b41-cd2a-4f60-b4c3-0909152c6819-extensions-socket-volume\") pod \"1f8c8b41-cd2a-4f60-b4c3-0909152c6819\" (UID: \"1f8c8b41-cd2a-4f60-b4c3-0909152c6819\") " Sep 06 12:25:41.386434 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:41.386410 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f8c8b41-cd2a-4f60-b4c3-0909152c6819-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1f8c8b41-cd2a-4f60-b4c3-0909152c6819" (UID: "1f8c8b41-cd2a-4f60-b4c3-0909152c6819"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 12:25:41.388101 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:41.388079 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f8c8b41-cd2a-4f60-b4c3-0909152c6819-kube-api-access-hb76x" (OuterVolumeSpecName: "kube-api-access-hb76x") pod "1f8c8b41-cd2a-4f60-b4c3-0909152c6819" (UID: "1f8c8b41-cd2a-4f60-b4c3-0909152c6819"). InnerVolumeSpecName "kube-api-access-hb76x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:25:41.486799 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:41.486774 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f8c8b41-cd2a-4f60-b4c3-0909152c6819-extensions-socket-volume\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:25:41.486799 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:41.486798 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb76x\" (UniqueName: \"kubernetes.io/projected/1f8c8b41-cd2a-4f60-b4c3-0909152c6819-kube-api-access-hb76x\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:25:41.927633 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:41.927605 2576 generic.go:356] "Generic (PLEG): container finished" podID="1f8c8b41-cd2a-4f60-b4c3-0909152c6819" containerID="540a6a8b51817b6d40e528614d44f476a8129ee21d53bc9d1967c86ffea78d0f" exitCode=0 Sep 06 12:25:41.927816 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:41.927642 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ptcxv" Sep 06 12:25:41.927816 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:41.927651 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ptcxv" event={"ID":"1f8c8b41-cd2a-4f60-b4c3-0909152c6819","Type":"ContainerDied","Data":"540a6a8b51817b6d40e528614d44f476a8129ee21d53bc9d1967c86ffea78d0f"} Sep 06 12:25:41.927816 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:41.927699 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ptcxv" event={"ID":"1f8c8b41-cd2a-4f60-b4c3-0909152c6819","Type":"ContainerDied","Data":"ff724231cc2f2c0dd65fe1d58da1382ccd40d3d4de960ed710b5c0f3f2aac499"} Sep 06 12:25:41.927816 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:41.927724 2576 scope.go:117] "RemoveContainer" containerID="540a6a8b51817b6d40e528614d44f476a8129ee21d53bc9d1967c86ffea78d0f" Sep 06 12:25:41.937061 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:41.937047 2576 scope.go:117] "RemoveContainer" containerID="540a6a8b51817b6d40e528614d44f476a8129ee21d53bc9d1967c86ffea78d0f" Sep 06 12:25:41.937327 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:25:41.937310 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"540a6a8b51817b6d40e528614d44f476a8129ee21d53bc9d1967c86ffea78d0f\": container with ID starting with 540a6a8b51817b6d40e528614d44f476a8129ee21d53bc9d1967c86ffea78d0f not found: ID does not exist" containerID="540a6a8b51817b6d40e528614d44f476a8129ee21d53bc9d1967c86ffea78d0f" Sep 06 12:25:41.937371 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:41.937335 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"540a6a8b51817b6d40e528614d44f476a8129ee21d53bc9d1967c86ffea78d0f"} err="failed to get container status \"540a6a8b51817b6d40e528614d44f476a8129ee21d53bc9d1967c86ffea78d0f\": rpc error: code = NotFound desc = could not find container \"540a6a8b51817b6d40e528614d44f476a8129ee21d53bc9d1967c86ffea78d0f\": container with ID starting with 540a6a8b51817b6d40e528614d44f476a8129ee21d53bc9d1967c86ffea78d0f not found: ID does not exist" Sep 06 12:25:41.953937 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:41.953915 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ptcxv"] Sep 06 12:25:41.960053 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:41.960032 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ptcxv"] Sep 06 12:25:42.653435 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:42.653400 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f8c8b41-cd2a-4f60-b4c3-0909152c6819" path="/var/lib/kubelet/pods/1f8c8b41-cd2a-4f60-b4c3-0909152c6819/volumes" Sep 06 12:25:59.982722 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:59.982683 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-x66gg"] Sep 06 12:25:59.983287 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:59.983226 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f8c8b41-cd2a-4f60-b4c3-0909152c6819" containerName="manager" Sep 06 12:25:59.983287 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:59.983248 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f8c8b41-cd2a-4f60-b4c3-0909152c6819" containerName="manager" Sep 06 12:25:59.983403 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:59.983360 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f8c8b41-cd2a-4f60-b4c3-0909152c6819" containerName="manager" Sep 06 12:25:59.986247 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:59.986227 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-x66gg" Sep 06 12:25:59.988444 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:59.988419 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 06 12:25:59.988551 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:59.988443 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qztpv\"" Sep 06 12:25:59.989278 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:59.989219 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 06 12:25:59.989446 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:59.989428 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 06 12:25:59.991273 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:25:59.991255 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-x66gg"] Sep 06 12:26:00.080971 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:00.080936 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-x66gg"] Sep 06 12:26:00.125194 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:00.125168 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9l9k\" (UniqueName: \"kubernetes.io/projected/3912181a-66a4-46a4-96bd-ff1bd8643bbc-kube-api-access-q9l9k\") pod \"limitador-limitador-859c4d96d6-x66gg\" (UID: \"3912181a-66a4-46a4-96bd-ff1bd8643bbc\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-x66gg" Sep 06 12:26:00.125334 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:00.125212 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3912181a-66a4-46a4-96bd-ff1bd8643bbc-config-file\") pod \"limitador-limitador-859c4d96d6-x66gg\" (UID: \"3912181a-66a4-46a4-96bd-ff1bd8643bbc\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-x66gg" Sep 06 12:26:00.226164 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:00.226135 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9l9k\" (UniqueName: \"kubernetes.io/projected/3912181a-66a4-46a4-96bd-ff1bd8643bbc-kube-api-access-q9l9k\") pod \"limitador-limitador-859c4d96d6-x66gg\" (UID: \"3912181a-66a4-46a4-96bd-ff1bd8643bbc\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-x66gg" Sep 06 12:26:00.226293 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:00.226178 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3912181a-66a4-46a4-96bd-ff1bd8643bbc-config-file\") pod \"limitador-limitador-859c4d96d6-x66gg\" (UID: \"3912181a-66a4-46a4-96bd-ff1bd8643bbc\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-x66gg" Sep 06 12:26:00.226734 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:00.226717 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3912181a-66a4-46a4-96bd-ff1bd8643bbc-config-file\") pod \"limitador-limitador-859c4d96d6-x66gg\" (UID: \"3912181a-66a4-46a4-96bd-ff1bd8643bbc\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-x66gg" Sep 06 12:26:00.233776 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:00.233714 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9l9k\" (UniqueName: \"kubernetes.io/projected/3912181a-66a4-46a4-96bd-ff1bd8643bbc-kube-api-access-q9l9k\") pod \"limitador-limitador-859c4d96d6-x66gg\" (UID: \"3912181a-66a4-46a4-96bd-ff1bd8643bbc\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-x66gg" Sep 06 12:26:00.297630 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:00.297607 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-x66gg" Sep 06 12:26:00.570970 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:00.570949 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-x66gg"] Sep 06 12:26:00.573453 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:26:00.573421 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3912181a_66a4_46a4_96bd_ff1bd8643bbc.slice/crio-e52a47c3302bcbce95d9541559c8c0b841c7f155b3efe5081ca7654a5f2df2c4 WatchSource:0}: Error finding container e52a47c3302bcbce95d9541559c8c0b841c7f155b3efe5081ca7654a5f2df2c4: Status 404 returned error can't find the container with id e52a47c3302bcbce95d9541559c8c0b841c7f155b3efe5081ca7654a5f2df2c4 Sep 06 12:26:01.026033 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:01.026000 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-x66gg" event={"ID":"3912181a-66a4-46a4-96bd-ff1bd8643bbc","Type":"ContainerStarted","Data":"e52a47c3302bcbce95d9541559c8c0b841c7f155b3efe5081ca7654a5f2df2c4"} Sep 06 12:26:02.038434 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:02.038396 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-x66gg" event={"ID":"3912181a-66a4-46a4-96bd-ff1bd8643bbc","Type":"ContainerStarted","Data":"c42b0648cfa953a881c7fa94ee56f6bf3ed1e7a6effea815463766c7160b3f54"} Sep 06 12:26:02.038872 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:02.038489 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-x66gg" Sep 06 12:26:02.053603 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:02.053558 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-x66gg" podStartSLOduration=1.79540813 podStartE2EDuration="3.053543857s" totalImagesPullingTime="1.258135727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:25:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:26:02.052036275 +0000 UTC m=+477.907794204" watchObservedRunningTime="2026-09-06 12:26:02.053543857 +0000 UTC m=+477.909301755" Sep 06 12:26:13.045001 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:13.044963 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-x66gg" Sep 06 12:26:13.232670 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:13.232639 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-x66gg"] Sep 06 12:26:13.232907 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:13.232881 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-x66gg" podUID="3912181a-66a4-46a4-96bd-ff1bd8643bbc" containerName="limitador" containerID="cri-o://c42b0648cfa953a881c7fa94ee56f6bf3ed1e7a6effea815463766c7160b3f54" gracePeriod=30 Sep 06 12:26:13.771984 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:13.771953 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-x66gg" Sep 06 12:26:13.835726 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:13.835657 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3912181a-66a4-46a4-96bd-ff1bd8643bbc-config-file\") pod \"3912181a-66a4-46a4-96bd-ff1bd8643bbc\" (UID: \"3912181a-66a4-46a4-96bd-ff1bd8643bbc\") " Sep 06 12:26:13.835726 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:13.835714 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9l9k\" (UniqueName: \"kubernetes.io/projected/3912181a-66a4-46a4-96bd-ff1bd8643bbc-kube-api-access-q9l9k\") pod \"3912181a-66a4-46a4-96bd-ff1bd8643bbc\" (UID: \"3912181a-66a4-46a4-96bd-ff1bd8643bbc\") " Sep 06 12:26:13.836014 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:13.835987 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3912181a-66a4-46a4-96bd-ff1bd8643bbc-config-file" (OuterVolumeSpecName: "config-file") pod "3912181a-66a4-46a4-96bd-ff1bd8643bbc" (UID: "3912181a-66a4-46a4-96bd-ff1bd8643bbc"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:26:13.837810 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:13.837789 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3912181a-66a4-46a4-96bd-ff1bd8643bbc-kube-api-access-q9l9k" (OuterVolumeSpecName: "kube-api-access-q9l9k") pod "3912181a-66a4-46a4-96bd-ff1bd8643bbc" (UID: "3912181a-66a4-46a4-96bd-ff1bd8643bbc"). InnerVolumeSpecName "kube-api-access-q9l9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:26:13.936841 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:13.936817 2576 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3912181a-66a4-46a4-96bd-ff1bd8643bbc-config-file\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:26:13.936841 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:13.936838 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9l9k\" (UniqueName: \"kubernetes.io/projected/3912181a-66a4-46a4-96bd-ff1bd8643bbc-kube-api-access-q9l9k\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:26:14.100416 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:14.100344 2576 generic.go:356] "Generic (PLEG): container finished" podID="3912181a-66a4-46a4-96bd-ff1bd8643bbc" containerID="c42b0648cfa953a881c7fa94ee56f6bf3ed1e7a6effea815463766c7160b3f54" exitCode=0 Sep 06 12:26:14.100416 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:14.100402 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-x66gg" Sep 06 12:26:14.100840 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:14.100441 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-x66gg" event={"ID":"3912181a-66a4-46a4-96bd-ff1bd8643bbc","Type":"ContainerDied","Data":"c42b0648cfa953a881c7fa94ee56f6bf3ed1e7a6effea815463766c7160b3f54"} Sep 06 12:26:14.100840 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:14.100534 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-x66gg" event={"ID":"3912181a-66a4-46a4-96bd-ff1bd8643bbc","Type":"ContainerDied","Data":"e52a47c3302bcbce95d9541559c8c0b841c7f155b3efe5081ca7654a5f2df2c4"} Sep 06 12:26:14.100840 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:14.100566 2576 scope.go:117] "RemoveContainer" containerID="c42b0648cfa953a881c7fa94ee56f6bf3ed1e7a6effea815463766c7160b3f54" Sep 06 12:26:14.110599 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:14.110578 2576 scope.go:117] "RemoveContainer" containerID="c42b0648cfa953a881c7fa94ee56f6bf3ed1e7a6effea815463766c7160b3f54" Sep 06 12:26:14.110979 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:26:14.110958 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c42b0648cfa953a881c7fa94ee56f6bf3ed1e7a6effea815463766c7160b3f54\": container with ID starting with c42b0648cfa953a881c7fa94ee56f6bf3ed1e7a6effea815463766c7160b3f54 not found: ID does not exist" containerID="c42b0648cfa953a881c7fa94ee56f6bf3ed1e7a6effea815463766c7160b3f54" Sep 06 12:26:14.111033 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:14.110989 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c42b0648cfa953a881c7fa94ee56f6bf3ed1e7a6effea815463766c7160b3f54"} err="failed to get container status \"c42b0648cfa953a881c7fa94ee56f6bf3ed1e7a6effea815463766c7160b3f54\": rpc error: code = NotFound desc = could not find container \"c42b0648cfa953a881c7fa94ee56f6bf3ed1e7a6effea815463766c7160b3f54\": container with ID starting with c42b0648cfa953a881c7fa94ee56f6bf3ed1e7a6effea815463766c7160b3f54 not found: ID does not exist" Sep 06 12:26:14.121411 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:14.121384 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-x66gg"] Sep 06 12:26:14.126456 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:14.126430 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-x66gg"] Sep 06 12:26:14.652811 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:14.652777 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3912181a-66a4-46a4-96bd-ff1bd8643bbc" path="/var/lib/kubelet/pods/3912181a-66a4-46a4-96bd-ff1bd8643bbc/volumes" Sep 06 12:26:16.832134 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:16.832096 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-qhlnq"] Sep 06 12:26:16.833187 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:16.833161 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3912181a-66a4-46a4-96bd-ff1bd8643bbc" containerName="limitador" Sep 06 12:26:16.833187 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:16.833190 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3912181a-66a4-46a4-96bd-ff1bd8643bbc" containerName="limitador" Sep 06 12:26:16.833338 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:16.833292 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3912181a-66a4-46a4-96bd-ff1bd8643bbc" containerName="limitador" Sep 06 12:26:16.838123 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:16.838099 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qhlnq" Sep 06 12:26:16.840727 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:16.840658 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 06 12:26:16.840727 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:16.840711 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 06 12:26:16.841006 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:16.840991 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 06 12:26:16.841286 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:16.841255 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-tfz9t\"" Sep 06 12:26:16.841783 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:16.841742 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-qhlnq"] Sep 06 12:26:16.859030 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:16.859009 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/de9a2051-cb39-412b-ae3e-2a135c639cf2-data\") pod \"postgres-69fb4594cf-qhlnq\" (UID: \"de9a2051-cb39-412b-ae3e-2a135c639cf2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qhlnq" Sep 06 12:26:16.859121 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:16.859042 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgkc8\" (UniqueName: \"kubernetes.io/projected/de9a2051-cb39-412b-ae3e-2a135c639cf2-kube-api-access-qgkc8\") pod \"postgres-69fb4594cf-qhlnq\" (UID: \"de9a2051-cb39-412b-ae3e-2a135c639cf2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qhlnq" Sep 06 12:26:16.959593 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:16.959571 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/de9a2051-cb39-412b-ae3e-2a135c639cf2-data\") pod \"postgres-69fb4594cf-qhlnq\" (UID: \"de9a2051-cb39-412b-ae3e-2a135c639cf2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qhlnq" Sep 06 12:26:16.959699 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:16.959605 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgkc8\" (UniqueName: \"kubernetes.io/projected/de9a2051-cb39-412b-ae3e-2a135c639cf2-kube-api-access-qgkc8\") pod \"postgres-69fb4594cf-qhlnq\" (UID: \"de9a2051-cb39-412b-ae3e-2a135c639cf2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qhlnq" Sep 06 12:26:16.959928 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:16.959909 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/de9a2051-cb39-412b-ae3e-2a135c639cf2-data\") pod \"postgres-69fb4594cf-qhlnq\" (UID: \"de9a2051-cb39-412b-ae3e-2a135c639cf2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qhlnq" Sep 06 12:26:16.967080 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:16.967061 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgkc8\" (UniqueName: \"kubernetes.io/projected/de9a2051-cb39-412b-ae3e-2a135c639cf2-kube-api-access-qgkc8\") pod \"postgres-69fb4594cf-qhlnq\" (UID: \"de9a2051-cb39-412b-ae3e-2a135c639cf2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qhlnq" Sep 06 12:26:17.149803 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:17.149776 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qhlnq" Sep 06 12:26:17.420627 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:17.420601 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-qhlnq"] Sep 06 12:26:17.423285 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:26:17.423255 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde9a2051_cb39_412b_ae3e_2a135c639cf2.slice/crio-60e74519864d7a9bc6aad14e6c4930d4fe5860fcc8cdc5cd27f679738e3f9876 WatchSource:0}: Error finding container 60e74519864d7a9bc6aad14e6c4930d4fe5860fcc8cdc5cd27f679738e3f9876: Status 404 returned error can't find the container with id 60e74519864d7a9bc6aad14e6c4930d4fe5860fcc8cdc5cd27f679738e3f9876 Sep 06 12:26:18.123143 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:18.123103 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qhlnq" event={"ID":"de9a2051-cb39-412b-ae3e-2a135c639cf2","Type":"ContainerStarted","Data":"60e74519864d7a9bc6aad14e6c4930d4fe5860fcc8cdc5cd27f679738e3f9876"} Sep 06 12:26:23.152657 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:23.152623 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qhlnq" event={"ID":"de9a2051-cb39-412b-ae3e-2a135c639cf2","Type":"ContainerStarted","Data":"ad6f7715f9d6d1cb3f2228d8b03908145c245f15c4c2563cc10ec838c60dc232"} Sep 06 12:26:23.153084 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:23.152782 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qhlnq" Sep 06 12:26:23.167995 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:23.167950 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qhlnq" podStartSLOduration=2.204413512 podStartE2EDuration="7.167937213s" totalImagesPullingTime="4.963523701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:26:23.165320895 +0000 UTC m=+499.021078781" watchObservedRunningTime="2026-09-06 12:26:23.167937213 +0000 UTC m=+499.023695099" Sep 06 12:26:29.189318 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:29.189286 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qhlnq" Sep 06 12:26:35.927650 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:35.927620 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5fdc88c8d6-q55j9"] Sep 06 12:26:35.951462 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:35.951434 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fdc88c8d6-q55j9"] Sep 06 12:26:35.951602 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:35.951523 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fdc88c8d6-q55j9" Sep 06 12:26:35.953964 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:35.953926 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 06 12:26:35.953964 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:35.953944 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 06 12:26:35.954152 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:35.953936 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-s9zss\"" Sep 06 12:26:35.954152 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:35.953933 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 06 12:26:36.022534 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:36.022509 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40373fd7-1df4-4ee2-b9f3-4ee79c368a70-metrics-certs\") pod \"maas-controller-5fdc88c8d6-q55j9\" (UID: \"40373fd7-1df4-4ee2-b9f3-4ee79c368a70\") " pod="opendatahub/maas-controller-5fdc88c8d6-q55j9" Sep 06 12:26:36.022647 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:36.022555 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55qwh\" (UniqueName: \"kubernetes.io/projected/40373fd7-1df4-4ee2-b9f3-4ee79c368a70-kube-api-access-55qwh\") pod \"maas-controller-5fdc88c8d6-q55j9\" (UID: \"40373fd7-1df4-4ee2-b9f3-4ee79c368a70\") " pod="opendatahub/maas-controller-5fdc88c8d6-q55j9" Sep 06 12:26:36.022692 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:36.022638 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40373fd7-1df4-4ee2-b9f3-4ee79c368a70-webhook-cert\") pod \"maas-controller-5fdc88c8d6-q55j9\" (UID: \"40373fd7-1df4-4ee2-b9f3-4ee79c368a70\") " pod="opendatahub/maas-controller-5fdc88c8d6-q55j9" Sep 06 12:26:36.097681 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:36.097653 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-679b9dbb-8z9qh"] Sep 06 12:26:36.101339 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:36.101322 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-679b9dbb-8z9qh" Sep 06 12:26:36.111930 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:36.111906 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-679b9dbb-8z9qh"] Sep 06 12:26:36.123451 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:36.123423 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40373fd7-1df4-4ee2-b9f3-4ee79c368a70-metrics-certs\") pod \"maas-controller-5fdc88c8d6-q55j9\" (UID: \"40373fd7-1df4-4ee2-b9f3-4ee79c368a70\") " pod="opendatahub/maas-controller-5fdc88c8d6-q55j9" Sep 06 12:26:36.123546 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:36.123473 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55qwh\" (UniqueName: \"kubernetes.io/projected/40373fd7-1df4-4ee2-b9f3-4ee79c368a70-kube-api-access-55qwh\") pod \"maas-controller-5fdc88c8d6-q55j9\" (UID: \"40373fd7-1df4-4ee2-b9f3-4ee79c368a70\") " pod="opendatahub/maas-controller-5fdc88c8d6-q55j9" Sep 06 12:26:36.123546 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:36.123522 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40373fd7-1df4-4ee2-b9f3-4ee79c368a70-webhook-cert\") pod \"maas-controller-5fdc88c8d6-q55j9\" (UID: \"40373fd7-1df4-4ee2-b9f3-4ee79c368a70\") " pod="opendatahub/maas-controller-5fdc88c8d6-q55j9" Sep 06 12:26:36.123616 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:26:36.123552 2576 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found Sep 06 12:26:36.123616 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:26:36.123615 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40373fd7-1df4-4ee2-b9f3-4ee79c368a70-metrics-certs podName:40373fd7-1df4-4ee2-b9f3-4ee79c368a70 nodeName:}" failed. No retries permitted until 2026-09-06 12:26:36.623599874 +0000 UTC m=+512.479357765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40373fd7-1df4-4ee2-b9f3-4ee79c368a70-metrics-certs") pod "maas-controller-5fdc88c8d6-q55j9" (UID: "40373fd7-1df4-4ee2-b9f3-4ee79c368a70") : secret "maas-controller-metrics-tls" not found Sep 06 12:26:36.123697 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:26:36.123626 2576 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Sep 06 12:26:36.123697 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:26:36.123667 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40373fd7-1df4-4ee2-b9f3-4ee79c368a70-webhook-cert podName:40373fd7-1df4-4ee2-b9f3-4ee79c368a70 nodeName:}" failed. No retries permitted until 2026-09-06 12:26:36.623650563 +0000 UTC m=+512.479408427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/40373fd7-1df4-4ee2-b9f3-4ee79c368a70-webhook-cert") pod "maas-controller-5fdc88c8d6-q55j9" (UID: "40373fd7-1df4-4ee2-b9f3-4ee79c368a70") : secret "maas-controller-webhook-cert" not found Sep 06 12:26:36.131209 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:36.131183 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55qwh\" (UniqueName: \"kubernetes.io/projected/40373fd7-1df4-4ee2-b9f3-4ee79c368a70-kube-api-access-55qwh\") pod \"maas-controller-5fdc88c8d6-q55j9\" (UID: \"40373fd7-1df4-4ee2-b9f3-4ee79c368a70\") " pod="opendatahub/maas-controller-5fdc88c8d6-q55j9" Sep 06 12:26:36.224379 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:36.224314 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/199c3984-3125-4b84-b273-58a686af4adc-webhook-cert\") pod \"maas-controller-679b9dbb-8z9qh\" (UID: \"199c3984-3125-4b84-b273-58a686af4adc\") " pod="opendatahub/maas-controller-679b9dbb-8z9qh" Sep 06 12:26:36.224487 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:36.224383 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/199c3984-3125-4b84-b273-58a686af4adc-metrics-certs\") pod \"maas-controller-679b9dbb-8z9qh\" (UID: \"199c3984-3125-4b84-b273-58a686af4adc\") " pod="opendatahub/maas-controller-679b9dbb-8z9qh" Sep 06 12:26:36.224544 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:36.224489 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jswb8\" (UniqueName: \"kubernetes.io/projected/199c3984-3125-4b84-b273-58a686af4adc-kube-api-access-jswb8\") pod \"maas-controller-679b9dbb-8z9qh\" (UID: \"199c3984-3125-4b84-b273-58a686af4adc\") " pod="opendatahub/maas-controller-679b9dbb-8z9qh" Sep 06 12:26:36.324902 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:36.324878 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/199c3984-3125-4b84-b273-58a686af4adc-webhook-cert\") pod \"maas-controller-679b9dbb-8z9qh\" (UID: \"199c3984-3125-4b84-b273-58a686af4adc\") " pod="opendatahub/maas-controller-679b9dbb-8z9qh" Sep 06 12:26:36.325027 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:36.324939 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/199c3984-3125-4b84-b273-58a686af4adc-metrics-certs\") pod \"maas-controller-679b9dbb-8z9qh\" (UID: \"199c3984-3125-4b84-b273-58a686af4adc\") " pod="opendatahub/maas-controller-679b9dbb-8z9qh" Sep 06 12:26:36.325027 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:26:36.325018 2576 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Sep 06 12:26:36.325141 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:26:36.325069 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/199c3984-3125-4b84-b273-58a686af4adc-webhook-cert podName:199c3984-3125-4b84-b273-58a686af4adc nodeName:}" failed. No retries permitted until 2026-09-06 12:26:36.825055615 +0000 UTC m=+512.680813484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/199c3984-3125-4b84-b273-58a686af4adc-webhook-cert") pod "maas-controller-679b9dbb-8z9qh" (UID: "199c3984-3125-4b84-b273-58a686af4adc") : secret "maas-controller-webhook-cert" not found Sep 06 12:26:36.325141 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:36.325020 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jswb8\" (UniqueName: \"kubernetes.io/projected/199c3984-3125-4b84-b273-58a686af4adc-kube-api-access-jswb8\") pod \"maas-controller-679b9dbb-8z9qh\" (UID: \"199c3984-3125-4b84-b273-58a686af4adc\") " pod="opendatahub/maas-controller-679b9dbb-8z9qh" Sep 06 12:26:36.327401 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:36.327382 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/199c3984-3125-4b84-b273-58a686af4adc-metrics-certs\") pod \"maas-controller-679b9dbb-8z9qh\" (UID: \"199c3984-3125-4b84-b273-58a686af4adc\") " pod="opendatahub/maas-controller-679b9dbb-8z9qh" Sep 06 12:26:36.334845 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:36.334826 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jswb8\" (UniqueName: \"kubernetes.io/projected/199c3984-3125-4b84-b273-58a686af4adc-kube-api-access-jswb8\") pod \"maas-controller-679b9dbb-8z9qh\" (UID: \"199c3984-3125-4b84-b273-58a686af4adc\") " pod="opendatahub/maas-controller-679b9dbb-8z9qh" Sep 06 12:26:36.627688 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:36.627624 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40373fd7-1df4-4ee2-b9f3-4ee79c368a70-webhook-cert\") pod \"maas-controller-5fdc88c8d6-q55j9\" (UID: \"40373fd7-1df4-4ee2-b9f3-4ee79c368a70\") " pod="opendatahub/maas-controller-5fdc88c8d6-q55j9" Sep 06 12:26:36.627808 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:36.627689 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40373fd7-1df4-4ee2-b9f3-4ee79c368a70-metrics-certs\") pod \"maas-controller-5fdc88c8d6-q55j9\" (UID: \"40373fd7-1df4-4ee2-b9f3-4ee79c368a70\") " pod="opendatahub/maas-controller-5fdc88c8d6-q55j9" Sep 06 12:26:36.630154 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:36.630123 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40373fd7-1df4-4ee2-b9f3-4ee79c368a70-webhook-cert\") pod \"maas-controller-5fdc88c8d6-q55j9\" (UID: \"40373fd7-1df4-4ee2-b9f3-4ee79c368a70\") " pod="opendatahub/maas-controller-5fdc88c8d6-q55j9" Sep 06 12:26:36.630274 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:36.630247 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40373fd7-1df4-4ee2-b9f3-4ee79c368a70-metrics-certs\") pod \"maas-controller-5fdc88c8d6-q55j9\" (UID: \"40373fd7-1df4-4ee2-b9f3-4ee79c368a70\") " pod="opendatahub/maas-controller-5fdc88c8d6-q55j9" Sep 06 12:26:36.829558 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:36.829538 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/199c3984-3125-4b84-b273-58a686af4adc-webhook-cert\") pod \"maas-controller-679b9dbb-8z9qh\" (UID: \"199c3984-3125-4b84-b273-58a686af4adc\") " pod="opendatahub/maas-controller-679b9dbb-8z9qh" Sep 06 12:26:36.831877 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:36.831854 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/199c3984-3125-4b84-b273-58a686af4adc-webhook-cert\") pod \"maas-controller-679b9dbb-8z9qh\" (UID: \"199c3984-3125-4b84-b273-58a686af4adc\") " pod="opendatahub/maas-controller-679b9dbb-8z9qh" Sep 06 12:26:36.862886 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:36.862864 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fdc88c8d6-q55j9" Sep 06 12:26:37.011902 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:37.011871 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-679b9dbb-8z9qh" Sep 06 12:26:37.131812 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:37.131789 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fdc88c8d6-q55j9"] Sep 06 12:26:37.133185 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:26:37.133160 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40373fd7_1df4_4ee2_b9f3_4ee79c368a70.slice/crio-056e9b96536b27c12b22beeb48a897039845e0079cd2f23aab93c0820423852b WatchSource:0}: Error finding container 056e9b96536b27c12b22beeb48a897039845e0079cd2f23aab93c0820423852b: Status 404 returned error can't find the container with id 056e9b96536b27c12b22beeb48a897039845e0079cd2f23aab93c0820423852b Sep 06 12:26:37.220705 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:37.220645 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fdc88c8d6-q55j9" event={"ID":"40373fd7-1df4-4ee2-b9f3-4ee79c368a70","Type":"ContainerStarted","Data":"056e9b96536b27c12b22beeb48a897039845e0079cd2f23aab93c0820423852b"} Sep 06 12:26:37.279558 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:37.279539 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-679b9dbb-8z9qh"] Sep 06 12:26:37.280912 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:26:37.280885 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod199c3984_3125_4b84_b273_58a686af4adc.slice/crio-6de52cd3d26db508ba07fed5b838ff0ef9dcd927632ee5c632f54782f31a63fe WatchSource:0}: Error finding container 6de52cd3d26db508ba07fed5b838ff0ef9dcd927632ee5c632f54782f31a63fe: Status 404 returned error can't find the container with id 6de52cd3d26db508ba07fed5b838ff0ef9dcd927632ee5c632f54782f31a63fe Sep 06 12:26:38.229143 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:38.229107 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-679b9dbb-8z9qh" event={"ID":"199c3984-3125-4b84-b273-58a686af4adc","Type":"ContainerStarted","Data":"6de52cd3d26db508ba07fed5b838ff0ef9dcd927632ee5c632f54782f31a63fe"} Sep 06 12:26:40.248266 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:40.248231 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fdc88c8d6-q55j9" event={"ID":"40373fd7-1df4-4ee2-b9f3-4ee79c368a70","Type":"ContainerStarted","Data":"6fc89e026443327196e0705e5ad48ae581b2ab98b35f0322be4611a7cfdef2ef"} Sep 06 12:26:40.248744 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:40.248437 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5fdc88c8d6-q55j9" Sep 06 12:26:40.250059 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:40.250034 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-679b9dbb-8z9qh" event={"ID":"199c3984-3125-4b84-b273-58a686af4adc","Type":"ContainerStarted","Data":"01a0ba9cbd53119175d77ef2e1b914d71b08cbe6a71eeb89418faff18228c8e2"} Sep 06 12:26:40.250173 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:40.250082 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-679b9dbb-8z9qh" Sep 06 12:26:40.264448 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:40.264398 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5fdc88c8d6-q55j9" podStartSLOduration=2.576087047 podStartE2EDuration="5.264387241s" totalImagesPullingTime="2.688300194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:26:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:26:40.26304103 +0000 UTC m=+516.118798963" watchObservedRunningTime="2026-09-06 12:26:40.264387241 +0000 UTC m=+516.120145127" Sep 06 12:26:40.280331 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:40.280295 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-679b9dbb-8z9qh" podStartSLOduration=1.729104873 podStartE2EDuration="4.280283805s" totalImagesPullingTime="2.551178932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:26:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:26:40.279704255 +0000 UTC m=+516.135462163" watchObservedRunningTime="2026-09-06 12:26:40.280283805 +0000 UTC m=+516.136041692" Sep 06 12:26:51.257154 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:51.257126 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5fdc88c8d6-q55j9" Sep 06 12:26:51.257539 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:51.257198 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-679b9dbb-8z9qh" Sep 06 12:26:51.319506 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:51.319474 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5fdc88c8d6-q55j9"] Sep 06 12:26:51.319695 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:51.319671 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5fdc88c8d6-q55j9" podUID="40373fd7-1df4-4ee2-b9f3-4ee79c368a70" containerName="manager" containerID="cri-o://6fc89e026443327196e0705e5ad48ae581b2ab98b35f0322be4611a7cfdef2ef" gracePeriod=30 Sep 06 12:26:51.564608 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:51.564584 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fdc88c8d6-q55j9" Sep 06 12:26:51.649575 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:51.649548 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40373fd7-1df4-4ee2-b9f3-4ee79c368a70-webhook-cert\") pod \"40373fd7-1df4-4ee2-b9f3-4ee79c368a70\" (UID: \"40373fd7-1df4-4ee2-b9f3-4ee79c368a70\") " Sep 06 12:26:51.649575 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:51.649578 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40373fd7-1df4-4ee2-b9f3-4ee79c368a70-metrics-certs\") pod \"40373fd7-1df4-4ee2-b9f3-4ee79c368a70\" (UID: \"40373fd7-1df4-4ee2-b9f3-4ee79c368a70\") " Sep 06 12:26:51.649794 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:51.649608 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55qwh\" (UniqueName: \"kubernetes.io/projected/40373fd7-1df4-4ee2-b9f3-4ee79c368a70-kube-api-access-55qwh\") pod \"40373fd7-1df4-4ee2-b9f3-4ee79c368a70\" (UID: \"40373fd7-1df4-4ee2-b9f3-4ee79c368a70\") " Sep 06 12:26:51.651860 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:51.651723 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40373fd7-1df4-4ee2-b9f3-4ee79c368a70-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "40373fd7-1df4-4ee2-b9f3-4ee79c368a70" (UID: "40373fd7-1df4-4ee2-b9f3-4ee79c368a70"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 12:26:51.651860 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:51.651827 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40373fd7-1df4-4ee2-b9f3-4ee79c368a70-kube-api-access-55qwh" (OuterVolumeSpecName: "kube-api-access-55qwh") pod "40373fd7-1df4-4ee2-b9f3-4ee79c368a70" (UID: "40373fd7-1df4-4ee2-b9f3-4ee79c368a70"). InnerVolumeSpecName "kube-api-access-55qwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:26:51.651990 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:51.651872 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40373fd7-1df4-4ee2-b9f3-4ee79c368a70-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "40373fd7-1df4-4ee2-b9f3-4ee79c368a70" (UID: "40373fd7-1df4-4ee2-b9f3-4ee79c368a70"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 12:26:51.750691 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:51.750658 2576 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40373fd7-1df4-4ee2-b9f3-4ee79c368a70-webhook-cert\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:26:51.750691 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:51.750686 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40373fd7-1df4-4ee2-b9f3-4ee79c368a70-metrics-certs\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:26:51.750859 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:51.750700 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55qwh\" (UniqueName: \"kubernetes.io/projected/40373fd7-1df4-4ee2-b9f3-4ee79c368a70-kube-api-access-55qwh\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:26:52.314624 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:52.314580 2576 generic.go:356] "Generic (PLEG): container finished" podID="40373fd7-1df4-4ee2-b9f3-4ee79c368a70" containerID="6fc89e026443327196e0705e5ad48ae581b2ab98b35f0322be4611a7cfdef2ef" exitCode=0 Sep 06 12:26:52.315015 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:52.314643 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fdc88c8d6-q55j9" Sep 06 12:26:52.315015 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:52.314667 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fdc88c8d6-q55j9" event={"ID":"40373fd7-1df4-4ee2-b9f3-4ee79c368a70","Type":"ContainerDied","Data":"6fc89e026443327196e0705e5ad48ae581b2ab98b35f0322be4611a7cfdef2ef"} Sep 06 12:26:52.315015 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:52.314771 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fdc88c8d6-q55j9" event={"ID":"40373fd7-1df4-4ee2-b9f3-4ee79c368a70","Type":"ContainerDied","Data":"056e9b96536b27c12b22beeb48a897039845e0079cd2f23aab93c0820423852b"} Sep 06 12:26:52.315015 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:52.314807 2576 scope.go:117] "RemoveContainer" containerID="6fc89e026443327196e0705e5ad48ae581b2ab98b35f0322be4611a7cfdef2ef" Sep 06 12:26:52.324171 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:52.323985 2576 scope.go:117] "RemoveContainer" containerID="6fc89e026443327196e0705e5ad48ae581b2ab98b35f0322be4611a7cfdef2ef" Sep 06 12:26:52.324302 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:26:52.324265 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fc89e026443327196e0705e5ad48ae581b2ab98b35f0322be4611a7cfdef2ef\": container with ID starting with 6fc89e026443327196e0705e5ad48ae581b2ab98b35f0322be4611a7cfdef2ef not found: ID does not exist" containerID="6fc89e026443327196e0705e5ad48ae581b2ab98b35f0322be4611a7cfdef2ef" Sep 06 12:26:52.324302 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:52.324292 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fc89e026443327196e0705e5ad48ae581b2ab98b35f0322be4611a7cfdef2ef"} err="failed to get container status \"6fc89e026443327196e0705e5ad48ae581b2ab98b35f0322be4611a7cfdef2ef\": rpc error: code = NotFound desc = could not find container \"6fc89e026443327196e0705e5ad48ae581b2ab98b35f0322be4611a7cfdef2ef\": container with ID starting with 6fc89e026443327196e0705e5ad48ae581b2ab98b35f0322be4611a7cfdef2ef not found: ID does not exist" Sep 06 12:26:52.334496 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:52.334468 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5fdc88c8d6-q55j9"] Sep 06 12:26:52.341812 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:52.341792 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5fdc88c8d6-q55j9"] Sep 06 12:26:52.652629 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:26:52.652604 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40373fd7-1df4-4ee2-b9f3-4ee79c368a70" path="/var/lib/kubelet/pods/40373fd7-1df4-4ee2-b9f3-4ee79c368a70/volumes" Sep 06 12:27:14.317345 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.317307 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5b4cdc449f-bdh4x"] Sep 06 12:27:14.317811 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.317695 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40373fd7-1df4-4ee2-b9f3-4ee79c368a70" containerName="manager" Sep 06 12:27:14.317811 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.317708 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="40373fd7-1df4-4ee2-b9f3-4ee79c368a70" containerName="manager" Sep 06 12:27:14.317887 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.317850 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="40373fd7-1df4-4ee2-b9f3-4ee79c368a70" containerName="manager" Sep 06 12:27:14.323599 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.323579 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b4cdc449f-bdh4x" Sep 06 12:27:14.326570 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.326541 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86425b02-9d6f-4d6b-b5e4-f5ade4648b78-metrics-certs\") pod \"maas-api-5b4cdc449f-bdh4x\" (UID: \"86425b02-9d6f-4d6b-b5e4-f5ade4648b78\") " pod="odh-ai-gateway-infra/maas-api-5b4cdc449f-bdh4x" Sep 06 12:27:14.326697 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.326602 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86425b02-9d6f-4d6b-b5e4-f5ade4648b78-openshift-service-ca\") pod \"maas-api-5b4cdc449f-bdh4x\" (UID: \"86425b02-9d6f-4d6b-b5e4-f5ade4648b78\") " pod="odh-ai-gateway-infra/maas-api-5b4cdc449f-bdh4x" Sep 06 12:27:14.326697 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.326646 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j85fr\" (UniqueName: \"kubernetes.io/projected/86425b02-9d6f-4d6b-b5e4-f5ade4648b78-kube-api-access-j85fr\") pod \"maas-api-5b4cdc449f-bdh4x\" (UID: \"86425b02-9d6f-4d6b-b5e4-f5ade4648b78\") " pod="odh-ai-gateway-infra/maas-api-5b4cdc449f-bdh4x" Sep 06 12:27:14.326844 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.326785 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 06 12:27:14.326844 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.326804 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86425b02-9d6f-4d6b-b5e4-f5ade4648b78-maas-api-tls\") pod \"maas-api-5b4cdc449f-bdh4x\" (UID: \"86425b02-9d6f-4d6b-b5e4-f5ade4648b78\") " pod="odh-ai-gateway-infra/maas-api-5b4cdc449f-bdh4x" Sep 06 12:27:14.326844 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.326815 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 06 12:27:14.326996 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.326785 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-w24cl\"" Sep 06 12:27:14.328971 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.328952 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b4cdc449f-bdh4x"] Sep 06 12:27:14.427328 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.427300 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86425b02-9d6f-4d6b-b5e4-f5ade4648b78-maas-api-tls\") pod \"maas-api-5b4cdc449f-bdh4x\" (UID: \"86425b02-9d6f-4d6b-b5e4-f5ade4648b78\") " pod="odh-ai-gateway-infra/maas-api-5b4cdc449f-bdh4x" Sep 06 12:27:14.427461 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.427357 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86425b02-9d6f-4d6b-b5e4-f5ade4648b78-metrics-certs\") pod \"maas-api-5b4cdc449f-bdh4x\" (UID: \"86425b02-9d6f-4d6b-b5e4-f5ade4648b78\") " pod="odh-ai-gateway-infra/maas-api-5b4cdc449f-bdh4x" Sep 06 12:27:14.427461 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.427390 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86425b02-9d6f-4d6b-b5e4-f5ade4648b78-openshift-service-ca\") pod \"maas-api-5b4cdc449f-bdh4x\" (UID: \"86425b02-9d6f-4d6b-b5e4-f5ade4648b78\") " pod="odh-ai-gateway-infra/maas-api-5b4cdc449f-bdh4x" Sep 06 12:27:14.427461 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.427421 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j85fr\" (UniqueName: \"kubernetes.io/projected/86425b02-9d6f-4d6b-b5e4-f5ade4648b78-kube-api-access-j85fr\") pod \"maas-api-5b4cdc449f-bdh4x\" (UID: \"86425b02-9d6f-4d6b-b5e4-f5ade4648b78\") " pod="odh-ai-gateway-infra/maas-api-5b4cdc449f-bdh4x" Sep 06 12:27:14.427988 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.427967 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86425b02-9d6f-4d6b-b5e4-f5ade4648b78-openshift-service-ca\") pod \"maas-api-5b4cdc449f-bdh4x\" (UID: \"86425b02-9d6f-4d6b-b5e4-f5ade4648b78\") " pod="odh-ai-gateway-infra/maas-api-5b4cdc449f-bdh4x" Sep 06 12:27:14.429825 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.429800 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86425b02-9d6f-4d6b-b5e4-f5ade4648b78-maas-api-tls\") pod \"maas-api-5b4cdc449f-bdh4x\" (UID: \"86425b02-9d6f-4d6b-b5e4-f5ade4648b78\") " pod="odh-ai-gateway-infra/maas-api-5b4cdc449f-bdh4x" Sep 06 12:27:14.429917 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.429902 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86425b02-9d6f-4d6b-b5e4-f5ade4648b78-metrics-certs\") pod \"maas-api-5b4cdc449f-bdh4x\" (UID: \"86425b02-9d6f-4d6b-b5e4-f5ade4648b78\") " pod="odh-ai-gateway-infra/maas-api-5b4cdc449f-bdh4x" Sep 06 12:27:14.435018 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.434998 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j85fr\" (UniqueName: \"kubernetes.io/projected/86425b02-9d6f-4d6b-b5e4-f5ade4648b78-kube-api-access-j85fr\") pod \"maas-api-5b4cdc449f-bdh4x\" (UID: \"86425b02-9d6f-4d6b-b5e4-f5ade4648b78\") " pod="odh-ai-gateway-infra/maas-api-5b4cdc449f-bdh4x" Sep 06 12:27:14.636386 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.636300 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b4cdc449f-bdh4x" Sep 06 12:27:14.768209 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.768178 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-t9grp"] Sep 06 12:27:14.771834 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.771813 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-t9grp" Sep 06 12:27:14.774105 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.774082 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 06 12:27:14.774262 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.774245 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-2ncrb\"" Sep 06 12:27:14.782953 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.782934 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-t9grp"] Sep 06 12:27:14.821822 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.821798 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-jlvzm"] Sep 06 12:27:14.825185 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.825170 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-jlvzm" Sep 06 12:27:14.830189 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.830163 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p28c\" (UniqueName: \"kubernetes.io/projected/2a0a23e6-8a48-4825-b2d7-e6bfd31fbe1f-kube-api-access-9p28c\") pod \"payload-processing-5b54b7fbb7-jlvzm\" (UID: \"2a0a23e6-8a48-4825-b2d7-e6bfd31fbe1f\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-jlvzm" Sep 06 12:27:14.830282 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.830216 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a0a23e6-8a48-4825-b2d7-e6bfd31fbe1f-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-jlvzm\" (UID: \"2a0a23e6-8a48-4825-b2d7-e6bfd31fbe1f\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-jlvzm" Sep 06 12:27:14.835891 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.835867 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-jlvzm"] Sep 06 12:27:14.920409 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.920390 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b4cdc449f-bdh4x"] Sep 06 12:27:14.922651 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:27:14.922629 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86425b02_9d6f_4d6b_b5e4_f5ade4648b78.slice/crio-4d5de4915e1f1072e11ed5ef2f21d0f11abc852f2971ac2ef4de9c7dda73ee13 WatchSource:0}: Error finding container 4d5de4915e1f1072e11ed5ef2f21d0f11abc852f2971ac2ef4de9c7dda73ee13: Status 404 returned error can't find the container with id 4d5de4915e1f1072e11ed5ef2f21d0f11abc852f2971ac2ef4de9c7dda73ee13 Sep 06 12:27:14.931264 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.931239 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a0a23e6-8a48-4825-b2d7-e6bfd31fbe1f-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-jlvzm\" (UID: \"2a0a23e6-8a48-4825-b2d7-e6bfd31fbe1f\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-jlvzm" Sep 06 12:27:14.931375 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.931279 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f24br\" (UniqueName: \"kubernetes.io/projected/33750f3b-84a4-4935-a768-e2f5bda50eb3-kube-api-access-f24br\") pod \"payload-pre-processing-59fbc47bf8-t9grp\" (UID: \"33750f3b-84a4-4935-a768-e2f5bda50eb3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-t9grp" Sep 06 12:27:14.931375 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.931362 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p28c\" (UniqueName: \"kubernetes.io/projected/2a0a23e6-8a48-4825-b2d7-e6bfd31fbe1f-kube-api-access-9p28c\") pod \"payload-processing-5b54b7fbb7-jlvzm\" (UID: \"2a0a23e6-8a48-4825-b2d7-e6bfd31fbe1f\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-jlvzm" Sep 06 12:27:14.931504 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.931418 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33750f3b-84a4-4935-a768-e2f5bda50eb3-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-t9grp\" (UID: \"33750f3b-84a4-4935-a768-e2f5bda50eb3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-t9grp" Sep 06 12:27:14.931950 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.931931 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a0a23e6-8a48-4825-b2d7-e6bfd31fbe1f-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-jlvzm\" (UID: \"2a0a23e6-8a48-4825-b2d7-e6bfd31fbe1f\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-jlvzm" Sep 06 12:27:14.939783 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:14.939743 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p28c\" (UniqueName: \"kubernetes.io/projected/2a0a23e6-8a48-4825-b2d7-e6bfd31fbe1f-kube-api-access-9p28c\") pod \"payload-processing-5b54b7fbb7-jlvzm\" (UID: \"2a0a23e6-8a48-4825-b2d7-e6bfd31fbe1f\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-jlvzm" Sep 06 12:27:15.031903 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:15.031879 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33750f3b-84a4-4935-a768-e2f5bda50eb3-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-t9grp\" (UID: \"33750f3b-84a4-4935-a768-e2f5bda50eb3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-t9grp" Sep 06 12:27:15.032036 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:15.031926 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f24br\" (UniqueName: \"kubernetes.io/projected/33750f3b-84a4-4935-a768-e2f5bda50eb3-kube-api-access-f24br\") pod \"payload-pre-processing-59fbc47bf8-t9grp\" (UID: \"33750f3b-84a4-4935-a768-e2f5bda50eb3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-t9grp" Sep 06 12:27:15.032564 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:15.032546 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33750f3b-84a4-4935-a768-e2f5bda50eb3-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-t9grp\" (UID: \"33750f3b-84a4-4935-a768-e2f5bda50eb3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-t9grp" Sep 06 12:27:15.041207 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:15.041185 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f24br\" (UniqueName: \"kubernetes.io/projected/33750f3b-84a4-4935-a768-e2f5bda50eb3-kube-api-access-f24br\") pod \"payload-pre-processing-59fbc47bf8-t9grp\" (UID: \"33750f3b-84a4-4935-a768-e2f5bda50eb3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-t9grp" Sep 06 12:27:15.084085 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:15.084061 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-t9grp" Sep 06 12:27:15.134946 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:15.134917 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-jlvzm" Sep 06 12:27:15.353764 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:15.353721 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-t9grp"] Sep 06 12:27:15.355874 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:27:15.355850 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33750f3b_84a4_4935_a768_e2f5bda50eb3.slice/crio-0c1cd8d472a011241ebd02211616b8034643287e1df4ba714ab3f72e68cab5b0 WatchSource:0}: Error finding container 0c1cd8d472a011241ebd02211616b8034643287e1df4ba714ab3f72e68cab5b0: Status 404 returned error can't find the container with id 0c1cd8d472a011241ebd02211616b8034643287e1df4ba714ab3f72e68cab5b0 Sep 06 12:27:15.411181 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:15.411157 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-jlvzm"] Sep 06 12:27:15.412778 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:27:15.412742 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0a23e6_8a48_4825_b2d7_e6bfd31fbe1f.slice/crio-77ce3e2c9f9fe699a40950158573198fa5227ca05c09219526488b59cb9be818 WatchSource:0}: Error finding container 77ce3e2c9f9fe699a40950158573198fa5227ca05c09219526488b59cb9be818: Status 404 returned error can't find the container with id 77ce3e2c9f9fe699a40950158573198fa5227ca05c09219526488b59cb9be818 Sep 06 12:27:15.432615 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:15.432588 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-jlvzm" event={"ID":"2a0a23e6-8a48-4825-b2d7-e6bfd31fbe1f","Type":"ContainerStarted","Data":"77ce3e2c9f9fe699a40950158573198fa5227ca05c09219526488b59cb9be818"} Sep 06 12:27:15.434219 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:15.434186 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-t9grp" event={"ID":"33750f3b-84a4-4935-a768-e2f5bda50eb3","Type":"ContainerStarted","Data":"0c1cd8d472a011241ebd02211616b8034643287e1df4ba714ab3f72e68cab5b0"} Sep 06 12:27:15.435614 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:15.435562 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b4cdc449f-bdh4x" event={"ID":"86425b02-9d6f-4d6b-b5e4-f5ade4648b78","Type":"ContainerStarted","Data":"4d5de4915e1f1072e11ed5ef2f21d0f11abc852f2971ac2ef4de9c7dda73ee13"} Sep 06 12:27:17.449301 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:17.449270 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b4cdc449f-bdh4x" event={"ID":"86425b02-9d6f-4d6b-b5e4-f5ade4648b78","Type":"ContainerStarted","Data":"924665ce853e2e35df52811355c34f5741042da9485041c4f18af17322dd41dc"} Sep 06 12:27:17.449708 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:17.449338 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5b4cdc449f-bdh4x" Sep 06 12:27:17.467204 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:17.467155 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5b4cdc449f-bdh4x" podStartSLOduration=1.705078504 podStartE2EDuration="3.467135945s" totalImagesPullingTime="1.762057441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:27:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:27:17.465423767 +0000 UTC m=+553.321181682" watchObservedRunningTime="2026-09-06 12:27:17.467135945 +0000 UTC m=+553.322893831" Sep 06 12:27:18.455846 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:18.455815 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-jlvzm" event={"ID":"2a0a23e6-8a48-4825-b2d7-e6bfd31fbe1f","Type":"ContainerStarted","Data":"65e7fdd0eca3979a9e423d93d176922f110a080f7fc674648817b1c1db0317fd"} Sep 06 12:27:18.456300 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:18.455973 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-jlvzm" Sep 06 12:27:18.457608 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:18.457588 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-t9grp" event={"ID":"33750f3b-84a4-4935-a768-e2f5bda50eb3","Type":"ContainerStarted","Data":"fd15642f5094866581668c7c260e83ec5535c0faa0291c753f0ad907409df2e2"} Sep 06 12:27:18.457778 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:18.457744 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-t9grp" Sep 06 12:27:18.471891 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:18.471835 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-jlvzm" podStartSLOduration=2.507348466 podStartE2EDuration="4.471820857s" totalImagesPullingTime="1.964472391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:27:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:27:18.46998139 +0000 UTC m=+554.325739280" watchObservedRunningTime="2026-09-06 12:27:18.471820857 +0000 UTC m=+554.327578743" Sep 06 12:27:18.484520 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:18.484484 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-t9grp" podStartSLOduration=2.469369721 podStartE2EDuration="4.484470913s" totalImagesPullingTime="2.015101192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:27:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:27:18.483341176 +0000 UTC m=+554.339099097" watchObservedRunningTime="2026-09-06 12:27:18.484470913 +0000 UTC m=+554.340228800" Sep 06 12:27:23.463005 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:23.462975 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5b4cdc449f-bdh4x" Sep 06 12:27:29.464491 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:29.464457 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-t9grp" Sep 06 12:27:29.464975 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:29.464804 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-jlvzm" Sep 06 12:27:38.508931 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:38.508899 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82"] Sep 06 12:27:38.512649 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:38.512627 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82" Sep 06 12:27:38.515938 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:38.515911 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 06 12:27:38.516088 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:38.515961 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-rs9bv\"" Sep 06 12:27:38.516088 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:38.516000 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 06 12:27:38.516088 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:38.515963 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 06 12:27:38.520319 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:38.520290 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82"] Sep 06 12:27:38.629748 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:38.629717 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ce6657f-df41-4452-8428-1e8b143d9a8b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82\" (UID: \"7ce6657f-df41-4452-8428-1e8b143d9a8b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82" Sep 06 12:27:38.629942 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:38.629788 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ce6657f-df41-4452-8428-1e8b143d9a8b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82\" (UID: \"7ce6657f-df41-4452-8428-1e8b143d9a8b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82" Sep 06 12:27:38.629942 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:38.629805 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ce6657f-df41-4452-8428-1e8b143d9a8b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82\" (UID: \"7ce6657f-df41-4452-8428-1e8b143d9a8b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82" Sep 06 12:27:38.629942 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:38.629836 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ce6657f-df41-4452-8428-1e8b143d9a8b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82\" (UID: \"7ce6657f-df41-4452-8428-1e8b143d9a8b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82" Sep 06 12:27:38.629942 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:38.629878 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ce6657f-df41-4452-8428-1e8b143d9a8b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82\" (UID: \"7ce6657f-df41-4452-8428-1e8b143d9a8b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82" Sep 06 12:27:38.629942 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:38.629905 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msswd\" (UniqueName: \"kubernetes.io/projected/7ce6657f-df41-4452-8428-1e8b143d9a8b-kube-api-access-msswd\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82\" (UID: \"7ce6657f-df41-4452-8428-1e8b143d9a8b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82" Sep 06 12:27:38.730274 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:38.730249 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ce6657f-df41-4452-8428-1e8b143d9a8b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82\" (UID: \"7ce6657f-df41-4452-8428-1e8b143d9a8b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82" Sep 06 12:27:38.730424 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:38.730278 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ce6657f-df41-4452-8428-1e8b143d9a8b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82\" (UID: \"7ce6657f-df41-4452-8428-1e8b143d9a8b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82" Sep 06 12:27:38.730424 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:38.730298 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ce6657f-df41-4452-8428-1e8b143d9a8b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82\" (UID: \"7ce6657f-df41-4452-8428-1e8b143d9a8b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82" Sep 06 12:27:38.730424 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:38.730340 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ce6657f-df41-4452-8428-1e8b143d9a8b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82\" (UID: \"7ce6657f-df41-4452-8428-1e8b143d9a8b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82" Sep 06 12:27:38.730424 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:38.730359 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msswd\" (UniqueName: \"kubernetes.io/projected/7ce6657f-df41-4452-8428-1e8b143d9a8b-kube-api-access-msswd\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82\" (UID: \"7ce6657f-df41-4452-8428-1e8b143d9a8b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82" Sep 06 12:27:38.730424 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:38.730400 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ce6657f-df41-4452-8428-1e8b143d9a8b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82\" (UID: \"7ce6657f-df41-4452-8428-1e8b143d9a8b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82" Sep 06 12:27:38.730727 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:38.730700 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ce6657f-df41-4452-8428-1e8b143d9a8b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82\" (UID: \"7ce6657f-df41-4452-8428-1e8b143d9a8b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82" Sep 06 12:27:38.730841 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:38.730821 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ce6657f-df41-4452-8428-1e8b143d9a8b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82\" (UID: \"7ce6657f-df41-4452-8428-1e8b143d9a8b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82" Sep 06 12:27:38.730883 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:38.730861 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ce6657f-df41-4452-8428-1e8b143d9a8b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82\" (UID: \"7ce6657f-df41-4452-8428-1e8b143d9a8b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82" Sep 06 12:27:38.732860 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:38.732840 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ce6657f-df41-4452-8428-1e8b143d9a8b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82\" (UID: \"7ce6657f-df41-4452-8428-1e8b143d9a8b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82" Sep 06 12:27:38.733053 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:38.733038 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ce6657f-df41-4452-8428-1e8b143d9a8b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82\" (UID: \"7ce6657f-df41-4452-8428-1e8b143d9a8b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82" Sep 06 12:27:38.737694 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:38.737672 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msswd\" (UniqueName: \"kubernetes.io/projected/7ce6657f-df41-4452-8428-1e8b143d9a8b-kube-api-access-msswd\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82\" (UID: \"7ce6657f-df41-4452-8428-1e8b143d9a8b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82" Sep 06 12:27:38.825625 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:38.825568 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82" Sep 06 12:27:39.102467 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:39.102385 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82"] Sep 06 12:27:39.105004 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:27:39.104973 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ce6657f_df41_4452_8428_1e8b143d9a8b.slice/crio-0431825bffca6d9cdd7375d9d18569c1ad8b6d68e0a9641aa28b764f1c545c48 WatchSource:0}: Error finding container 0431825bffca6d9cdd7375d9d18569c1ad8b6d68e0a9641aa28b764f1c545c48: Status 404 returned error can't find the container with id 0431825bffca6d9cdd7375d9d18569c1ad8b6d68e0a9641aa28b764f1c545c48 Sep 06 12:27:39.568541 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:39.568502 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82" event={"ID":"7ce6657f-df41-4452-8428-1e8b143d9a8b","Type":"ContainerStarted","Data":"0431825bffca6d9cdd7375d9d18569c1ad8b6d68e0a9641aa28b764f1c545c48"} Sep 06 12:27:40.574512 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:40.574438 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82" event={"ID":"7ce6657f-df41-4452-8428-1e8b143d9a8b","Type":"ContainerStarted","Data":"4c571d3b68cd6d11220bb582d870522d48dcf841b1302bfb699c2b2ad522120e"} Sep 06 12:27:40.591867 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:40.591824 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82" podStartSLOduration=1.5209506350000002 podStartE2EDuration="2.591808837s" totalImagesPullingTime="1.070858202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:27:40.589535322 +0000 UTC m=+576.445293243" watchObservedRunningTime="2026-09-06 12:27:40.591808837 +0000 UTC m=+576.447566725" Sep 06 12:27:48.826330 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:48.826295 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82" Sep 06 12:27:48.826840 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:48.826350 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82" Sep 06 12:27:48.839004 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:48.838981 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82" Sep 06 12:27:49.631735 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:27:49.631709 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82" Sep 06 12:28:04.591447 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:28:04.591412 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98vz4_4103c09f-dd43-419d-92e5-2374af5b5e01/ovn-acl-logging/0.log" Sep 06 12:28:04.591842 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:28:04.591555 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98vz4_4103c09f-dd43-419d-92e5-2374af5b5e01/ovn-acl-logging/0.log" Sep 06 12:28:54.904669 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:28:54.904639 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-wptjv"] Sep 06 12:28:54.907039 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:28:54.907024 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wptjv" Sep 06 12:28:54.909161 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:28:54.909137 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 06 12:28:54.916720 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:28:54.916698 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-wptjv"] Sep 06 12:28:55.051613 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:28:55.051578 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b41e00d7-fdb8-4a5e-850f-e9492a62f085-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wptjv\" (UID: \"b41e00d7-fdb8-4a5e-850f-e9492a62f085\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wptjv" Sep 06 12:28:55.051789 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:28:55.051619 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b41e00d7-fdb8-4a5e-850f-e9492a62f085-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wptjv\" (UID: \"b41e00d7-fdb8-4a5e-850f-e9492a62f085\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wptjv" Sep 06 12:28:55.051789 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:28:55.051655 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b41e00d7-fdb8-4a5e-850f-e9492a62f085-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wptjv\" (UID: \"b41e00d7-fdb8-4a5e-850f-e9492a62f085\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wptjv" Sep 06 12:28:55.051789 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:28:55.051700 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zkrm\" (UniqueName: \"kubernetes.io/projected/b41e00d7-fdb8-4a5e-850f-e9492a62f085-kube-api-access-2zkrm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wptjv\" (UID: \"b41e00d7-fdb8-4a5e-850f-e9492a62f085\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wptjv" Sep 06 12:28:55.051915 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:28:55.051806 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b41e00d7-fdb8-4a5e-850f-e9492a62f085-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wptjv\" (UID: \"b41e00d7-fdb8-4a5e-850f-e9492a62f085\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wptjv" Sep 06 12:28:55.051915 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:28:55.051832 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b41e00d7-fdb8-4a5e-850f-e9492a62f085-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wptjv\" (UID: \"b41e00d7-fdb8-4a5e-850f-e9492a62f085\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wptjv" Sep 06 12:28:55.152447 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:28:55.152419 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b41e00d7-fdb8-4a5e-850f-e9492a62f085-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wptjv\" (UID: \"b41e00d7-fdb8-4a5e-850f-e9492a62f085\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wptjv" Sep 06 12:28:55.152447 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:28:55.152447 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b41e00d7-fdb8-4a5e-850f-e9492a62f085-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wptjv\" (UID: \"b41e00d7-fdb8-4a5e-850f-e9492a62f085\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wptjv" Sep 06 12:28:55.152640 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:28:55.152464 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b41e00d7-fdb8-4a5e-850f-e9492a62f085-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wptjv\" (UID: \"b41e00d7-fdb8-4a5e-850f-e9492a62f085\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wptjv" Sep 06 12:28:55.152640 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:28:55.152485 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b41e00d7-fdb8-4a5e-850f-e9492a62f085-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wptjv\" (UID: \"b41e00d7-fdb8-4a5e-850f-e9492a62f085\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wptjv" Sep 06 12:28:55.152640 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:28:55.152531 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b41e00d7-fdb8-4a5e-850f-e9492a62f085-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wptjv\" (UID: \"b41e00d7-fdb8-4a5e-850f-e9492a62f085\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wptjv" Sep 06 12:28:55.152640 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:28:55.152574 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zkrm\" (UniqueName: \"kubernetes.io/projected/b41e00d7-fdb8-4a5e-850f-e9492a62f085-kube-api-access-2zkrm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wptjv\" (UID: \"b41e00d7-fdb8-4a5e-850f-e9492a62f085\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wptjv" Sep 06 12:28:55.152864 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:28:55.152843 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b41e00d7-fdb8-4a5e-850f-e9492a62f085-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wptjv\" (UID: \"b41e00d7-fdb8-4a5e-850f-e9492a62f085\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wptjv" Sep 06 12:28:55.152962 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:28:55.152941 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b41e00d7-fdb8-4a5e-850f-e9492a62f085-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wptjv\" (UID: \"b41e00d7-fdb8-4a5e-850f-e9492a62f085\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wptjv" Sep 06 12:28:55.153010 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:28:55.152979 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b41e00d7-fdb8-4a5e-850f-e9492a62f085-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wptjv\" (UID: \"b41e00d7-fdb8-4a5e-850f-e9492a62f085\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wptjv" Sep 06 12:28:55.155005 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:28:55.154957 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b41e00d7-fdb8-4a5e-850f-e9492a62f085-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wptjv\" (UID: \"b41e00d7-fdb8-4a5e-850f-e9492a62f085\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wptjv" Sep 06 12:28:55.155075 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:28:55.155059 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b41e00d7-fdb8-4a5e-850f-e9492a62f085-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wptjv\" (UID: \"b41e00d7-fdb8-4a5e-850f-e9492a62f085\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wptjv" Sep 06 12:28:55.159564 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:28:55.159542 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zkrm\" (UniqueName: \"kubernetes.io/projected/b41e00d7-fdb8-4a5e-850f-e9492a62f085-kube-api-access-2zkrm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wptjv\" (UID: \"b41e00d7-fdb8-4a5e-850f-e9492a62f085\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wptjv" Sep 06 12:28:55.217493 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:28:55.217471 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wptjv" Sep 06 12:28:55.485886 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:28:55.485825 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-wptjv"] Sep 06 12:28:55.487217 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:28:55.487189 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb41e00d7_fdb8_4a5e_850f_e9492a62f085.slice/crio-4197b6ac16342dd17aea0d0a937cd094265a6bf88df90c1dea3ae653c176aae1 WatchSource:0}: Error finding container 4197b6ac16342dd17aea0d0a937cd094265a6bf88df90c1dea3ae653c176aae1: Status 404 returned error can't find the container with id 4197b6ac16342dd17aea0d0a937cd094265a6bf88df90c1dea3ae653c176aae1 Sep 06 12:28:55.489028 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:28:55.489009 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 06 12:28:55.949988 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:28:55.949955 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wptjv" event={"ID":"b41e00d7-fdb8-4a5e-850f-e9492a62f085","Type":"ContainerStarted","Data":"190bfcc0d329e403c7b7b564d3a25ed57757892206ddb35d2c25ffe3fa6f9906"} Sep 06 12:28:55.950354 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:28:55.950008 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wptjv" event={"ID":"b41e00d7-fdb8-4a5e-850f-e9492a62f085","Type":"ContainerStarted","Data":"4197b6ac16342dd17aea0d0a937cd094265a6bf88df90c1dea3ae653c176aae1"} Sep 06 12:28:55.968905 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:28:55.968854 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wptjv" podStartSLOduration=1.806079601 podStartE2EDuration="1.968837785s" totalImagesPullingTime="162.758184ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:28:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:28:55.967809385 +0000 UTC m=+651.823567306" watchObservedRunningTime="2026-09-06 12:28:55.968837785 +0000 UTC m=+651.824595672" Sep 06 12:29:05.217743 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:29:05.217714 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wptjv" Sep 06 12:29:05.218140 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:29:05.218009 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wptjv" Sep 06 12:29:05.230571 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:29:05.230545 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wptjv" Sep 06 12:29:06.022539 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:29:06.022507 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wptjv" Sep 06 12:30:18.606963 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:18.606928 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8"] Sep 06 12:30:18.610459 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:18.610437 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8" Sep 06 12:30:18.612596 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:18.612575 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-36bcab-serving-cert\"" Sep 06 12:30:18.623022 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:18.622994 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8"] Sep 06 12:30:18.651198 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:18.651154 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a12d7af-a130-4f4e-9796-86dc6d65fb3f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8\" (UID: \"2a12d7af-a130-4f4e-9796-86dc6d65fb3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8" Sep 06 12:30:18.651455 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:18.651229 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52bmb\" (UniqueName: \"kubernetes.io/projected/2a12d7af-a130-4f4e-9796-86dc6d65fb3f-kube-api-access-52bmb\") pod \"maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8\" (UID: \"2a12d7af-a130-4f4e-9796-86dc6d65fb3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8" Sep 06 12:30:18.651455 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:18.651296 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a12d7af-a130-4f4e-9796-86dc6d65fb3f-maas-api-tls\") pod \"maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8\" (UID: \"2a12d7af-a130-4f4e-9796-86dc6d65fb3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8" Sep 06 12:30:18.651455 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:18.651319 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a12d7af-a130-4f4e-9796-86dc6d65fb3f-metrics-certs\") pod \"maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8\" (UID: \"2a12d7af-a130-4f4e-9796-86dc6d65fb3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8" Sep 06 12:30:18.752303 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:18.752273 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a12d7af-a130-4f4e-9796-86dc6d65fb3f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8\" (UID: \"2a12d7af-a130-4f4e-9796-86dc6d65fb3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8" Sep 06 12:30:18.752432 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:18.752311 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52bmb\" (UniqueName: \"kubernetes.io/projected/2a12d7af-a130-4f4e-9796-86dc6d65fb3f-kube-api-access-52bmb\") pod \"maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8\" (UID: \"2a12d7af-a130-4f4e-9796-86dc6d65fb3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8" Sep 06 12:30:18.752432 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:18.752335 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a12d7af-a130-4f4e-9796-86dc6d65fb3f-maas-api-tls\") pod \"maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8\" (UID: \"2a12d7af-a130-4f4e-9796-86dc6d65fb3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8" Sep 06 12:30:18.752432 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:18.752349 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a12d7af-a130-4f4e-9796-86dc6d65fb3f-metrics-certs\") pod \"maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8\" (UID: \"2a12d7af-a130-4f4e-9796-86dc6d65fb3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8" Sep 06 12:30:18.752581 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:30:18.752475 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-36bcab-serving-cert: secret "maas-api-e2e-inf-a-36bcab-serving-cert" not found Sep 06 12:30:18.752581 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:30:18.752541 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a12d7af-a130-4f4e-9796-86dc6d65fb3f-maas-api-tls podName:2a12d7af-a130-4f4e-9796-86dc6d65fb3f nodeName:}" failed. No retries permitted until 2026-09-06 12:30:19.252519709 +0000 UTC m=+735.108277592 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2a12d7af-a130-4f4e-9796-86dc6d65fb3f-maas-api-tls") pod "maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8" (UID: "2a12d7af-a130-4f4e-9796-86dc6d65fb3f") : secret "maas-api-e2e-inf-a-36bcab-serving-cert" not found Sep 06 12:30:18.753070 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:18.753048 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a12d7af-a130-4f4e-9796-86dc6d65fb3f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8\" (UID: \"2a12d7af-a130-4f4e-9796-86dc6d65fb3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8" Sep 06 12:30:18.754965 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:18.754947 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a12d7af-a130-4f4e-9796-86dc6d65fb3f-metrics-certs\") pod \"maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8\" (UID: \"2a12d7af-a130-4f4e-9796-86dc6d65fb3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8" Sep 06 12:30:18.768144 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:18.768121 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52bmb\" (UniqueName: \"kubernetes.io/projected/2a12d7af-a130-4f4e-9796-86dc6d65fb3f-kube-api-access-52bmb\") pod \"maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8\" (UID: \"2a12d7af-a130-4f4e-9796-86dc6d65fb3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8" Sep 06 12:30:19.256540 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:19.256504 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a12d7af-a130-4f4e-9796-86dc6d65fb3f-maas-api-tls\") pod \"maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8\" (UID: \"2a12d7af-a130-4f4e-9796-86dc6d65fb3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8" Sep 06 12:30:19.259089 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:19.259066 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a12d7af-a130-4f4e-9796-86dc6d65fb3f-maas-api-tls\") pod \"maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8\" (UID: \"2a12d7af-a130-4f4e-9796-86dc6d65fb3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8" Sep 06 12:30:19.271772 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:19.269409 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-36bcab-6447494fbc-mtb22"] Sep 06 12:30:19.275083 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:19.275056 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-36bcab-6447494fbc-mtb22" Sep 06 12:30:19.277803 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:19.277781 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-36bcab-dockercfg-7lp9f\"" Sep 06 12:30:19.277910 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:19.277783 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-36bcab\"" Sep 06 12:30:19.286159 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:19.286137 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-36bcab-6447494fbc-mtb22"] Sep 06 12:30:19.298076 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:19.298048 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-36bcab-6bd6d9df98-dz7qf"] Sep 06 12:30:19.301637 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:19.301616 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-36bcab-6bd6d9df98-dz7qf" Sep 06 12:30:19.311124 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:19.311100 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-36bcab-6bd6d9df98-dz7qf"] Sep 06 12:30:19.357469 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:19.357354 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdv9v\" (UniqueName: \"kubernetes.io/projected/ff5bc9b8-9277-4869-8846-0313557b5949-kube-api-access-bdv9v\") pod \"payload-pre-processing-e2e-inf-a-36bcab-6bd6d9df98-dz7qf\" (UID: \"ff5bc9b8-9277-4869-8846-0313557b5949\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-36bcab-6bd6d9df98-dz7qf" Sep 06 12:30:19.357606 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:19.357544 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff5bc9b8-9277-4869-8846-0313557b5949-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-36bcab-6bd6d9df98-dz7qf\" (UID: \"ff5bc9b8-9277-4869-8846-0313557b5949\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-36bcab-6bd6d9df98-dz7qf" Sep 06 12:30:19.357665 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:19.357623 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpwpd\" (UniqueName: \"kubernetes.io/projected/316d29e5-cab3-4085-a3d7-0604766545ed-kube-api-access-jpwpd\") pod \"payload-processing-e2e-inf-a-36bcab-6447494fbc-mtb22\" (UID: \"316d29e5-cab3-4085-a3d7-0604766545ed\") " pod="openshift-ingress/payload-processing-e2e-inf-a-36bcab-6447494fbc-mtb22" Sep 06 12:30:19.357726 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:19.357702 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/316d29e5-cab3-4085-a3d7-0604766545ed-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-36bcab-6447494fbc-mtb22\" (UID: \"316d29e5-cab3-4085-a3d7-0604766545ed\") " pod="openshift-ingress/payload-processing-e2e-inf-a-36bcab-6447494fbc-mtb22" Sep 06 12:30:19.459018 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:19.458992 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/316d29e5-cab3-4085-a3d7-0604766545ed-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-36bcab-6447494fbc-mtb22\" (UID: \"316d29e5-cab3-4085-a3d7-0604766545ed\") " pod="openshift-ingress/payload-processing-e2e-inf-a-36bcab-6447494fbc-mtb22" Sep 06 12:30:19.459161 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:19.459048 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdv9v\" (UniqueName: \"kubernetes.io/projected/ff5bc9b8-9277-4869-8846-0313557b5949-kube-api-access-bdv9v\") pod \"payload-pre-processing-e2e-inf-a-36bcab-6bd6d9df98-dz7qf\" (UID: \"ff5bc9b8-9277-4869-8846-0313557b5949\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-36bcab-6bd6d9df98-dz7qf" Sep 06 12:30:19.459161 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:19.459078 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff5bc9b8-9277-4869-8846-0313557b5949-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-36bcab-6bd6d9df98-dz7qf\" (UID: \"ff5bc9b8-9277-4869-8846-0313557b5949\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-36bcab-6bd6d9df98-dz7qf" Sep 06 12:30:19.459161 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:19.459107 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpwpd\" (UniqueName: \"kubernetes.io/projected/316d29e5-cab3-4085-a3d7-0604766545ed-kube-api-access-jpwpd\") pod \"payload-processing-e2e-inf-a-36bcab-6447494fbc-mtb22\" (UID: \"316d29e5-cab3-4085-a3d7-0604766545ed\") " pod="openshift-ingress/payload-processing-e2e-inf-a-36bcab-6447494fbc-mtb22" Sep 06 12:30:19.459944 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:19.459922 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/316d29e5-cab3-4085-a3d7-0604766545ed-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-36bcab-6447494fbc-mtb22\" (UID: \"316d29e5-cab3-4085-a3d7-0604766545ed\") " pod="openshift-ingress/payload-processing-e2e-inf-a-36bcab-6447494fbc-mtb22" Sep 06 12:30:19.460005 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:19.459922 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff5bc9b8-9277-4869-8846-0313557b5949-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-36bcab-6bd6d9df98-dz7qf\" (UID: \"ff5bc9b8-9277-4869-8846-0313557b5949\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-36bcab-6bd6d9df98-dz7qf" Sep 06 12:30:19.468561 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:19.468541 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdv9v\" (UniqueName: \"kubernetes.io/projected/ff5bc9b8-9277-4869-8846-0313557b5949-kube-api-access-bdv9v\") pod \"payload-pre-processing-e2e-inf-a-36bcab-6bd6d9df98-dz7qf\" (UID: \"ff5bc9b8-9277-4869-8846-0313557b5949\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-36bcab-6bd6d9df98-dz7qf" Sep 06 12:30:19.468772 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:19.468738 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpwpd\" (UniqueName: \"kubernetes.io/projected/316d29e5-cab3-4085-a3d7-0604766545ed-kube-api-access-jpwpd\") pod \"payload-processing-e2e-inf-a-36bcab-6447494fbc-mtb22\" (UID: \"316d29e5-cab3-4085-a3d7-0604766545ed\") " pod="openshift-ingress/payload-processing-e2e-inf-a-36bcab-6447494fbc-mtb22" Sep 06 12:30:19.522492 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:19.522435 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8" Sep 06 12:30:19.587966 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:19.587743 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-36bcab-6447494fbc-mtb22" Sep 06 12:30:19.614319 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:19.614227 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-36bcab-6bd6d9df98-dz7qf" Sep 06 12:30:19.792160 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:19.792133 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8"] Sep 06 12:30:19.793965 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:30:19.793934 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a12d7af_a130_4f4e_9796_86dc6d65fb3f.slice/crio-7455f13656573ef4a5ae939808a92536a17080e32ee3ac6a2d8868214d9c2e7d WatchSource:0}: Error finding container 7455f13656573ef4a5ae939808a92536a17080e32ee3ac6a2d8868214d9c2e7d: Status 404 returned error can't find the container with id 7455f13656573ef4a5ae939808a92536a17080e32ee3ac6a2d8868214d9c2e7d Sep 06 12:30:19.864730 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:19.864705 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-36bcab-6447494fbc-mtb22"] Sep 06 12:30:19.866543 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:30:19.866507 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod316d29e5_cab3_4085_a3d7_0604766545ed.slice/crio-e16d324bbc8595b507e13fc5e0cb14ca2bd292bb9bfdb04d106a7635f5e229fc WatchSource:0}: Error finding container e16d324bbc8595b507e13fc5e0cb14ca2bd292bb9bfdb04d106a7635f5e229fc: Status 404 returned error can't find the container with id e16d324bbc8595b507e13fc5e0cb14ca2bd292bb9bfdb04d106a7635f5e229fc Sep 06 12:30:19.892908 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:19.892868 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-36bcab-6bd6d9df98-dz7qf"] Sep 06 12:30:19.895394 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:30:19.895373 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff5bc9b8_9277_4869_8846_0313557b5949.slice/crio-4dce191200259be7ffd4c2872b9dc052bd81510a3627d97bd74684ff4bce0701 WatchSource:0}: Error finding container 4dce191200259be7ffd4c2872b9dc052bd81510a3627d97bd74684ff4bce0701: Status 404 returned error can't find the container with id 4dce191200259be7ffd4c2872b9dc052bd81510a3627d97bd74684ff4bce0701 Sep 06 12:30:20.394705 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:20.394674 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-36bcab-6447494fbc-mtb22" event={"ID":"316d29e5-cab3-4085-a3d7-0604766545ed","Type":"ContainerStarted","Data":"f220cb256b1c2addc7d13d1333b2c5f0fee97c25f27fe727dd5c584d03a45c15"} Sep 06 12:30:20.394864 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:20.394739 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-36bcab-6447494fbc-mtb22" event={"ID":"316d29e5-cab3-4085-a3d7-0604766545ed","Type":"ContainerStarted","Data":"e16d324bbc8595b507e13fc5e0cb14ca2bd292bb9bfdb04d106a7635f5e229fc"} Sep 06 12:30:20.394864 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:20.394785 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-36bcab-6447494fbc-mtb22" Sep 06 12:30:20.396489 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:20.396467 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8" event={"ID":"2a12d7af-a130-4f4e-9796-86dc6d65fb3f","Type":"ContainerStarted","Data":"1922a1e21ad9615b1d572e64caf62a3390446ae700b5e9bff1c33563ecffd7e4"} Sep 06 12:30:20.396603 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:20.396513 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8" event={"ID":"2a12d7af-a130-4f4e-9796-86dc6d65fb3f","Type":"ContainerStarted","Data":"7455f13656573ef4a5ae939808a92536a17080e32ee3ac6a2d8868214d9c2e7d"} Sep 06 12:30:20.396603 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:20.396541 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8" Sep 06 12:30:20.397981 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:20.397957 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-36bcab-6bd6d9df98-dz7qf" event={"ID":"ff5bc9b8-9277-4869-8846-0313557b5949","Type":"ContainerStarted","Data":"f739537f3a38c0fe8c4a3122574bc18286ae107b8dbe169f618c458c52a6f219"} Sep 06 12:30:20.398091 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:20.397997 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-36bcab-6bd6d9df98-dz7qf" event={"ID":"ff5bc9b8-9277-4869-8846-0313557b5949","Type":"ContainerStarted","Data":"4dce191200259be7ffd4c2872b9dc052bd81510a3627d97bd74684ff4bce0701"} Sep 06 12:30:20.398091 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:20.398072 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-36bcab-6bd6d9df98-dz7qf" Sep 06 12:30:20.410710 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:20.410662 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-36bcab-6447494fbc-mtb22" podStartSLOduration=1.4106427049999999 podStartE2EDuration="1.410642705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:30:20.409863082 +0000 UTC m=+736.265620994" watchObservedRunningTime="2026-09-06 12:30:20.410642705 +0000 UTC m=+736.266400595" Sep 06 12:30:20.425294 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:20.425248 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-36bcab-6bd6d9df98-dz7qf" podStartSLOduration=1.425234396 podStartE2EDuration="1.425234396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:30:20.423276599 +0000 UTC m=+736.279034510" watchObservedRunningTime="2026-09-06 12:30:20.425234396 +0000 UTC m=+736.280992273" Sep 06 12:30:20.439503 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:20.439467 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8" podStartSLOduration=1.953552843 podStartE2EDuration="2.439454271s" totalImagesPullingTime="485.901428ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:30:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:30:20.437571434 +0000 UTC m=+736.293329313" watchObservedRunningTime="2026-09-06 12:30:20.439454271 +0000 UTC m=+736.295212157" Sep 06 12:30:26.162842 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:26.162809 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf"] Sep 06 12:30:26.168595 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:26.168562 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf" Sep 06 12:30:26.170747 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:26.170725 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8f9eca99-serving-cert\"" Sep 06 12:30:26.174674 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:26.174648 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf"] Sep 06 12:30:26.225422 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:26.225396 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e7df151-3df3-462d-af7b-d447d4c0d0c2-maas-api-tls\") pod \"maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf\" (UID: \"2e7df151-3df3-462d-af7b-d447d4c0d0c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf" Sep 06 12:30:26.225564 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:26.225455 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bgr9\" (UniqueName: \"kubernetes.io/projected/2e7df151-3df3-462d-af7b-d447d4c0d0c2-kube-api-access-5bgr9\") pod \"maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf\" (UID: \"2e7df151-3df3-462d-af7b-d447d4c0d0c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf" Sep 06 12:30:26.225564 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:26.225521 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e7df151-3df3-462d-af7b-d447d4c0d0c2-metrics-certs\") pod \"maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf\" (UID: \"2e7df151-3df3-462d-af7b-d447d4c0d0c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf" Sep 06 12:30:26.225678 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:26.225605 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e7df151-3df3-462d-af7b-d447d4c0d0c2-openshift-service-ca\") pod \"maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf\" (UID: \"2e7df151-3df3-462d-af7b-d447d4c0d0c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf" Sep 06 12:30:26.327066 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:26.327036 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bgr9\" (UniqueName: \"kubernetes.io/projected/2e7df151-3df3-462d-af7b-d447d4c0d0c2-kube-api-access-5bgr9\") pod \"maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf\" (UID: \"2e7df151-3df3-462d-af7b-d447d4c0d0c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf" Sep 06 12:30:26.327240 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:26.327088 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e7df151-3df3-462d-af7b-d447d4c0d0c2-metrics-certs\") pod \"maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf\" (UID: \"2e7df151-3df3-462d-af7b-d447d4c0d0c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf" Sep 06 12:30:26.327240 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:26.327132 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e7df151-3df3-462d-af7b-d447d4c0d0c2-openshift-service-ca\") pod \"maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf\" (UID: \"2e7df151-3df3-462d-af7b-d447d4c0d0c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf" Sep 06 12:30:26.327240 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:26.327181 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e7df151-3df3-462d-af7b-d447d4c0d0c2-maas-api-tls\") pod \"maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf\" (UID: \"2e7df151-3df3-462d-af7b-d447d4c0d0c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf" Sep 06 12:30:26.327773 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:26.327732 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e7df151-3df3-462d-af7b-d447d4c0d0c2-openshift-service-ca\") pod \"maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf\" (UID: \"2e7df151-3df3-462d-af7b-d447d4c0d0c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf" Sep 06 12:30:26.329725 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:26.329702 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e7df151-3df3-462d-af7b-d447d4c0d0c2-metrics-certs\") pod \"maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf\" (UID: \"2e7df151-3df3-462d-af7b-d447d4c0d0c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf" Sep 06 12:30:26.329893 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:26.329871 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e7df151-3df3-462d-af7b-d447d4c0d0c2-maas-api-tls\") pod \"maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf\" (UID: \"2e7df151-3df3-462d-af7b-d447d4c0d0c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf" Sep 06 12:30:26.334604 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:26.334583 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bgr9\" (UniqueName: \"kubernetes.io/projected/2e7df151-3df3-462d-af7b-d447d4c0d0c2-kube-api-access-5bgr9\") pod \"maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf\" (UID: \"2e7df151-3df3-462d-af7b-d447d4c0d0c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf" Sep 06 12:30:26.408415 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:26.408390 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8" Sep 06 12:30:26.482234 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:26.482156 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf" Sep 06 12:30:26.768307 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:26.768086 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf"] Sep 06 12:30:26.770676 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:30:26.770650 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e7df151_3df3_462d_af7b_d447d4c0d0c2.slice/crio-3e16dca68c3c5fe8a6e6a4ea7798fe794137f5bb6073dfc517b228460403585f WatchSource:0}: Error finding container 3e16dca68c3c5fe8a6e6a4ea7798fe794137f5bb6073dfc517b228460403585f: Status 404 returned error can't find the container with id 3e16dca68c3c5fe8a6e6a4ea7798fe794137f5bb6073dfc517b228460403585f Sep 06 12:30:26.777331 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:26.777295 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8f9eca99-58fc7d74-ftm2b"] Sep 06 12:30:26.782071 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:26.782055 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8f9eca99-58fc7d74-ftm2b" Sep 06 12:30:26.784480 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:26.784459 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8f9eca99-dockercfg-b4prh\"" Sep 06 12:30:26.784575 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:26.784463 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8f9eca99\"" Sep 06 12:30:26.789696 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:26.789673 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8f9eca99-58fc7d74-ftm2b"] Sep 06 12:30:26.932927 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:26.932856 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b145a0a-2cdd-4109-8a20-760575f03651-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8f9eca99-58fc7d74-ftm2b\" (UID: \"8b145a0a-2cdd-4109-8a20-760575f03651\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8f9eca99-58fc7d74-ftm2b" Sep 06 12:30:26.932927 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:26.932889 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqlkp\" (UniqueName: \"kubernetes.io/projected/8b145a0a-2cdd-4109-8a20-760575f03651-kube-api-access-tqlkp\") pod \"payload-pre-processing-e2e-ait-8f9eca99-58fc7d74-ftm2b\" (UID: \"8b145a0a-2cdd-4109-8a20-760575f03651\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8f9eca99-58fc7d74-ftm2b" Sep 06 12:30:27.033659 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:27.033580 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b145a0a-2cdd-4109-8a20-760575f03651-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8f9eca99-58fc7d74-ftm2b\" (UID: \"8b145a0a-2cdd-4109-8a20-760575f03651\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8f9eca99-58fc7d74-ftm2b" Sep 06 12:30:27.033659 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:27.033623 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqlkp\" (UniqueName: \"kubernetes.io/projected/8b145a0a-2cdd-4109-8a20-760575f03651-kube-api-access-tqlkp\") pod \"payload-pre-processing-e2e-ait-8f9eca99-58fc7d74-ftm2b\" (UID: \"8b145a0a-2cdd-4109-8a20-760575f03651\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8f9eca99-58fc7d74-ftm2b" Sep 06 12:30:27.034340 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:27.034320 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b145a0a-2cdd-4109-8a20-760575f03651-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8f9eca99-58fc7d74-ftm2b\" (UID: \"8b145a0a-2cdd-4109-8a20-760575f03651\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8f9eca99-58fc7d74-ftm2b" Sep 06 12:30:27.041034 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:27.041011 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqlkp\" (UniqueName: \"kubernetes.io/projected/8b145a0a-2cdd-4109-8a20-760575f03651-kube-api-access-tqlkp\") pod \"payload-pre-processing-e2e-ait-8f9eca99-58fc7d74-ftm2b\" (UID: \"8b145a0a-2cdd-4109-8a20-760575f03651\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8f9eca99-58fc7d74-ftm2b" Sep 06 12:30:27.095380 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:27.095360 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8f9eca99-58fc7d74-ftm2b" Sep 06 12:30:27.368522 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:27.368455 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8f9eca99-58fc7d74-ftm2b"] Sep 06 12:30:27.372552 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:30:27.372518 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b145a0a_2cdd_4109_8a20_760575f03651.slice/crio-672666d5a8e4d62c5118db14d0aff3873cf5d6c570535a6039bd354d6caf7f53 WatchSource:0}: Error finding container 672666d5a8e4d62c5118db14d0aff3873cf5d6c570535a6039bd354d6caf7f53: Status 404 returned error can't find the container with id 672666d5a8e4d62c5118db14d0aff3873cf5d6c570535a6039bd354d6caf7f53 Sep 06 12:30:27.434815 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:27.434785 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8f9eca99-58fc7d74-ftm2b" event={"ID":"8b145a0a-2cdd-4109-8a20-760575f03651","Type":"ContainerStarted","Data":"672666d5a8e4d62c5118db14d0aff3873cf5d6c570535a6039bd354d6caf7f53"} Sep 06 12:30:27.436405 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:27.436384 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf" event={"ID":"2e7df151-3df3-462d-af7b-d447d4c0d0c2","Type":"ContainerStarted","Data":"6ee9a68a066e2d647d06a5a6fe78121a04fd821295de5371910e32b73c670262"} Sep 06 12:30:27.436504 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:27.436425 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf" event={"ID":"2e7df151-3df3-462d-af7b-d447d4c0d0c2","Type":"ContainerStarted","Data":"3e16dca68c3c5fe8a6e6a4ea7798fe794137f5bb6073dfc517b228460403585f"} Sep 06 12:30:27.436555 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:27.436519 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf" Sep 06 12:30:27.454920 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:27.454872 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf" podStartSLOduration=1.171031674 podStartE2EDuration="1.454856878s" totalImagesPullingTime="283.825204ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:30:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:30:27.452402015 +0000 UTC m=+743.308159925" watchObservedRunningTime="2026-09-06 12:30:27.454856878 +0000 UTC m=+743.310614799" Sep 06 12:30:28.443205 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:28.443162 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8f9eca99-58fc7d74-ftm2b" event={"ID":"8b145a0a-2cdd-4109-8a20-760575f03651","Type":"ContainerStarted","Data":"1c3ede3cbc3832ca48d5b9de8f48a242983e2127d90d4ea77cae0820149ea43d"} Sep 06 12:30:28.443669 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:28.443387 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8f9eca99-58fc7d74-ftm2b" Sep 06 12:30:28.458861 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:28.458813 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8f9eca99-58fc7d74-ftm2b" podStartSLOduration=2.458796638 podStartE2EDuration="2.458796638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:30:28.45701198 +0000 UTC m=+744.312769859" watchObservedRunningTime="2026-09-06 12:30:28.458796638 +0000 UTC m=+744.314554525" Sep 06 12:30:28.809918 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:28.809840 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-0e6ab1-b887684c4-vjk27"] Sep 06 12:30:28.814208 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:28.814189 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e6ab1-b887684c4-vjk27" Sep 06 12:30:28.816319 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:28.816294 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-0e6ab1-serving-cert\"" Sep 06 12:30:28.822741 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:28.822716 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-0e6ab1-b887684c4-vjk27"] Sep 06 12:30:28.952462 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:28.952430 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a25f137-55e3-49d2-bfa3-4781e20b6bf0-maas-api-tls\") pod \"maas-api-e2e-autores-0e6ab1-b887684c4-vjk27\" (UID: \"6a25f137-55e3-49d2-bfa3-4781e20b6bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e6ab1-b887684c4-vjk27" Sep 06 12:30:28.952617 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:28.952497 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a25f137-55e3-49d2-bfa3-4781e20b6bf0-metrics-certs\") pod \"maas-api-e2e-autores-0e6ab1-b887684c4-vjk27\" (UID: \"6a25f137-55e3-49d2-bfa3-4781e20b6bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e6ab1-b887684c4-vjk27" Sep 06 12:30:28.952617 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:28.952519 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4kd9\" (UniqueName: \"kubernetes.io/projected/6a25f137-55e3-49d2-bfa3-4781e20b6bf0-kube-api-access-d4kd9\") pod \"maas-api-e2e-autores-0e6ab1-b887684c4-vjk27\" (UID: \"6a25f137-55e3-49d2-bfa3-4781e20b6bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e6ab1-b887684c4-vjk27" Sep 06 12:30:28.952617 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:28.952546 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a25f137-55e3-49d2-bfa3-4781e20b6bf0-openshift-service-ca\") pod \"maas-api-e2e-autores-0e6ab1-b887684c4-vjk27\" (UID: \"6a25f137-55e3-49d2-bfa3-4781e20b6bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e6ab1-b887684c4-vjk27" Sep 06 12:30:29.053733 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:29.053703 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a25f137-55e3-49d2-bfa3-4781e20b6bf0-metrics-certs\") pod \"maas-api-e2e-autores-0e6ab1-b887684c4-vjk27\" (UID: \"6a25f137-55e3-49d2-bfa3-4781e20b6bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e6ab1-b887684c4-vjk27" Sep 06 12:30:29.053882 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:29.053736 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4kd9\" (UniqueName: \"kubernetes.io/projected/6a25f137-55e3-49d2-bfa3-4781e20b6bf0-kube-api-access-d4kd9\") pod \"maas-api-e2e-autores-0e6ab1-b887684c4-vjk27\" (UID: \"6a25f137-55e3-49d2-bfa3-4781e20b6bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e6ab1-b887684c4-vjk27" Sep 06 12:30:29.053882 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:29.053787 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a25f137-55e3-49d2-bfa3-4781e20b6bf0-openshift-service-ca\") pod \"maas-api-e2e-autores-0e6ab1-b887684c4-vjk27\" (UID: \"6a25f137-55e3-49d2-bfa3-4781e20b6bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e6ab1-b887684c4-vjk27" Sep 06 12:30:29.053882 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:29.053872 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a25f137-55e3-49d2-bfa3-4781e20b6bf0-maas-api-tls\") pod \"maas-api-e2e-autores-0e6ab1-b887684c4-vjk27\" (UID: \"6a25f137-55e3-49d2-bfa3-4781e20b6bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e6ab1-b887684c4-vjk27" Sep 06 12:30:29.054628 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:29.054544 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a25f137-55e3-49d2-bfa3-4781e20b6bf0-openshift-service-ca\") pod \"maas-api-e2e-autores-0e6ab1-b887684c4-vjk27\" (UID: \"6a25f137-55e3-49d2-bfa3-4781e20b6bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e6ab1-b887684c4-vjk27" Sep 06 12:30:29.056457 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:29.056436 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a25f137-55e3-49d2-bfa3-4781e20b6bf0-metrics-certs\") pod \"maas-api-e2e-autores-0e6ab1-b887684c4-vjk27\" (UID: \"6a25f137-55e3-49d2-bfa3-4781e20b6bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e6ab1-b887684c4-vjk27" Sep 06 12:30:29.056580 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:29.056563 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a25f137-55e3-49d2-bfa3-4781e20b6bf0-maas-api-tls\") pod \"maas-api-e2e-autores-0e6ab1-b887684c4-vjk27\" (UID: \"6a25f137-55e3-49d2-bfa3-4781e20b6bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e6ab1-b887684c4-vjk27" Sep 06 12:30:29.062145 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:29.062097 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4kd9\" (UniqueName: \"kubernetes.io/projected/6a25f137-55e3-49d2-bfa3-4781e20b6bf0-kube-api-access-d4kd9\") pod \"maas-api-e2e-autores-0e6ab1-b887684c4-vjk27\" (UID: \"6a25f137-55e3-49d2-bfa3-4781e20b6bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e6ab1-b887684c4-vjk27" Sep 06 12:30:29.129044 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:29.129018 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e6ab1-b887684c4-vjk27" Sep 06 12:30:29.407350 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:29.407302 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-0e6ab1-b887684c4-vjk27"] Sep 06 12:30:29.409591 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:30:29.409564 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a25f137_55e3_49d2_bfa3_4781e20b6bf0.slice/crio-c89b3e78c7738c2cfdfde903cae545f510d4a40fc8d38320da417b548e39291d WatchSource:0}: Error finding container c89b3e78c7738c2cfdfde903cae545f510d4a40fc8d38320da417b548e39291d: Status 404 returned error can't find the container with id c89b3e78c7738c2cfdfde903cae545f510d4a40fc8d38320da417b548e39291d Sep 06 12:30:29.451043 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:29.451007 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e6ab1-b887684c4-vjk27" event={"ID":"6a25f137-55e3-49d2-bfa3-4781e20b6bf0","Type":"ContainerStarted","Data":"c89b3e78c7738c2cfdfde903cae545f510d4a40fc8d38320da417b548e39291d"} Sep 06 12:30:30.457786 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:30.457731 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e6ab1-b887684c4-vjk27" event={"ID":"6a25f137-55e3-49d2-bfa3-4781e20b6bf0","Type":"ContainerStarted","Data":"cb9cbde7ffb936fddd4f03701bd7e840532ed00fbdd8ff2750ac779a891af744"} Sep 06 12:30:30.474729 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:30.474674 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e6ab1-b887684c4-vjk27" podStartSLOduration=2.1582384 podStartE2EDuration="2.474653497s" totalImagesPullingTime="316.415097ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:30:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:30:30.472007024 +0000 UTC m=+746.327764936" watchObservedRunningTime="2026-09-06 12:30:30.474653497 +0000 UTC m=+746.330411385" Sep 06 12:30:31.405104 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:31.405064 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-36bcab-6bd6d9df98-dz7qf" Sep 06 12:30:31.405295 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:31.405159 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-36bcab-6447494fbc-mtb22" Sep 06 12:30:31.463469 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:31.463432 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e6ab1-b887684c4-vjk27" Sep 06 12:30:33.448639 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:33.448610 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf" Sep 06 12:30:37.474959 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:37.474918 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e6ab1-b887684c4-vjk27" Sep 06 12:30:39.454129 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:39.454102 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8f9eca99-58fc7d74-ftm2b" Sep 06 12:30:47.957519 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:47.957483 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh"] Sep 06 12:30:47.961585 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:47.961563 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh" Sep 06 12:30:47.963563 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:47.963535 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-58053a-serving-cert\"" Sep 06 12:30:47.972213 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:47.972189 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh"] Sep 06 12:30:48.124526 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:48.124498 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4f0223d-3b53-4b7d-ad1d-af0ae3d35738-metrics-certs\") pod \"maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh\" (UID: \"a4f0223d-3b53-4b7d-ad1d-af0ae3d35738\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh" Sep 06 12:30:48.124670 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:48.124563 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqn75\" (UniqueName: \"kubernetes.io/projected/a4f0223d-3b53-4b7d-ad1d-af0ae3d35738-kube-api-access-kqn75\") pod \"maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh\" (UID: \"a4f0223d-3b53-4b7d-ad1d-af0ae3d35738\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh" Sep 06 12:30:48.124709 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:48.124678 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4f0223d-3b53-4b7d-ad1d-af0ae3d35738-openshift-service-ca\") pod \"maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh\" (UID: \"a4f0223d-3b53-4b7d-ad1d-af0ae3d35738\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh" Sep 06 12:30:48.124796 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:48.124736 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4f0223d-3b53-4b7d-ad1d-af0ae3d35738-maas-api-tls\") pod \"maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh\" (UID: \"a4f0223d-3b53-4b7d-ad1d-af0ae3d35738\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh" Sep 06 12:30:48.226143 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:48.226076 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqn75\" (UniqueName: \"kubernetes.io/projected/a4f0223d-3b53-4b7d-ad1d-af0ae3d35738-kube-api-access-kqn75\") pod \"maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh\" (UID: \"a4f0223d-3b53-4b7d-ad1d-af0ae3d35738\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh" Sep 06 12:30:48.226143 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:48.226135 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4f0223d-3b53-4b7d-ad1d-af0ae3d35738-openshift-service-ca\") pod \"maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh\" (UID: \"a4f0223d-3b53-4b7d-ad1d-af0ae3d35738\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh" Sep 06 12:30:48.226344 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:48.226161 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4f0223d-3b53-4b7d-ad1d-af0ae3d35738-maas-api-tls\") pod \"maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh\" (UID: \"a4f0223d-3b53-4b7d-ad1d-af0ae3d35738\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh" Sep 06 12:30:48.226344 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:48.226194 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4f0223d-3b53-4b7d-ad1d-af0ae3d35738-metrics-certs\") pod \"maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh\" (UID: \"a4f0223d-3b53-4b7d-ad1d-af0ae3d35738\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh" Sep 06 12:30:48.226932 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:48.226910 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4f0223d-3b53-4b7d-ad1d-af0ae3d35738-openshift-service-ca\") pod \"maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh\" (UID: \"a4f0223d-3b53-4b7d-ad1d-af0ae3d35738\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh" Sep 06 12:30:48.228764 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:48.228740 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4f0223d-3b53-4b7d-ad1d-af0ae3d35738-metrics-certs\") pod \"maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh\" (UID: \"a4f0223d-3b53-4b7d-ad1d-af0ae3d35738\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh" Sep 06 12:30:48.228910 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:48.228893 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4f0223d-3b53-4b7d-ad1d-af0ae3d35738-maas-api-tls\") pod \"maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh\" (UID: \"a4f0223d-3b53-4b7d-ad1d-af0ae3d35738\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh" Sep 06 12:30:48.233576 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:48.233557 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqn75\" (UniqueName: \"kubernetes.io/projected/a4f0223d-3b53-4b7d-ad1d-af0ae3d35738-kube-api-access-kqn75\") pod \"maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh\" (UID: \"a4f0223d-3b53-4b7d-ad1d-af0ae3d35738\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh" Sep 06 12:30:48.275137 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:48.275109 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh" Sep 06 12:30:48.549865 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:48.549840 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh"] Sep 06 12:30:48.551503 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:30:48.551465 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4f0223d_3b53_4b7d_ad1d_af0ae3d35738.slice/crio-40f1357a130ad501a2b31f1bc63aa07d9c829bee072a430467f75ca950a5cbf5 WatchSource:0}: Error finding container 40f1357a130ad501a2b31f1bc63aa07d9c829bee072a430467f75ca950a5cbf5: Status 404 returned error can't find the container with id 40f1357a130ad501a2b31f1bc63aa07d9c829bee072a430467f75ca950a5cbf5 Sep 06 12:30:49.563326 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:49.563292 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh" event={"ID":"a4f0223d-3b53-4b7d-ad1d-af0ae3d35738","Type":"ContainerStarted","Data":"11773e520d2d16225e6a5d760ced48bf7f399c4c07adcd311cc698bc9d71dc4b"} Sep 06 12:30:49.563688 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:49.563377 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh" event={"ID":"a4f0223d-3b53-4b7d-ad1d-af0ae3d35738","Type":"ContainerStarted","Data":"40f1357a130ad501a2b31f1bc63aa07d9c829bee072a430467f75ca950a5cbf5"} Sep 06 12:30:49.563688 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:49.563439 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh" Sep 06 12:30:49.582791 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:49.582722 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh" podStartSLOduration=2.18059094 podStartE2EDuration="2.582710982s" totalImagesPullingTime="402.120042ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:30:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:30:49.581849532 +0000 UTC m=+765.437607444" watchObservedRunningTime="2026-09-06 12:30:49.582710982 +0000 UTC m=+765.438468868" Sep 06 12:30:55.573839 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:55.573802 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh" Sep 06 12:30:56.908072 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:56.908040 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf"] Sep 06 12:30:56.908553 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:56.908286 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf" podUID="2e7df151-3df3-462d-af7b-d447d4c0d0c2" containerName="maas-api" containerID="cri-o://6ee9a68a066e2d647d06a5a6fe78121a04fd821295de5371910e32b73c670262" gracePeriod=30 Sep 06 12:30:56.972899 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:56.972866 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8f9eca99-58fc7d74-ftm2b"] Sep 06 12:30:56.973213 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:56.973164 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8f9eca99-58fc7d74-ftm2b" podUID="8b145a0a-2cdd-4109-8a20-760575f03651" containerName="payload-pre-processing" containerID="cri-o://1c3ede3cbc3832ca48d5b9de8f48a242983e2127d90d4ea77cae0820149ea43d" gracePeriod=30 Sep 06 12:30:57.026386 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:30:57.026347 2576 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b145a0a_2cdd_4109_8a20_760575f03651.slice/crio-1c3ede3cbc3832ca48d5b9de8f48a242983e2127d90d4ea77cae0820149ea43d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b145a0a_2cdd_4109_8a20_760575f03651.slice/crio-conmon-1c3ede3cbc3832ca48d5b9de8f48a242983e2127d90d4ea77cae0820149ea43d.scope\": RecentStats: unable to find data in memory cache]" Sep 06 12:30:57.261603 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:57.261555 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf" Sep 06 12:30:57.302778 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:57.297914 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8f9eca99-58fc7d74-ftm2b" Sep 06 12:30:57.308646 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:57.308618 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e7df151-3df3-462d-af7b-d447d4c0d0c2-metrics-certs\") pod \"2e7df151-3df3-462d-af7b-d447d4c0d0c2\" (UID: \"2e7df151-3df3-462d-af7b-d447d4c0d0c2\") " Sep 06 12:30:57.308814 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:57.308724 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e7df151-3df3-462d-af7b-d447d4c0d0c2-maas-api-tls\") pod \"2e7df151-3df3-462d-af7b-d447d4c0d0c2\" (UID: \"2e7df151-3df3-462d-af7b-d447d4c0d0c2\") " Sep 06 12:30:57.308814 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:57.308769 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e7df151-3df3-462d-af7b-d447d4c0d0c2-openshift-service-ca\") pod \"2e7df151-3df3-462d-af7b-d447d4c0d0c2\" (UID: \"2e7df151-3df3-462d-af7b-d447d4c0d0c2\") " Sep 06 12:30:57.308943 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:57.308868 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bgr9\" (UniqueName: \"kubernetes.io/projected/2e7df151-3df3-462d-af7b-d447d4c0d0c2-kube-api-access-5bgr9\") pod \"2e7df151-3df3-462d-af7b-d447d4c0d0c2\" (UID: \"2e7df151-3df3-462d-af7b-d447d4c0d0c2\") " Sep 06 12:30:57.309321 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:57.309132 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e7df151-3df3-462d-af7b-d447d4c0d0c2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2e7df151-3df3-462d-af7b-d447d4c0d0c2" (UID: "2e7df151-3df3-462d-af7b-d447d4c0d0c2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:30:57.309321 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:57.309243 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e7df151-3df3-462d-af7b-d447d4c0d0c2-openshift-service-ca\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:30:57.311090 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:57.311054 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e7df151-3df3-462d-af7b-d447d4c0d0c2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2e7df151-3df3-462d-af7b-d447d4c0d0c2" (UID: "2e7df151-3df3-462d-af7b-d447d4c0d0c2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 12:30:57.311612 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:57.311592 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e7df151-3df3-462d-af7b-d447d4c0d0c2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2e7df151-3df3-462d-af7b-d447d4c0d0c2" (UID: "2e7df151-3df3-462d-af7b-d447d4c0d0c2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 12:30:57.311908 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:57.311888 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e7df151-3df3-462d-af7b-d447d4c0d0c2-kube-api-access-5bgr9" (OuterVolumeSpecName: "kube-api-access-5bgr9") pod "2e7df151-3df3-462d-af7b-d447d4c0d0c2" (UID: "2e7df151-3df3-462d-af7b-d447d4c0d0c2"). InnerVolumeSpecName "kube-api-access-5bgr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:30:57.409976 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:57.409947 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b145a0a-2cdd-4109-8a20-760575f03651-plugins-config-volume\") pod \"8b145a0a-2cdd-4109-8a20-760575f03651\" (UID: \"8b145a0a-2cdd-4109-8a20-760575f03651\") " Sep 06 12:30:57.410132 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:57.410024 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqlkp\" (UniqueName: \"kubernetes.io/projected/8b145a0a-2cdd-4109-8a20-760575f03651-kube-api-access-tqlkp\") pod \"8b145a0a-2cdd-4109-8a20-760575f03651\" (UID: \"8b145a0a-2cdd-4109-8a20-760575f03651\") " Sep 06 12:30:57.410231 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:57.410218 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e7df151-3df3-462d-af7b-d447d4c0d0c2-maas-api-tls\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:30:57.410301 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:57.410232 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bgr9\" (UniqueName: \"kubernetes.io/projected/2e7df151-3df3-462d-af7b-d447d4c0d0c2-kube-api-access-5bgr9\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:30:57.410301 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:57.410241 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e7df151-3df3-462d-af7b-d447d4c0d0c2-metrics-certs\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:30:57.410402 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:57.410350 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b145a0a-2cdd-4109-8a20-760575f03651-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8b145a0a-2cdd-4109-8a20-760575f03651" (UID: "8b145a0a-2cdd-4109-8a20-760575f03651"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:30:57.412417 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:57.412395 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b145a0a-2cdd-4109-8a20-760575f03651-kube-api-access-tqlkp" (OuterVolumeSpecName: "kube-api-access-tqlkp") pod "8b145a0a-2cdd-4109-8a20-760575f03651" (UID: "8b145a0a-2cdd-4109-8a20-760575f03651"). InnerVolumeSpecName "kube-api-access-tqlkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:30:57.511273 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:57.511222 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqlkp\" (UniqueName: \"kubernetes.io/projected/8b145a0a-2cdd-4109-8a20-760575f03651-kube-api-access-tqlkp\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:30:57.511273 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:57.511242 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b145a0a-2cdd-4109-8a20-760575f03651-plugins-config-volume\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:30:57.611591 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:57.611558 2576 generic.go:356] "Generic (PLEG): container finished" podID="8b145a0a-2cdd-4109-8a20-760575f03651" containerID="1c3ede3cbc3832ca48d5b9de8f48a242983e2127d90d4ea77cae0820149ea43d" exitCode=0 Sep 06 12:30:57.611717 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:57.611609 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8f9eca99-58fc7d74-ftm2b" Sep 06 12:30:57.611717 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:57.611634 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8f9eca99-58fc7d74-ftm2b" event={"ID":"8b145a0a-2cdd-4109-8a20-760575f03651","Type":"ContainerDied","Data":"1c3ede3cbc3832ca48d5b9de8f48a242983e2127d90d4ea77cae0820149ea43d"} Sep 06 12:30:57.611717 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:57.611685 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8f9eca99-58fc7d74-ftm2b" event={"ID":"8b145a0a-2cdd-4109-8a20-760575f03651","Type":"ContainerDied","Data":"672666d5a8e4d62c5118db14d0aff3873cf5d6c570535a6039bd354d6caf7f53"} Sep 06 12:30:57.611717 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:57.611715 2576 scope.go:117] "RemoveContainer" containerID="1c3ede3cbc3832ca48d5b9de8f48a242983e2127d90d4ea77cae0820149ea43d" Sep 06 12:30:57.613214 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:57.613192 2576 generic.go:356] "Generic (PLEG): container finished" podID="2e7df151-3df3-462d-af7b-d447d4c0d0c2" containerID="6ee9a68a066e2d647d06a5a6fe78121a04fd821295de5371910e32b73c670262" exitCode=0 Sep 06 12:30:57.613316 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:57.613241 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf" Sep 06 12:30:57.613316 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:57.613298 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf" event={"ID":"2e7df151-3df3-462d-af7b-d447d4c0d0c2","Type":"ContainerDied","Data":"6ee9a68a066e2d647d06a5a6fe78121a04fd821295de5371910e32b73c670262"} Sep 06 12:30:57.613517 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:57.613367 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf" event={"ID":"2e7df151-3df3-462d-af7b-d447d4c0d0c2","Type":"ContainerDied","Data":"3e16dca68c3c5fe8a6e6a4ea7798fe794137f5bb6073dfc517b228460403585f"} Sep 06 12:30:57.622231 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:57.622213 2576 scope.go:117] "RemoveContainer" containerID="1c3ede3cbc3832ca48d5b9de8f48a242983e2127d90d4ea77cae0820149ea43d" Sep 06 12:30:57.622532 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:30:57.622513 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c3ede3cbc3832ca48d5b9de8f48a242983e2127d90d4ea77cae0820149ea43d\": container with ID starting with 1c3ede3cbc3832ca48d5b9de8f48a242983e2127d90d4ea77cae0820149ea43d not found: ID does not exist" containerID="1c3ede3cbc3832ca48d5b9de8f48a242983e2127d90d4ea77cae0820149ea43d" Sep 06 12:30:57.622591 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:57.622540 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c3ede3cbc3832ca48d5b9de8f48a242983e2127d90d4ea77cae0820149ea43d"} err="failed to get container status \"1c3ede3cbc3832ca48d5b9de8f48a242983e2127d90d4ea77cae0820149ea43d\": rpc error: code = NotFound desc = could not find container \"1c3ede3cbc3832ca48d5b9de8f48a242983e2127d90d4ea77cae0820149ea43d\": container with ID starting with 1c3ede3cbc3832ca48d5b9de8f48a242983e2127d90d4ea77cae0820149ea43d not found: ID does not exist" Sep 06 12:30:57.622591 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:57.622557 2576 scope.go:117] "RemoveContainer" containerID="6ee9a68a066e2d647d06a5a6fe78121a04fd821295de5371910e32b73c670262" Sep 06 12:30:57.630814 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:57.630797 2576 scope.go:117] "RemoveContainer" containerID="6ee9a68a066e2d647d06a5a6fe78121a04fd821295de5371910e32b73c670262" Sep 06 12:30:57.631070 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:30:57.631055 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ee9a68a066e2d647d06a5a6fe78121a04fd821295de5371910e32b73c670262\": container with ID starting with 6ee9a68a066e2d647d06a5a6fe78121a04fd821295de5371910e32b73c670262 not found: ID does not exist" containerID="6ee9a68a066e2d647d06a5a6fe78121a04fd821295de5371910e32b73c670262" Sep 06 12:30:57.631118 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:57.631077 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee9a68a066e2d647d06a5a6fe78121a04fd821295de5371910e32b73c670262"} err="failed to get container status \"6ee9a68a066e2d647d06a5a6fe78121a04fd821295de5371910e32b73c670262\": rpc error: code = NotFound desc = could not find container \"6ee9a68a066e2d647d06a5a6fe78121a04fd821295de5371910e32b73c670262\": container with ID starting with 6ee9a68a066e2d647d06a5a6fe78121a04fd821295de5371910e32b73c670262 not found: ID does not exist" Sep 06 12:30:57.638625 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:57.638605 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8f9eca99-58fc7d74-ftm2b"] Sep 06 12:30:57.643893 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:57.643874 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8f9eca99-58fc7d74-ftm2b"] Sep 06 12:30:57.653525 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:57.653505 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf"] Sep 06 12:30:57.659111 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:57.659090 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8f9eca99-74d7f74b64-6kqzf"] Sep 06 12:30:58.652923 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:58.652890 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e7df151-3df3-462d-af7b-d447d4c0d0c2" path="/var/lib/kubelet/pods/2e7df151-3df3-462d-af7b-d447d4c0d0c2/volumes" Sep 06 12:30:58.653308 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:30:58.653247 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b145a0a-2cdd-4109-8a20-760575f03651" path="/var/lib/kubelet/pods/8b145a0a-2cdd-4109-8a20-760575f03651/volumes" Sep 06 12:31:00.847187 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:00.847156 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n"] Sep 06 12:31:00.847948 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:00.847922 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e7df151-3df3-462d-af7b-d447d4c0d0c2" containerName="maas-api" Sep 06 12:31:00.848117 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:00.848101 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e7df151-3df3-462d-af7b-d447d4c0d0c2" containerName="maas-api" Sep 06 12:31:00.848242 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:00.848223 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b145a0a-2cdd-4109-8a20-760575f03651" containerName="payload-pre-processing" Sep 06 12:31:00.848313 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:00.848242 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b145a0a-2cdd-4109-8a20-760575f03651" containerName="payload-pre-processing" Sep 06 12:31:00.848488 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:00.848382 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e7df151-3df3-462d-af7b-d447d4c0d0c2" containerName="maas-api" Sep 06 12:31:00.848488 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:00.848400 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b145a0a-2cdd-4109-8a20-760575f03651" containerName="payload-pre-processing" Sep 06 12:31:00.851692 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:00.851671 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n" Sep 06 12:31:00.854102 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:00.854081 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-924ef9c2-serving-cert\"" Sep 06 12:31:00.859418 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:00.859393 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n"] Sep 06 12:31:00.943141 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:00.943116 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fafe8de8-1764-4ffd-9f32-3f0af8761b39-metrics-certs\") pod \"maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n\" (UID: \"fafe8de8-1764-4ffd-9f32-3f0af8761b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n" Sep 06 12:31:00.943259 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:00.943152 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kffxk\" (UniqueName: \"kubernetes.io/projected/fafe8de8-1764-4ffd-9f32-3f0af8761b39-kube-api-access-kffxk\") pod \"maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n\" (UID: \"fafe8de8-1764-4ffd-9f32-3f0af8761b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n" Sep 06 12:31:00.943259 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:00.943229 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fafe8de8-1764-4ffd-9f32-3f0af8761b39-maas-api-tls\") pod \"maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n\" (UID: \"fafe8de8-1764-4ffd-9f32-3f0af8761b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n" Sep 06 12:31:00.943333 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:00.943302 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fafe8de8-1764-4ffd-9f32-3f0af8761b39-openshift-service-ca\") pod \"maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n\" (UID: \"fafe8de8-1764-4ffd-9f32-3f0af8761b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n" Sep 06 12:31:01.044400 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:01.044375 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fafe8de8-1764-4ffd-9f32-3f0af8761b39-openshift-service-ca\") pod \"maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n\" (UID: \"fafe8de8-1764-4ffd-9f32-3f0af8761b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n" Sep 06 12:31:01.044537 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:01.044421 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fafe8de8-1764-4ffd-9f32-3f0af8761b39-metrics-certs\") pod \"maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n\" (UID: \"fafe8de8-1764-4ffd-9f32-3f0af8761b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n" Sep 06 12:31:01.044537 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:01.044439 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kffxk\" (UniqueName: \"kubernetes.io/projected/fafe8de8-1764-4ffd-9f32-3f0af8761b39-kube-api-access-kffxk\") pod \"maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n\" (UID: \"fafe8de8-1764-4ffd-9f32-3f0af8761b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n" Sep 06 12:31:01.044537 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:01.044482 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fafe8de8-1764-4ffd-9f32-3f0af8761b39-maas-api-tls\") pod \"maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n\" (UID: \"fafe8de8-1764-4ffd-9f32-3f0af8761b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n" Sep 06 12:31:01.045120 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:01.045087 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fafe8de8-1764-4ffd-9f32-3f0af8761b39-openshift-service-ca\") pod \"maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n\" (UID: \"fafe8de8-1764-4ffd-9f32-3f0af8761b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n" Sep 06 12:31:01.046903 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:01.046884 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fafe8de8-1764-4ffd-9f32-3f0af8761b39-metrics-certs\") pod \"maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n\" (UID: \"fafe8de8-1764-4ffd-9f32-3f0af8761b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n" Sep 06 12:31:01.046970 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:01.046939 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fafe8de8-1764-4ffd-9f32-3f0af8761b39-maas-api-tls\") pod \"maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n\" (UID: \"fafe8de8-1764-4ffd-9f32-3f0af8761b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n" Sep 06 12:31:01.052225 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:01.052198 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kffxk\" (UniqueName: \"kubernetes.io/projected/fafe8de8-1764-4ffd-9f32-3f0af8761b39-kube-api-access-kffxk\") pod \"maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n\" (UID: \"fafe8de8-1764-4ffd-9f32-3f0af8761b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n" Sep 06 12:31:01.166417 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:01.166391 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n" Sep 06 12:31:01.456480 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:01.456452 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n"] Sep 06 12:31:01.459285 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:31:01.459254 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfafe8de8_1764_4ffd_9f32_3f0af8761b39.slice/crio-8a18f24681e14b6a3c10cb99ddd31e216771bf290864313ad7dadb620384e25f WatchSource:0}: Error finding container 8a18f24681e14b6a3c10cb99ddd31e216771bf290864313ad7dadb620384e25f: Status 404 returned error can't find the container with id 8a18f24681e14b6a3c10cb99ddd31e216771bf290864313ad7dadb620384e25f Sep 06 12:31:01.581623 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:01.581594 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-924ef9c2-76bbdc69fd-p8p7l"] Sep 06 12:31:01.586592 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:01.586574 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-924ef9c2-76bbdc69fd-p8p7l" Sep 06 12:31:01.589245 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:01.589224 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-924ef9c2-dockercfg-qcnst\"" Sep 06 12:31:01.589328 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:01.589276 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-924ef9c2\"" Sep 06 12:31:01.592900 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:01.592879 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-924ef9c2-76bbdc69fd-p8p7l"] Sep 06 12:31:01.637106 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:01.637080 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n" event={"ID":"fafe8de8-1764-4ffd-9f32-3f0af8761b39","Type":"ContainerStarted","Data":"8a18f24681e14b6a3c10cb99ddd31e216771bf290864313ad7dadb620384e25f"} Sep 06 12:31:01.650796 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:01.650732 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abb610ad-efb8-4d33-8051-628d87bc05d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-924ef9c2-76bbdc69fd-p8p7l\" (UID: \"abb610ad-efb8-4d33-8051-628d87bc05d5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-924ef9c2-76bbdc69fd-p8p7l" Sep 06 12:31:01.650908 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:01.650890 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgsxx\" (UniqueName: \"kubernetes.io/projected/abb610ad-efb8-4d33-8051-628d87bc05d5-kube-api-access-cgsxx\") pod \"payload-pre-processing-e2e-migrate-924ef9c2-76bbdc69fd-p8p7l\" (UID: \"abb610ad-efb8-4d33-8051-628d87bc05d5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-924ef9c2-76bbdc69fd-p8p7l" Sep 06 12:31:01.752023 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:01.751934 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgsxx\" (UniqueName: \"kubernetes.io/projected/abb610ad-efb8-4d33-8051-628d87bc05d5-kube-api-access-cgsxx\") pod \"payload-pre-processing-e2e-migrate-924ef9c2-76bbdc69fd-p8p7l\" (UID: \"abb610ad-efb8-4d33-8051-628d87bc05d5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-924ef9c2-76bbdc69fd-p8p7l" Sep 06 12:31:01.752196 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:01.752033 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abb610ad-efb8-4d33-8051-628d87bc05d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-924ef9c2-76bbdc69fd-p8p7l\" (UID: \"abb610ad-efb8-4d33-8051-628d87bc05d5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-924ef9c2-76bbdc69fd-p8p7l" Sep 06 12:31:01.752704 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:01.752685 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abb610ad-efb8-4d33-8051-628d87bc05d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-924ef9c2-76bbdc69fd-p8p7l\" (UID: \"abb610ad-efb8-4d33-8051-628d87bc05d5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-924ef9c2-76bbdc69fd-p8p7l" Sep 06 12:31:01.760279 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:01.760250 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgsxx\" (UniqueName: \"kubernetes.io/projected/abb610ad-efb8-4d33-8051-628d87bc05d5-kube-api-access-cgsxx\") pod \"payload-pre-processing-e2e-migrate-924ef9c2-76bbdc69fd-p8p7l\" (UID: \"abb610ad-efb8-4d33-8051-628d87bc05d5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-924ef9c2-76bbdc69fd-p8p7l" Sep 06 12:31:01.900271 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:01.900144 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-924ef9c2-76bbdc69fd-p8p7l" Sep 06 12:31:02.170720 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:02.170694 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-924ef9c2-76bbdc69fd-p8p7l"] Sep 06 12:31:02.172683 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:31:02.172658 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb610ad_efb8_4d33_8051_628d87bc05d5.slice/crio-937ddcdf8249d05cc8d662987ec648388217d5211c948f4f77e0ff286b430930 WatchSource:0}: Error finding container 937ddcdf8249d05cc8d662987ec648388217d5211c948f4f77e0ff286b430930: Status 404 returned error can't find the container with id 937ddcdf8249d05cc8d662987ec648388217d5211c948f4f77e0ff286b430930 Sep 06 12:31:02.644047 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:02.644016 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n" event={"ID":"fafe8de8-1764-4ffd-9f32-3f0af8761b39","Type":"ContainerStarted","Data":"d41dc47aca8c6e90b48b3bef6078c5ca6d38e168422de43fe64bc356510d7219"} Sep 06 12:31:02.644250 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:02.644139 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n" Sep 06 12:31:02.645826 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:02.645799 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-924ef9c2-76bbdc69fd-p8p7l" event={"ID":"abb610ad-efb8-4d33-8051-628d87bc05d5","Type":"ContainerStarted","Data":"7ed8d1031db45311977416dd794a78f0d6cc295738bc34f9f3ae3f3a5b55b06c"} Sep 06 12:31:02.645942 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:02.645841 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-924ef9c2-76bbdc69fd-p8p7l" event={"ID":"abb610ad-efb8-4d33-8051-628d87bc05d5","Type":"ContainerStarted","Data":"937ddcdf8249d05cc8d662987ec648388217d5211c948f4f77e0ff286b430930"} Sep 06 12:31:02.645942 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:02.645874 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-924ef9c2-76bbdc69fd-p8p7l" Sep 06 12:31:02.663623 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:02.663584 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n" podStartSLOduration=2.33089362 podStartE2EDuration="2.663569393s" totalImagesPullingTime="332.675773ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:31:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:31:02.660176177 +0000 UTC m=+778.515934086" watchObservedRunningTime="2026-09-06 12:31:02.663569393 +0000 UTC m=+778.519327280" Sep 06 12:31:02.675799 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:02.675724 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-924ef9c2-76bbdc69fd-p8p7l" podStartSLOduration=1.675710152 podStartE2EDuration="1.675710152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:31:02.673576697 +0000 UTC m=+778.529334607" watchObservedRunningTime="2026-09-06 12:31:02.675710152 +0000 UTC m=+778.531468075" Sep 06 12:31:08.656436 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:08.656408 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n" Sep 06 12:31:13.653683 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:13.653653 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-924ef9c2-76bbdc69fd-p8p7l" Sep 06 12:31:31.417224 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:31.417139 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n"] Sep 06 12:31:31.417717 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:31.417382 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n" podUID="fafe8de8-1764-4ffd-9f32-3f0af8761b39" containerName="maas-api" containerID="cri-o://d41dc47aca8c6e90b48b3bef6078c5ca6d38e168422de43fe64bc356510d7219" gracePeriod=30 Sep 06 12:31:31.482361 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:31.482330 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-924ef9c2-76bbdc69fd-p8p7l"] Sep 06 12:31:31.482823 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:31.482791 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-924ef9c2-76bbdc69fd-p8p7l" podUID="abb610ad-efb8-4d33-8051-628d87bc05d5" containerName="payload-pre-processing" containerID="cri-o://7ed8d1031db45311977416dd794a78f0d6cc295738bc34f9f3ae3f3a5b55b06c" gracePeriod=30 Sep 06 12:31:31.742780 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:31.742297 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n" Sep 06 12:31:31.775563 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:31.775537 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-924ef9c2-76bbdc69fd-p8p7l" Sep 06 12:31:31.797637 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:31.797610 2576 generic.go:356] "Generic (PLEG): container finished" podID="fafe8de8-1764-4ffd-9f32-3f0af8761b39" containerID="d41dc47aca8c6e90b48b3bef6078c5ca6d38e168422de43fe64bc356510d7219" exitCode=0 Sep 06 12:31:31.797785 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:31.797655 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n" Sep 06 12:31:31.797785 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:31.797692 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n" event={"ID":"fafe8de8-1764-4ffd-9f32-3f0af8761b39","Type":"ContainerDied","Data":"d41dc47aca8c6e90b48b3bef6078c5ca6d38e168422de43fe64bc356510d7219"} Sep 06 12:31:31.797785 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:31.797769 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n" event={"ID":"fafe8de8-1764-4ffd-9f32-3f0af8761b39","Type":"ContainerDied","Data":"8a18f24681e14b6a3c10cb99ddd31e216771bf290864313ad7dadb620384e25f"} Sep 06 12:31:31.798005 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:31.797803 2576 scope.go:117] "RemoveContainer" containerID="d41dc47aca8c6e90b48b3bef6078c5ca6d38e168422de43fe64bc356510d7219" Sep 06 12:31:31.799021 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:31.798996 2576 generic.go:356] "Generic (PLEG): container finished" podID="abb610ad-efb8-4d33-8051-628d87bc05d5" containerID="7ed8d1031db45311977416dd794a78f0d6cc295738bc34f9f3ae3f3a5b55b06c" exitCode=0 Sep 06 12:31:31.799100 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:31.799032 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-924ef9c2-76bbdc69fd-p8p7l" Sep 06 12:31:31.799100 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:31.799080 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-924ef9c2-76bbdc69fd-p8p7l" event={"ID":"abb610ad-efb8-4d33-8051-628d87bc05d5","Type":"ContainerDied","Data":"7ed8d1031db45311977416dd794a78f0d6cc295738bc34f9f3ae3f3a5b55b06c"} Sep 06 12:31:31.799211 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:31.799132 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-924ef9c2-76bbdc69fd-p8p7l" event={"ID":"abb610ad-efb8-4d33-8051-628d87bc05d5","Type":"ContainerDied","Data":"937ddcdf8249d05cc8d662987ec648388217d5211c948f4f77e0ff286b430930"} Sep 06 12:31:31.808278 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:31.808258 2576 scope.go:117] "RemoveContainer" containerID="d41dc47aca8c6e90b48b3bef6078c5ca6d38e168422de43fe64bc356510d7219" Sep 06 12:31:31.808610 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:31:31.808588 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d41dc47aca8c6e90b48b3bef6078c5ca6d38e168422de43fe64bc356510d7219\": container with ID starting with d41dc47aca8c6e90b48b3bef6078c5ca6d38e168422de43fe64bc356510d7219 not found: ID does not exist" containerID="d41dc47aca8c6e90b48b3bef6078c5ca6d38e168422de43fe64bc356510d7219" Sep 06 12:31:31.808712 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:31.808621 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d41dc47aca8c6e90b48b3bef6078c5ca6d38e168422de43fe64bc356510d7219"} err="failed to get container status \"d41dc47aca8c6e90b48b3bef6078c5ca6d38e168422de43fe64bc356510d7219\": rpc error: code = NotFound desc = could not find container \"d41dc47aca8c6e90b48b3bef6078c5ca6d38e168422de43fe64bc356510d7219\": container with ID starting with d41dc47aca8c6e90b48b3bef6078c5ca6d38e168422de43fe64bc356510d7219 not found: ID does not exist" Sep 06 12:31:31.808712 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:31.808643 2576 scope.go:117] "RemoveContainer" containerID="7ed8d1031db45311977416dd794a78f0d6cc295738bc34f9f3ae3f3a5b55b06c" Sep 06 12:31:31.817185 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:31.817166 2576 scope.go:117] "RemoveContainer" containerID="7ed8d1031db45311977416dd794a78f0d6cc295738bc34f9f3ae3f3a5b55b06c" Sep 06 12:31:31.817512 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:31:31.817489 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ed8d1031db45311977416dd794a78f0d6cc295738bc34f9f3ae3f3a5b55b06c\": container with ID starting with 7ed8d1031db45311977416dd794a78f0d6cc295738bc34f9f3ae3f3a5b55b06c not found: ID does not exist" containerID="7ed8d1031db45311977416dd794a78f0d6cc295738bc34f9f3ae3f3a5b55b06c" Sep 06 12:31:31.817592 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:31.817522 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ed8d1031db45311977416dd794a78f0d6cc295738bc34f9f3ae3f3a5b55b06c"} err="failed to get container status \"7ed8d1031db45311977416dd794a78f0d6cc295738bc34f9f3ae3f3a5b55b06c\": rpc error: code = NotFound desc = could not find container \"7ed8d1031db45311977416dd794a78f0d6cc295738bc34f9f3ae3f3a5b55b06c\": container with ID starting with 7ed8d1031db45311977416dd794a78f0d6cc295738bc34f9f3ae3f3a5b55b06c not found: ID does not exist" Sep 06 12:31:31.818921 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:31.818899 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fafe8de8-1764-4ffd-9f32-3f0af8761b39-openshift-service-ca\") pod \"fafe8de8-1764-4ffd-9f32-3f0af8761b39\" (UID: \"fafe8de8-1764-4ffd-9f32-3f0af8761b39\") " Sep 06 12:31:31.819132 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:31.818991 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abb610ad-efb8-4d33-8051-628d87bc05d5-plugins-config-volume\") pod \"abb610ad-efb8-4d33-8051-628d87bc05d5\" (UID: \"abb610ad-efb8-4d33-8051-628d87bc05d5\") " Sep 06 12:31:31.819132 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:31.819027 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kffxk\" (UniqueName: \"kubernetes.io/projected/fafe8de8-1764-4ffd-9f32-3f0af8761b39-kube-api-access-kffxk\") pod \"fafe8de8-1764-4ffd-9f32-3f0af8761b39\" (UID: \"fafe8de8-1764-4ffd-9f32-3f0af8761b39\") " Sep 06 12:31:31.819132 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:31.819075 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fafe8de8-1764-4ffd-9f32-3f0af8761b39-maas-api-tls\") pod \"fafe8de8-1764-4ffd-9f32-3f0af8761b39\" (UID: \"fafe8de8-1764-4ffd-9f32-3f0af8761b39\") " Sep 06 12:31:31.819132 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:31.819095 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fafe8de8-1764-4ffd-9f32-3f0af8761b39-metrics-certs\") pod \"fafe8de8-1764-4ffd-9f32-3f0af8761b39\" (UID: \"fafe8de8-1764-4ffd-9f32-3f0af8761b39\") " Sep 06 12:31:31.819280 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:31.819136 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgsxx\" (UniqueName: \"kubernetes.io/projected/abb610ad-efb8-4d33-8051-628d87bc05d5-kube-api-access-cgsxx\") pod \"abb610ad-efb8-4d33-8051-628d87bc05d5\" (UID: \"abb610ad-efb8-4d33-8051-628d87bc05d5\") " Sep 06 12:31:31.819997 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:31.819967 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fafe8de8-1764-4ffd-9f32-3f0af8761b39-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fafe8de8-1764-4ffd-9f32-3f0af8761b39" (UID: "fafe8de8-1764-4ffd-9f32-3f0af8761b39"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:31:31.820745 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:31.820055 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abb610ad-efb8-4d33-8051-628d87bc05d5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "abb610ad-efb8-4d33-8051-628d87bc05d5" (UID: "abb610ad-efb8-4d33-8051-628d87bc05d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:31:31.822864 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:31.822837 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fafe8de8-1764-4ffd-9f32-3f0af8761b39-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fafe8de8-1764-4ffd-9f32-3f0af8761b39" (UID: "fafe8de8-1764-4ffd-9f32-3f0af8761b39"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 12:31:31.822977 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:31.822961 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abb610ad-efb8-4d33-8051-628d87bc05d5-kube-api-access-cgsxx" (OuterVolumeSpecName: "kube-api-access-cgsxx") pod "abb610ad-efb8-4d33-8051-628d87bc05d5" (UID: "abb610ad-efb8-4d33-8051-628d87bc05d5"). InnerVolumeSpecName "kube-api-access-cgsxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:31:31.823067 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:31.823051 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fafe8de8-1764-4ffd-9f32-3f0af8761b39-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fafe8de8-1764-4ffd-9f32-3f0af8761b39" (UID: "fafe8de8-1764-4ffd-9f32-3f0af8761b39"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 12:31:31.823318 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:31.823292 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fafe8de8-1764-4ffd-9f32-3f0af8761b39-kube-api-access-kffxk" (OuterVolumeSpecName: "kube-api-access-kffxk") pod "fafe8de8-1764-4ffd-9f32-3f0af8761b39" (UID: "fafe8de8-1764-4ffd-9f32-3f0af8761b39"). InnerVolumeSpecName "kube-api-access-kffxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:31:31.919870 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:31.919843 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgsxx\" (UniqueName: \"kubernetes.io/projected/abb610ad-efb8-4d33-8051-628d87bc05d5-kube-api-access-cgsxx\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:31:31.919870 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:31.919866 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fafe8de8-1764-4ffd-9f32-3f0af8761b39-openshift-service-ca\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:31:31.919870 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:31.919874 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abb610ad-efb8-4d33-8051-628d87bc05d5-plugins-config-volume\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:31:31.920075 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:31.919884 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kffxk\" (UniqueName: \"kubernetes.io/projected/fafe8de8-1764-4ffd-9f32-3f0af8761b39-kube-api-access-kffxk\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:31:31.920075 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:31.919893 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fafe8de8-1764-4ffd-9f32-3f0af8761b39-maas-api-tls\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:31:31.920075 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:31.919901 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fafe8de8-1764-4ffd-9f32-3f0af8761b39-metrics-certs\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:31:32.124743 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:32.124712 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-924ef9c2-76bbdc69fd-p8p7l"] Sep 06 12:31:32.127990 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:32.127969 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-924ef9c2-76bbdc69fd-p8p7l"] Sep 06 12:31:32.136916 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:32.136894 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n"] Sep 06 12:31:32.142435 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:32.142409 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-924ef9c2-88f7684c5-7kg6n"] Sep 06 12:31:32.654420 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:32.654381 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abb610ad-efb8-4d33-8051-628d87bc05d5" path="/var/lib/kubelet/pods/abb610ad-efb8-4d33-8051-628d87bc05d5/volumes" Sep 06 12:31:32.655005 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:32.654968 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fafe8de8-1764-4ffd-9f32-3f0af8761b39" path="/var/lib/kubelet/pods/fafe8de8-1764-4ffd-9f32-3f0af8761b39/volumes" Sep 06 12:31:35.651467 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:35.651432 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww"] Sep 06 12:31:35.652160 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:35.652135 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abb610ad-efb8-4d33-8051-628d87bc05d5" containerName="payload-pre-processing" Sep 06 12:31:35.652160 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:35.652160 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="abb610ad-efb8-4d33-8051-628d87bc05d5" containerName="payload-pre-processing" Sep 06 12:31:35.652370 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:35.652173 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fafe8de8-1764-4ffd-9f32-3f0af8761b39" containerName="maas-api" Sep 06 12:31:35.652370 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:35.652182 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fafe8de8-1764-4ffd-9f32-3f0af8761b39" containerName="maas-api" Sep 06 12:31:35.652370 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:35.652285 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="abb610ad-efb8-4d33-8051-628d87bc05d5" containerName="payload-pre-processing" Sep 06 12:31:35.652370 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:35.652298 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fafe8de8-1764-4ffd-9f32-3f0af8761b39" containerName="maas-api" Sep 06 12:31:35.666501 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:35.666478 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww"] Sep 06 12:31:35.666661 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:35.666589 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww" Sep 06 12:31:35.668812 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:35.668786 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c85cb5d0-serving-cert\"" Sep 06 12:31:35.756665 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:35.756637 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5abb0af6-72ca-437e-a75a-83bb4c942de8-maas-api-tls\") pod \"maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww\" (UID: \"5abb0af6-72ca-437e-a75a-83bb4c942de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww" Sep 06 12:31:35.756807 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:35.756700 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5abb0af6-72ca-437e-a75a-83bb4c942de8-openshift-service-ca\") pod \"maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww\" (UID: \"5abb0af6-72ca-437e-a75a-83bb4c942de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww" Sep 06 12:31:35.756860 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:35.756808 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5abb0af6-72ca-437e-a75a-83bb4c942de8-metrics-certs\") pod \"maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww\" (UID: \"5abb0af6-72ca-437e-a75a-83bb4c942de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww" Sep 06 12:31:35.756860 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:35.756843 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlphh\" (UniqueName: \"kubernetes.io/projected/5abb0af6-72ca-437e-a75a-83bb4c942de8-kube-api-access-dlphh\") pod \"maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww\" (UID: \"5abb0af6-72ca-437e-a75a-83bb4c942de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww" Sep 06 12:31:35.858076 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:35.858047 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5abb0af6-72ca-437e-a75a-83bb4c942de8-openshift-service-ca\") pod \"maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww\" (UID: \"5abb0af6-72ca-437e-a75a-83bb4c942de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww" Sep 06 12:31:35.858236 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:35.858099 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5abb0af6-72ca-437e-a75a-83bb4c942de8-metrics-certs\") pod \"maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww\" (UID: \"5abb0af6-72ca-437e-a75a-83bb4c942de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww" Sep 06 12:31:35.858236 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:35.858133 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlphh\" (UniqueName: \"kubernetes.io/projected/5abb0af6-72ca-437e-a75a-83bb4c942de8-kube-api-access-dlphh\") pod \"maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww\" (UID: \"5abb0af6-72ca-437e-a75a-83bb4c942de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww" Sep 06 12:31:35.858236 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:35.858177 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5abb0af6-72ca-437e-a75a-83bb4c942de8-maas-api-tls\") pod \"maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww\" (UID: \"5abb0af6-72ca-437e-a75a-83bb4c942de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww" Sep 06 12:31:35.858651 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:35.858632 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5abb0af6-72ca-437e-a75a-83bb4c942de8-openshift-service-ca\") pod \"maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww\" (UID: \"5abb0af6-72ca-437e-a75a-83bb4c942de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww" Sep 06 12:31:35.860728 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:35.860703 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5abb0af6-72ca-437e-a75a-83bb4c942de8-metrics-certs\") pod \"maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww\" (UID: \"5abb0af6-72ca-437e-a75a-83bb4c942de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww" Sep 06 12:31:35.860728 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:35.860716 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5abb0af6-72ca-437e-a75a-83bb4c942de8-maas-api-tls\") pod \"maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww\" (UID: \"5abb0af6-72ca-437e-a75a-83bb4c942de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww" Sep 06 12:31:35.865020 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:35.865001 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlphh\" (UniqueName: \"kubernetes.io/projected/5abb0af6-72ca-437e-a75a-83bb4c942de8-kube-api-access-dlphh\") pod \"maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww\" (UID: \"5abb0af6-72ca-437e-a75a-83bb4c942de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww" Sep 06 12:31:35.980972 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:35.980920 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww" Sep 06 12:31:36.254514 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:36.254489 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww"] Sep 06 12:31:36.256985 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:31:36.256959 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5abb0af6_72ca_437e_a75a_83bb4c942de8.slice/crio-3e8b51989c9e2754773db5aed10632719f45f5bf9c71a7495635a39ef39119be WatchSource:0}: Error finding container 3e8b51989c9e2754773db5aed10632719f45f5bf9c71a7495635a39ef39119be: Status 404 returned error can't find the container with id 3e8b51989c9e2754773db5aed10632719f45f5bf9c71a7495635a39ef39119be Sep 06 12:31:36.476917 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:36.476888 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c85cb5d0-cb5947666-km4nj"] Sep 06 12:31:36.481586 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:36.481567 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c85cb5d0-cb5947666-km4nj" Sep 06 12:31:36.483914 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:36.483842 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c85cb5d0-dockercfg-rpv5d\"" Sep 06 12:31:36.483914 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:36.483862 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c85cb5d0\"" Sep 06 12:31:36.487239 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:36.487214 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c85cb5d0-cb5947666-km4nj"] Sep 06 12:31:36.565518 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:36.565423 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7ljc\" (UniqueName: \"kubernetes.io/projected/62056ff7-6c48-46de-9069-3f6e0e39ad17-kube-api-access-t7ljc\") pod \"payload-pre-processing-e2e-ait-c85cb5d0-cb5947666-km4nj\" (UID: \"62056ff7-6c48-46de-9069-3f6e0e39ad17\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c85cb5d0-cb5947666-km4nj" Sep 06 12:31:36.565518 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:36.565496 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62056ff7-6c48-46de-9069-3f6e0e39ad17-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c85cb5d0-cb5947666-km4nj\" (UID: \"62056ff7-6c48-46de-9069-3f6e0e39ad17\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c85cb5d0-cb5947666-km4nj" Sep 06 12:31:36.666500 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:36.666468 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62056ff7-6c48-46de-9069-3f6e0e39ad17-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c85cb5d0-cb5947666-km4nj\" (UID: \"62056ff7-6c48-46de-9069-3f6e0e39ad17\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c85cb5d0-cb5947666-km4nj" Sep 06 12:31:36.666896 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:36.666655 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7ljc\" (UniqueName: \"kubernetes.io/projected/62056ff7-6c48-46de-9069-3f6e0e39ad17-kube-api-access-t7ljc\") pod \"payload-pre-processing-e2e-ait-c85cb5d0-cb5947666-km4nj\" (UID: \"62056ff7-6c48-46de-9069-3f6e0e39ad17\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c85cb5d0-cb5947666-km4nj" Sep 06 12:31:36.667560 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:36.667537 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62056ff7-6c48-46de-9069-3f6e0e39ad17-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c85cb5d0-cb5947666-km4nj\" (UID: \"62056ff7-6c48-46de-9069-3f6e0e39ad17\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c85cb5d0-cb5947666-km4nj" Sep 06 12:31:36.674381 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:36.674362 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7ljc\" (UniqueName: \"kubernetes.io/projected/62056ff7-6c48-46de-9069-3f6e0e39ad17-kube-api-access-t7ljc\") pod \"payload-pre-processing-e2e-ait-c85cb5d0-cb5947666-km4nj\" (UID: \"62056ff7-6c48-46de-9069-3f6e0e39ad17\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c85cb5d0-cb5947666-km4nj" Sep 06 12:31:36.794705 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:36.794681 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c85cb5d0-cb5947666-km4nj" Sep 06 12:31:36.828256 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:36.828162 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww" event={"ID":"5abb0af6-72ca-437e-a75a-83bb4c942de8","Type":"ContainerStarted","Data":"c49e59fcdf1ce846d33422d3d845f7dbdae6d4d1e82f9b3e012039b477337671"} Sep 06 12:31:36.828256 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:36.828220 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww" event={"ID":"5abb0af6-72ca-437e-a75a-83bb4c942de8","Type":"ContainerStarted","Data":"3e8b51989c9e2754773db5aed10632719f45f5bf9c71a7495635a39ef39119be"} Sep 06 12:31:36.828256 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:36.828242 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww" Sep 06 12:31:36.844286 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:36.844228 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww" podStartSLOduration=1.478206776 podStartE2EDuration="1.844207672s" totalImagesPullingTime="366.000896ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:31:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:31:36.84247155 +0000 UTC m=+812.698229477" watchObservedRunningTime="2026-09-06 12:31:36.844207672 +0000 UTC m=+812.699965560" Sep 06 12:31:37.068649 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:37.068618 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c85cb5d0-cb5947666-km4nj"] Sep 06 12:31:37.071958 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:31:37.071931 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62056ff7_6c48_46de_9069_3f6e0e39ad17.slice/crio-b8465e841e436186c1c627a937343f2a2ee3db0e16c0e9d76bb8a41c3a9dd2b2 WatchSource:0}: Error finding container b8465e841e436186c1c627a937343f2a2ee3db0e16c0e9d76bb8a41c3a9dd2b2: Status 404 returned error can't find the container with id b8465e841e436186c1c627a937343f2a2ee3db0e16c0e9d76bb8a41c3a9dd2b2 Sep 06 12:31:37.834833 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:37.834795 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c85cb5d0-cb5947666-km4nj" event={"ID":"62056ff7-6c48-46de-9069-3f6e0e39ad17","Type":"ContainerStarted","Data":"1be28568150b139be3226546f8f7a55fd0eba613e91fe9d16ea6c1cf82af5b4e"} Sep 06 12:31:37.835296 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:37.834848 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c85cb5d0-cb5947666-km4nj" event={"ID":"62056ff7-6c48-46de-9069-3f6e0e39ad17","Type":"ContainerStarted","Data":"b8465e841e436186c1c627a937343f2a2ee3db0e16c0e9d76bb8a41c3a9dd2b2"} Sep 06 12:31:37.835296 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:37.834953 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c85cb5d0-cb5947666-km4nj" Sep 06 12:31:37.852321 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:37.852265 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c85cb5d0-cb5947666-km4nj" podStartSLOduration=1.85224885 podStartE2EDuration="1.85224885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:31:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:31:37.851099421 +0000 UTC m=+813.706857315" watchObservedRunningTime="2026-09-06 12:31:37.85224885 +0000 UTC m=+813.708006737" Sep 06 12:31:42.840047 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:42.840017 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww" Sep 06 12:31:48.846245 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:31:48.846216 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c85cb5d0-cb5947666-km4nj" Sep 06 12:32:06.130060 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:06.130029 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww"] Sep 06 12:32:06.130537 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:06.130261 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww" podUID="5abb0af6-72ca-437e-a75a-83bb4c942de8" containerName="maas-api" containerID="cri-o://c49e59fcdf1ce846d33422d3d845f7dbdae6d4d1e82f9b3e012039b477337671" gracePeriod=30 Sep 06 12:32:06.193224 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:06.193191 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c85cb5d0-cb5947666-km4nj"] Sep 06 12:32:06.193441 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:06.193405 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c85cb5d0-cb5947666-km4nj" podUID="62056ff7-6c48-46de-9069-3f6e0e39ad17" containerName="payload-pre-processing" containerID="cri-o://1be28568150b139be3226546f8f7a55fd0eba613e91fe9d16ea6c1cf82af5b4e" gracePeriod=30 Sep 06 12:32:06.460532 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:06.460508 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww" Sep 06 12:32:06.493610 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:06.493580 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c85cb5d0-cb5947666-km4nj" Sep 06 12:32:06.647122 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:06.647094 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5abb0af6-72ca-437e-a75a-83bb4c942de8-openshift-service-ca\") pod \"5abb0af6-72ca-437e-a75a-83bb4c942de8\" (UID: \"5abb0af6-72ca-437e-a75a-83bb4c942de8\") " Sep 06 12:32:06.647311 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:06.647179 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7ljc\" (UniqueName: \"kubernetes.io/projected/62056ff7-6c48-46de-9069-3f6e0e39ad17-kube-api-access-t7ljc\") pod \"62056ff7-6c48-46de-9069-3f6e0e39ad17\" (UID: \"62056ff7-6c48-46de-9069-3f6e0e39ad17\") " Sep 06 12:32:06.647311 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:06.647229 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5abb0af6-72ca-437e-a75a-83bb4c942de8-metrics-certs\") pod \"5abb0af6-72ca-437e-a75a-83bb4c942de8\" (UID: \"5abb0af6-72ca-437e-a75a-83bb4c942de8\") " Sep 06 12:32:06.647311 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:06.647267 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5abb0af6-72ca-437e-a75a-83bb4c942de8-maas-api-tls\") pod \"5abb0af6-72ca-437e-a75a-83bb4c942de8\" (UID: \"5abb0af6-72ca-437e-a75a-83bb4c942de8\") " Sep 06 12:32:06.647311 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:06.647305 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62056ff7-6c48-46de-9069-3f6e0e39ad17-plugins-config-volume\") pod \"62056ff7-6c48-46de-9069-3f6e0e39ad17\" (UID: \"62056ff7-6c48-46de-9069-3f6e0e39ad17\") " Sep 06 12:32:06.647537 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:06.647327 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlphh\" (UniqueName: \"kubernetes.io/projected/5abb0af6-72ca-437e-a75a-83bb4c942de8-kube-api-access-dlphh\") pod \"5abb0af6-72ca-437e-a75a-83bb4c942de8\" (UID: \"5abb0af6-72ca-437e-a75a-83bb4c942de8\") " Sep 06 12:32:06.647537 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:06.647514 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5abb0af6-72ca-437e-a75a-83bb4c942de8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5abb0af6-72ca-437e-a75a-83bb4c942de8" (UID: "5abb0af6-72ca-437e-a75a-83bb4c942de8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:32:06.647704 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:06.647684 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5abb0af6-72ca-437e-a75a-83bb4c942de8-openshift-service-ca\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:32:06.648381 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:06.648310 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62056ff7-6c48-46de-9069-3f6e0e39ad17-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "62056ff7-6c48-46de-9069-3f6e0e39ad17" (UID: "62056ff7-6c48-46de-9069-3f6e0e39ad17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:32:06.649612 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:06.649575 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62056ff7-6c48-46de-9069-3f6e0e39ad17-kube-api-access-t7ljc" (OuterVolumeSpecName: "kube-api-access-t7ljc") pod "62056ff7-6c48-46de-9069-3f6e0e39ad17" (UID: "62056ff7-6c48-46de-9069-3f6e0e39ad17"). InnerVolumeSpecName "kube-api-access-t7ljc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:32:06.649723 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:06.649653 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5abb0af6-72ca-437e-a75a-83bb4c942de8-kube-api-access-dlphh" (OuterVolumeSpecName: "kube-api-access-dlphh") pod "5abb0af6-72ca-437e-a75a-83bb4c942de8" (UID: "5abb0af6-72ca-437e-a75a-83bb4c942de8"). InnerVolumeSpecName "kube-api-access-dlphh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:32:06.649723 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:06.649665 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5abb0af6-72ca-437e-a75a-83bb4c942de8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5abb0af6-72ca-437e-a75a-83bb4c942de8" (UID: "5abb0af6-72ca-437e-a75a-83bb4c942de8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 12:32:06.650065 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:06.650049 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5abb0af6-72ca-437e-a75a-83bb4c942de8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5abb0af6-72ca-437e-a75a-83bb4c942de8" (UID: "5abb0af6-72ca-437e-a75a-83bb4c942de8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 12:32:06.748888 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:06.748863 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7ljc\" (UniqueName: \"kubernetes.io/projected/62056ff7-6c48-46de-9069-3f6e0e39ad17-kube-api-access-t7ljc\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:32:06.748976 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:06.748891 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5abb0af6-72ca-437e-a75a-83bb4c942de8-metrics-certs\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:32:06.748976 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:06.748901 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5abb0af6-72ca-437e-a75a-83bb4c942de8-maas-api-tls\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:32:06.748976 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:06.748909 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62056ff7-6c48-46de-9069-3f6e0e39ad17-plugins-config-volume\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:32:06.748976 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:06.748917 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlphh\" (UniqueName: \"kubernetes.io/projected/5abb0af6-72ca-437e-a75a-83bb4c942de8-kube-api-access-dlphh\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:32:07.004658 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:07.004596 2576 generic.go:356] "Generic (PLEG): container finished" podID="5abb0af6-72ca-437e-a75a-83bb4c942de8" containerID="c49e59fcdf1ce846d33422d3d845f7dbdae6d4d1e82f9b3e012039b477337671" exitCode=0 Sep 06 12:32:07.004658 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:07.004651 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww" Sep 06 12:32:07.004880 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:07.004688 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww" event={"ID":"5abb0af6-72ca-437e-a75a-83bb4c942de8","Type":"ContainerDied","Data":"c49e59fcdf1ce846d33422d3d845f7dbdae6d4d1e82f9b3e012039b477337671"} Sep 06 12:32:07.004880 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:07.004748 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww" event={"ID":"5abb0af6-72ca-437e-a75a-83bb4c942de8","Type":"ContainerDied","Data":"3e8b51989c9e2754773db5aed10632719f45f5bf9c71a7495635a39ef39119be"} Sep 06 12:32:07.004880 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:07.004786 2576 scope.go:117] "RemoveContainer" containerID="c49e59fcdf1ce846d33422d3d845f7dbdae6d4d1e82f9b3e012039b477337671" Sep 06 12:32:07.006609 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:07.006392 2576 generic.go:356] "Generic (PLEG): container finished" podID="62056ff7-6c48-46de-9069-3f6e0e39ad17" containerID="1be28568150b139be3226546f8f7a55fd0eba613e91fe9d16ea6c1cf82af5b4e" exitCode=0 Sep 06 12:32:07.006609 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:07.006468 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c85cb5d0-cb5947666-km4nj" Sep 06 12:32:07.006609 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:07.006492 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c85cb5d0-cb5947666-km4nj" event={"ID":"62056ff7-6c48-46de-9069-3f6e0e39ad17","Type":"ContainerDied","Data":"1be28568150b139be3226546f8f7a55fd0eba613e91fe9d16ea6c1cf82af5b4e"} Sep 06 12:32:07.006609 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:07.006556 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c85cb5d0-cb5947666-km4nj" event={"ID":"62056ff7-6c48-46de-9069-3f6e0e39ad17","Type":"ContainerDied","Data":"b8465e841e436186c1c627a937343f2a2ee3db0e16c0e9d76bb8a41c3a9dd2b2"} Sep 06 12:32:07.015156 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:07.015126 2576 scope.go:117] "RemoveContainer" containerID="c49e59fcdf1ce846d33422d3d845f7dbdae6d4d1e82f9b3e012039b477337671" Sep 06 12:32:07.015426 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:32:07.015401 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c49e59fcdf1ce846d33422d3d845f7dbdae6d4d1e82f9b3e012039b477337671\": container with ID starting with c49e59fcdf1ce846d33422d3d845f7dbdae6d4d1e82f9b3e012039b477337671 not found: ID does not exist" containerID="c49e59fcdf1ce846d33422d3d845f7dbdae6d4d1e82f9b3e012039b477337671" Sep 06 12:32:07.015515 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:07.015431 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c49e59fcdf1ce846d33422d3d845f7dbdae6d4d1e82f9b3e012039b477337671"} err="failed to get container status \"c49e59fcdf1ce846d33422d3d845f7dbdae6d4d1e82f9b3e012039b477337671\": rpc error: code = NotFound desc = could not find container \"c49e59fcdf1ce846d33422d3d845f7dbdae6d4d1e82f9b3e012039b477337671\": container with ID starting with c49e59fcdf1ce846d33422d3d845f7dbdae6d4d1e82f9b3e012039b477337671 not found: ID does not exist" Sep 06 12:32:07.015515 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:07.015447 2576 scope.go:117] "RemoveContainer" containerID="1be28568150b139be3226546f8f7a55fd0eba613e91fe9d16ea6c1cf82af5b4e" Sep 06 12:32:07.023867 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:07.023851 2576 scope.go:117] "RemoveContainer" containerID="1be28568150b139be3226546f8f7a55fd0eba613e91fe9d16ea6c1cf82af5b4e" Sep 06 12:32:07.024165 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:32:07.024144 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1be28568150b139be3226546f8f7a55fd0eba613e91fe9d16ea6c1cf82af5b4e\": container with ID starting with 1be28568150b139be3226546f8f7a55fd0eba613e91fe9d16ea6c1cf82af5b4e not found: ID does not exist" containerID="1be28568150b139be3226546f8f7a55fd0eba613e91fe9d16ea6c1cf82af5b4e" Sep 06 12:32:07.024219 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:07.024178 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1be28568150b139be3226546f8f7a55fd0eba613e91fe9d16ea6c1cf82af5b4e"} err="failed to get container status \"1be28568150b139be3226546f8f7a55fd0eba613e91fe9d16ea6c1cf82af5b4e\": rpc error: code = NotFound desc = could not find container \"1be28568150b139be3226546f8f7a55fd0eba613e91fe9d16ea6c1cf82af5b4e\": container with ID starting with 1be28568150b139be3226546f8f7a55fd0eba613e91fe9d16ea6c1cf82af5b4e not found: ID does not exist" Sep 06 12:32:07.026066 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:07.026048 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww"] Sep 06 12:32:07.030946 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:07.030926 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c85cb5d0-8b6c97dd4-svxww"] Sep 06 12:32:07.039993 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:07.039972 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c85cb5d0-cb5947666-km4nj"] Sep 06 12:32:07.044914 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:07.044896 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c85cb5d0-cb5947666-km4nj"] Sep 06 12:32:08.653803 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:08.653746 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5abb0af6-72ca-437e-a75a-83bb4c942de8" path="/var/lib/kubelet/pods/5abb0af6-72ca-437e-a75a-83bb4c942de8/volumes" Sep 06 12:32:08.654194 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:08.654136 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62056ff7-6c48-46de-9069-3f6e0e39ad17" path="/var/lib/kubelet/pods/62056ff7-6c48-46de-9069-3f6e0e39ad17/volumes" Sep 06 12:32:12.056118 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:12.056088 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ff8cf183-8496596874-h2rjq"] Sep 06 12:32:12.056770 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:12.056731 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62056ff7-6c48-46de-9069-3f6e0e39ad17" containerName="payload-pre-processing" Sep 06 12:32:12.056911 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:12.056816 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="62056ff7-6c48-46de-9069-3f6e0e39ad17" containerName="payload-pre-processing" Sep 06 12:32:12.056911 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:12.056878 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5abb0af6-72ca-437e-a75a-83bb4c942de8" containerName="maas-api" Sep 06 12:32:12.056911 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:12.056887 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5abb0af6-72ca-437e-a75a-83bb4c942de8" containerName="maas-api" Sep 06 12:32:12.057071 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:12.057018 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5abb0af6-72ca-437e-a75a-83bb4c942de8" containerName="maas-api" Sep 06 12:32:12.057071 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:12.057031 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="62056ff7-6c48-46de-9069-3f6e0e39ad17" containerName="payload-pre-processing" Sep 06 12:32:12.061850 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:12.061828 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff8cf183-8496596874-h2rjq" Sep 06 12:32:12.063991 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:12.063970 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-ff8cf183-serving-cert\"" Sep 06 12:32:12.070632 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:12.070609 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ff8cf183-8496596874-h2rjq"] Sep 06 12:32:12.096189 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:12.096147 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dj8m\" (UniqueName: \"kubernetes.io/projected/95611764-8ebf-4f13-986c-612a8b9ffdf6-kube-api-access-5dj8m\") pod \"maas-api-e2e-derive-ff8cf183-8496596874-h2rjq\" (UID: \"95611764-8ebf-4f13-986c-612a8b9ffdf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff8cf183-8496596874-h2rjq" Sep 06 12:32:12.096329 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:12.096201 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95611764-8ebf-4f13-986c-612a8b9ffdf6-openshift-service-ca\") pod \"maas-api-e2e-derive-ff8cf183-8496596874-h2rjq\" (UID: \"95611764-8ebf-4f13-986c-612a8b9ffdf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff8cf183-8496596874-h2rjq" Sep 06 12:32:12.096329 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:12.096285 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95611764-8ebf-4f13-986c-612a8b9ffdf6-maas-api-tls\") pod \"maas-api-e2e-derive-ff8cf183-8496596874-h2rjq\" (UID: \"95611764-8ebf-4f13-986c-612a8b9ffdf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff8cf183-8496596874-h2rjq" Sep 06 12:32:12.096433 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:12.096345 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95611764-8ebf-4f13-986c-612a8b9ffdf6-metrics-certs\") pod \"maas-api-e2e-derive-ff8cf183-8496596874-h2rjq\" (UID: \"95611764-8ebf-4f13-986c-612a8b9ffdf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff8cf183-8496596874-h2rjq" Sep 06 12:32:12.197116 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:12.197088 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95611764-8ebf-4f13-986c-612a8b9ffdf6-metrics-certs\") pod \"maas-api-e2e-derive-ff8cf183-8496596874-h2rjq\" (UID: \"95611764-8ebf-4f13-986c-612a8b9ffdf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff8cf183-8496596874-h2rjq" Sep 06 12:32:12.197239 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:12.197159 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dj8m\" (UniqueName: \"kubernetes.io/projected/95611764-8ebf-4f13-986c-612a8b9ffdf6-kube-api-access-5dj8m\") pod \"maas-api-e2e-derive-ff8cf183-8496596874-h2rjq\" (UID: \"95611764-8ebf-4f13-986c-612a8b9ffdf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff8cf183-8496596874-h2rjq" Sep 06 12:32:12.197239 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:12.197181 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95611764-8ebf-4f13-986c-612a8b9ffdf6-openshift-service-ca\") pod \"maas-api-e2e-derive-ff8cf183-8496596874-h2rjq\" (UID: \"95611764-8ebf-4f13-986c-612a8b9ffdf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff8cf183-8496596874-h2rjq" Sep 06 12:32:12.197239 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:12.197206 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95611764-8ebf-4f13-986c-612a8b9ffdf6-maas-api-tls\") pod \"maas-api-e2e-derive-ff8cf183-8496596874-h2rjq\" (UID: \"95611764-8ebf-4f13-986c-612a8b9ffdf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff8cf183-8496596874-h2rjq" Sep 06 12:32:12.197916 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:12.197849 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95611764-8ebf-4f13-986c-612a8b9ffdf6-openshift-service-ca\") pod \"maas-api-e2e-derive-ff8cf183-8496596874-h2rjq\" (UID: \"95611764-8ebf-4f13-986c-612a8b9ffdf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff8cf183-8496596874-h2rjq" Sep 06 12:32:12.199783 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:12.199728 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95611764-8ebf-4f13-986c-612a8b9ffdf6-maas-api-tls\") pod \"maas-api-e2e-derive-ff8cf183-8496596874-h2rjq\" (UID: \"95611764-8ebf-4f13-986c-612a8b9ffdf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff8cf183-8496596874-h2rjq" Sep 06 12:32:12.199884 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:12.199791 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95611764-8ebf-4f13-986c-612a8b9ffdf6-metrics-certs\") pod \"maas-api-e2e-derive-ff8cf183-8496596874-h2rjq\" (UID: \"95611764-8ebf-4f13-986c-612a8b9ffdf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff8cf183-8496596874-h2rjq" Sep 06 12:32:12.204623 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:12.204583 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dj8m\" (UniqueName: \"kubernetes.io/projected/95611764-8ebf-4f13-986c-612a8b9ffdf6-kube-api-access-5dj8m\") pod \"maas-api-e2e-derive-ff8cf183-8496596874-h2rjq\" (UID: \"95611764-8ebf-4f13-986c-612a8b9ffdf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff8cf183-8496596874-h2rjq" Sep 06 12:32:12.376615 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:12.376549 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff8cf183-8496596874-h2rjq" Sep 06 12:32:12.655015 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:12.654991 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ff8cf183-96c8c6478-tx5ct"] Sep 06 12:32:12.657046 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:32:12.657022 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95611764_8ebf_4f13_986c_612a8b9ffdf6.slice/crio-5c24e30125f2b4c992a8b60bcda31dd2fbae75a55f4139ac3fe8e8aa2d3f822c WatchSource:0}: Error finding container 5c24e30125f2b4c992a8b60bcda31dd2fbae75a55f4139ac3fe8e8aa2d3f822c: Status 404 returned error can't find the container with id 5c24e30125f2b4c992a8b60bcda31dd2fbae75a55f4139ac3fe8e8aa2d3f822c Sep 06 12:32:12.659937 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:12.659917 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ff8cf183-8496596874-h2rjq"] Sep 06 12:32:12.660046 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:12.660022 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ff8cf183-96c8c6478-tx5ct" Sep 06 12:32:12.662369 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:12.662351 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-ff8cf183-dockercfg-g4ml8\"" Sep 06 12:32:12.662453 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:12.662393 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-ff8cf183\"" Sep 06 12:32:12.670488 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:12.670467 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ff8cf183-96c8c6478-tx5ct"] Sep 06 12:32:12.700621 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:12.700598 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0592de14-a464-4ca3-94ae-a2bd0efe3d6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ff8cf183-96c8c6478-tx5ct\" (UID: \"0592de14-a464-4ca3-94ae-a2bd0efe3d6c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ff8cf183-96c8c6478-tx5ct" Sep 06 12:32:12.700728 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:12.700680 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5lxr\" (UniqueName: \"kubernetes.io/projected/0592de14-a464-4ca3-94ae-a2bd0efe3d6c-kube-api-access-v5lxr\") pod \"payload-pre-processing-e2e-derive-ff8cf183-96c8c6478-tx5ct\" (UID: \"0592de14-a464-4ca3-94ae-a2bd0efe3d6c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ff8cf183-96c8c6478-tx5ct" Sep 06 12:32:12.801689 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:12.801652 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0592de14-a464-4ca3-94ae-a2bd0efe3d6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ff8cf183-96c8c6478-tx5ct\" (UID: \"0592de14-a464-4ca3-94ae-a2bd0efe3d6c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ff8cf183-96c8c6478-tx5ct" Sep 06 12:32:12.801881 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:12.801831 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5lxr\" (UniqueName: \"kubernetes.io/projected/0592de14-a464-4ca3-94ae-a2bd0efe3d6c-kube-api-access-v5lxr\") pod \"payload-pre-processing-e2e-derive-ff8cf183-96c8c6478-tx5ct\" (UID: \"0592de14-a464-4ca3-94ae-a2bd0efe3d6c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ff8cf183-96c8c6478-tx5ct" Sep 06 12:32:12.802340 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:12.802319 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0592de14-a464-4ca3-94ae-a2bd0efe3d6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ff8cf183-96c8c6478-tx5ct\" (UID: \"0592de14-a464-4ca3-94ae-a2bd0efe3d6c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ff8cf183-96c8c6478-tx5ct" Sep 06 12:32:12.809466 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:12.809447 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5lxr\" (UniqueName: \"kubernetes.io/projected/0592de14-a464-4ca3-94ae-a2bd0efe3d6c-kube-api-access-v5lxr\") pod \"payload-pre-processing-e2e-derive-ff8cf183-96c8c6478-tx5ct\" (UID: \"0592de14-a464-4ca3-94ae-a2bd0efe3d6c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ff8cf183-96c8c6478-tx5ct" Sep 06 12:32:12.977694 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:12.977633 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ff8cf183-96c8c6478-tx5ct" Sep 06 12:32:13.043012 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:13.042970 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff8cf183-8496596874-h2rjq" event={"ID":"95611764-8ebf-4f13-986c-612a8b9ffdf6","Type":"ContainerStarted","Data":"5c24e30125f2b4c992a8b60bcda31dd2fbae75a55f4139ac3fe8e8aa2d3f822c"} Sep 06 12:32:13.259963 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:13.259846 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ff8cf183-96c8c6478-tx5ct"] Sep 06 12:32:13.263057 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:32:13.263029 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0592de14_a464_4ca3_94ae_a2bd0efe3d6c.slice/crio-9da0db370b0fce3f28fd74fc1be3f40f386e62f2b436db5f09524f380144ca59 WatchSource:0}: Error finding container 9da0db370b0fce3f28fd74fc1be3f40f386e62f2b436db5f09524f380144ca59: Status 404 returned error can't find the container with id 9da0db370b0fce3f28fd74fc1be3f40f386e62f2b436db5f09524f380144ca59 Sep 06 12:32:14.057935 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:14.057897 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ff8cf183-96c8c6478-tx5ct" event={"ID":"0592de14-a464-4ca3-94ae-a2bd0efe3d6c","Type":"ContainerStarted","Data":"1f53d6b9c019d1f7cec08a1dc03f5c1fc27ca60d23c17b9ed1814e00280b7eaf"} Sep 06 12:32:14.058128 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:14.057991 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ff8cf183-96c8c6478-tx5ct" event={"ID":"0592de14-a464-4ca3-94ae-a2bd0efe3d6c","Type":"ContainerStarted","Data":"9da0db370b0fce3f28fd74fc1be3f40f386e62f2b436db5f09524f380144ca59"} Sep 06 12:32:14.058128 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:14.058022 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-ff8cf183-96c8c6478-tx5ct" Sep 06 12:32:14.059523 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:14.059499 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff8cf183-8496596874-h2rjq" event={"ID":"95611764-8ebf-4f13-986c-612a8b9ffdf6","Type":"ContainerStarted","Data":"971e465ce2d0353e196c7916951611c4c5673a640ae1af5651594f3d200ec627"} Sep 06 12:32:14.059656 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:14.059624 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff8cf183-8496596874-h2rjq" Sep 06 12:32:14.082688 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:14.082652 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-ff8cf183-96c8c6478-tx5ct" podStartSLOduration=2.082638473 podStartE2EDuration="2.082638473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:32:14.080503172 +0000 UTC m=+849.936261081" watchObservedRunningTime="2026-09-06 12:32:14.082638473 +0000 UTC m=+849.938396359" Sep 06 12:32:14.099345 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:14.099302 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff8cf183-8496596874-h2rjq" podStartSLOduration=1.7660847 podStartE2EDuration="2.099287809s" totalImagesPullingTime="333.203109ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:32:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:32:14.098468419 +0000 UTC m=+849.954226329" watchObservedRunningTime="2026-09-06 12:32:14.099287809 +0000 UTC m=+849.955045696" Sep 06 12:32:20.070240 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:20.070207 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff8cf183-8496596874-h2rjq" Sep 06 12:32:25.065987 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:25.065957 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-ff8cf183-96c8c6478-tx5ct" Sep 06 12:32:42.300642 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.300602 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ff8cf183-8496596874-h2rjq"] Sep 06 12:32:42.301138 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.300881 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff8cf183-8496596874-h2rjq" podUID="95611764-8ebf-4f13-986c-612a8b9ffdf6" containerName="maas-api" containerID="cri-o://971e465ce2d0353e196c7916951611c4c5673a640ae1af5651594f3d200ec627" gracePeriod=30 Sep 06 12:32:42.365507 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.365474 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ff8cf183-96c8c6478-tx5ct"] Sep 06 12:32:42.365776 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.365714 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-ff8cf183-96c8c6478-tx5ct" podUID="0592de14-a464-4ca3-94ae-a2bd0efe3d6c" containerName="payload-pre-processing" containerID="cri-o://1f53d6b9c019d1f7cec08a1dc03f5c1fc27ca60d23c17b9ed1814e00280b7eaf" gracePeriod=30 Sep 06 12:32:42.469781 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.459917 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-0e6ab1-b887684c4-vjk27"] Sep 06 12:32:42.469781 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.460165 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e6ab1-b887684c4-vjk27" podUID="6a25f137-55e3-49d2-bfa3-4781e20b6bf0" containerName="maas-api" containerID="cri-o://cb9cbde7ffb936fddd4f03701bd7e840532ed00fbdd8ff2750ac779a891af744" gracePeriod=30 Sep 06 12:32:42.488252 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.473567 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e6ab1-b887684c4-vjk27" podUID="6a25f137-55e3-49d2-bfa3-4781e20b6bf0" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.45:8443/health\": dial tcp 10.134.0.45:8443: connect: connection refused" Sep 06 12:32:42.677161 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.677004 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff8cf183-8496596874-h2rjq" Sep 06 12:32:42.707610 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.707502 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ff8cf183-96c8c6478-tx5ct" Sep 06 12:32:42.754531 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.754502 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e6ab1-b887684c4-vjk27" Sep 06 12:32:42.766463 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.766440 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95611764-8ebf-4f13-986c-612a8b9ffdf6-maas-api-tls\") pod \"95611764-8ebf-4f13-986c-612a8b9ffdf6\" (UID: \"95611764-8ebf-4f13-986c-612a8b9ffdf6\") " Sep 06 12:32:42.766571 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.766497 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5lxr\" (UniqueName: \"kubernetes.io/projected/0592de14-a464-4ca3-94ae-a2bd0efe3d6c-kube-api-access-v5lxr\") pod \"0592de14-a464-4ca3-94ae-a2bd0efe3d6c\" (UID: \"0592de14-a464-4ca3-94ae-a2bd0efe3d6c\") " Sep 06 12:32:42.766571 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.766560 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95611764-8ebf-4f13-986c-612a8b9ffdf6-metrics-certs\") pod \"95611764-8ebf-4f13-986c-612a8b9ffdf6\" (UID: \"95611764-8ebf-4f13-986c-612a8b9ffdf6\") " Sep 06 12:32:42.766684 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.766577 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0592de14-a464-4ca3-94ae-a2bd0efe3d6c-plugins-config-volume\") pod \"0592de14-a464-4ca3-94ae-a2bd0efe3d6c\" (UID: \"0592de14-a464-4ca3-94ae-a2bd0efe3d6c\") " Sep 06 12:32:42.766684 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.766612 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95611764-8ebf-4f13-986c-612a8b9ffdf6-openshift-service-ca\") pod \"95611764-8ebf-4f13-986c-612a8b9ffdf6\" (UID: \"95611764-8ebf-4f13-986c-612a8b9ffdf6\") " Sep 06 12:32:42.766684 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.766672 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dj8m\" (UniqueName: \"kubernetes.io/projected/95611764-8ebf-4f13-986c-612a8b9ffdf6-kube-api-access-5dj8m\") pod \"95611764-8ebf-4f13-986c-612a8b9ffdf6\" (UID: \"95611764-8ebf-4f13-986c-612a8b9ffdf6\") " Sep 06 12:32:42.767033 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.766996 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95611764-8ebf-4f13-986c-612a8b9ffdf6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "95611764-8ebf-4f13-986c-612a8b9ffdf6" (UID: "95611764-8ebf-4f13-986c-612a8b9ffdf6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:32:42.767121 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.767012 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0592de14-a464-4ca3-94ae-a2bd0efe3d6c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0592de14-a464-4ca3-94ae-a2bd0efe3d6c" (UID: "0592de14-a464-4ca3-94ae-a2bd0efe3d6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:32:42.767406 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.767380 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0592de14-a464-4ca3-94ae-a2bd0efe3d6c-plugins-config-volume\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:32:42.767510 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.767409 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95611764-8ebf-4f13-986c-612a8b9ffdf6-openshift-service-ca\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:32:42.768919 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.768894 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95611764-8ebf-4f13-986c-612a8b9ffdf6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "95611764-8ebf-4f13-986c-612a8b9ffdf6" (UID: "95611764-8ebf-4f13-986c-612a8b9ffdf6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 12:32:42.768919 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.768908 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0592de14-a464-4ca3-94ae-a2bd0efe3d6c-kube-api-access-v5lxr" (OuterVolumeSpecName: "kube-api-access-v5lxr") pod "0592de14-a464-4ca3-94ae-a2bd0efe3d6c" (UID: "0592de14-a464-4ca3-94ae-a2bd0efe3d6c"). InnerVolumeSpecName "kube-api-access-v5lxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:32:42.769353 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.769336 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95611764-8ebf-4f13-986c-612a8b9ffdf6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "95611764-8ebf-4f13-986c-612a8b9ffdf6" (UID: "95611764-8ebf-4f13-986c-612a8b9ffdf6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 12:32:42.769353 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.769340 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95611764-8ebf-4f13-986c-612a8b9ffdf6-kube-api-access-5dj8m" (OuterVolumeSpecName: "kube-api-access-5dj8m") pod "95611764-8ebf-4f13-986c-612a8b9ffdf6" (UID: "95611764-8ebf-4f13-986c-612a8b9ffdf6"). InnerVolumeSpecName "kube-api-access-5dj8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:32:42.868004 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.867932 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a25f137-55e3-49d2-bfa3-4781e20b6bf0-metrics-certs\") pod \"6a25f137-55e3-49d2-bfa3-4781e20b6bf0\" (UID: \"6a25f137-55e3-49d2-bfa3-4781e20b6bf0\") " Sep 06 12:32:42.868004 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.867991 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a25f137-55e3-49d2-bfa3-4781e20b6bf0-openshift-service-ca\") pod \"6a25f137-55e3-49d2-bfa3-4781e20b6bf0\" (UID: \"6a25f137-55e3-49d2-bfa3-4781e20b6bf0\") " Sep 06 12:32:42.868202 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.868023 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a25f137-55e3-49d2-bfa3-4781e20b6bf0-maas-api-tls\") pod \"6a25f137-55e3-49d2-bfa3-4781e20b6bf0\" (UID: \"6a25f137-55e3-49d2-bfa3-4781e20b6bf0\") " Sep 06 12:32:42.868202 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.868093 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4kd9\" (UniqueName: \"kubernetes.io/projected/6a25f137-55e3-49d2-bfa3-4781e20b6bf0-kube-api-access-d4kd9\") pod \"6a25f137-55e3-49d2-bfa3-4781e20b6bf0\" (UID: \"6a25f137-55e3-49d2-bfa3-4781e20b6bf0\") " Sep 06 12:32:42.868435 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.868402 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a25f137-55e3-49d2-bfa3-4781e20b6bf0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6a25f137-55e3-49d2-bfa3-4781e20b6bf0" (UID: "6a25f137-55e3-49d2-bfa3-4781e20b6bf0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:32:42.868435 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.868430 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dj8m\" (UniqueName: \"kubernetes.io/projected/95611764-8ebf-4f13-986c-612a8b9ffdf6-kube-api-access-5dj8m\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:32:42.868590 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.868449 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95611764-8ebf-4f13-986c-612a8b9ffdf6-maas-api-tls\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:32:42.868590 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.868461 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5lxr\" (UniqueName: \"kubernetes.io/projected/0592de14-a464-4ca3-94ae-a2bd0efe3d6c-kube-api-access-v5lxr\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:32:42.868590 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.868469 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95611764-8ebf-4f13-986c-612a8b9ffdf6-metrics-certs\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:32:42.870079 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.870049 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a25f137-55e3-49d2-bfa3-4781e20b6bf0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6a25f137-55e3-49d2-bfa3-4781e20b6bf0" (UID: "6a25f137-55e3-49d2-bfa3-4781e20b6bf0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 12:32:42.870198 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.870170 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a25f137-55e3-49d2-bfa3-4781e20b6bf0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6a25f137-55e3-49d2-bfa3-4781e20b6bf0" (UID: "6a25f137-55e3-49d2-bfa3-4781e20b6bf0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 12:32:42.870241 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.870189 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a25f137-55e3-49d2-bfa3-4781e20b6bf0-kube-api-access-d4kd9" (OuterVolumeSpecName: "kube-api-access-d4kd9") pod "6a25f137-55e3-49d2-bfa3-4781e20b6bf0" (UID: "6a25f137-55e3-49d2-bfa3-4781e20b6bf0"). InnerVolumeSpecName "kube-api-access-d4kd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:32:42.969231 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.969204 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a25f137-55e3-49d2-bfa3-4781e20b6bf0-metrics-certs\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:32:42.969231 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.969228 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a25f137-55e3-49d2-bfa3-4781e20b6bf0-openshift-service-ca\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:32:42.969384 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.969242 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a25f137-55e3-49d2-bfa3-4781e20b6bf0-maas-api-tls\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:32:42.969384 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:42.969252 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4kd9\" (UniqueName: \"kubernetes.io/projected/6a25f137-55e3-49d2-bfa3-4781e20b6bf0-kube-api-access-d4kd9\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:32:43.226449 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:43.226422 2576 generic.go:356] "Generic (PLEG): container finished" podID="0592de14-a464-4ca3-94ae-a2bd0efe3d6c" containerID="1f53d6b9c019d1f7cec08a1dc03f5c1fc27ca60d23c17b9ed1814e00280b7eaf" exitCode=0 Sep 06 12:32:43.226616 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:43.226478 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ff8cf183-96c8c6478-tx5ct" Sep 06 12:32:43.226616 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:43.226501 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ff8cf183-96c8c6478-tx5ct" event={"ID":"0592de14-a464-4ca3-94ae-a2bd0efe3d6c","Type":"ContainerDied","Data":"1f53d6b9c019d1f7cec08a1dc03f5c1fc27ca60d23c17b9ed1814e00280b7eaf"} Sep 06 12:32:43.226616 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:43.226555 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ff8cf183-96c8c6478-tx5ct" event={"ID":"0592de14-a464-4ca3-94ae-a2bd0efe3d6c","Type":"ContainerDied","Data":"9da0db370b0fce3f28fd74fc1be3f40f386e62f2b436db5f09524f380144ca59"} Sep 06 12:32:43.226616 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:43.226583 2576 scope.go:117] "RemoveContainer" containerID="1f53d6b9c019d1f7cec08a1dc03f5c1fc27ca60d23c17b9ed1814e00280b7eaf" Sep 06 12:32:43.228024 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:43.228002 2576 generic.go:356] "Generic (PLEG): container finished" podID="6a25f137-55e3-49d2-bfa3-4781e20b6bf0" containerID="cb9cbde7ffb936fddd4f03701bd7e840532ed00fbdd8ff2750ac779a891af744" exitCode=0 Sep 06 12:32:43.228160 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:43.228055 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e6ab1-b887684c4-vjk27" Sep 06 12:32:43.228160 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:43.228086 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e6ab1-b887684c4-vjk27" event={"ID":"6a25f137-55e3-49d2-bfa3-4781e20b6bf0","Type":"ContainerDied","Data":"cb9cbde7ffb936fddd4f03701bd7e840532ed00fbdd8ff2750ac779a891af744"} Sep 06 12:32:43.228160 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:43.228143 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e6ab1-b887684c4-vjk27" event={"ID":"6a25f137-55e3-49d2-bfa3-4781e20b6bf0","Type":"ContainerDied","Data":"c89b3e78c7738c2cfdfde903cae545f510d4a40fc8d38320da417b548e39291d"} Sep 06 12:32:43.233313 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:43.230425 2576 generic.go:356] "Generic (PLEG): container finished" podID="95611764-8ebf-4f13-986c-612a8b9ffdf6" containerID="971e465ce2d0353e196c7916951611c4c5673a640ae1af5651594f3d200ec627" exitCode=0 Sep 06 12:32:43.233313 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:43.230467 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff8cf183-8496596874-h2rjq" Sep 06 12:32:43.233313 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:43.230467 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff8cf183-8496596874-h2rjq" event={"ID":"95611764-8ebf-4f13-986c-612a8b9ffdf6","Type":"ContainerDied","Data":"971e465ce2d0353e196c7916951611c4c5673a640ae1af5651594f3d200ec627"} Sep 06 12:32:43.233313 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:43.230535 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff8cf183-8496596874-h2rjq" event={"ID":"95611764-8ebf-4f13-986c-612a8b9ffdf6","Type":"ContainerDied","Data":"5c24e30125f2b4c992a8b60bcda31dd2fbae75a55f4139ac3fe8e8aa2d3f822c"} Sep 06 12:32:43.238338 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:43.238320 2576 scope.go:117] "RemoveContainer" containerID="1f53d6b9c019d1f7cec08a1dc03f5c1fc27ca60d23c17b9ed1814e00280b7eaf" Sep 06 12:32:43.238650 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:32:43.238627 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f53d6b9c019d1f7cec08a1dc03f5c1fc27ca60d23c17b9ed1814e00280b7eaf\": container with ID starting with 1f53d6b9c019d1f7cec08a1dc03f5c1fc27ca60d23c17b9ed1814e00280b7eaf not found: ID does not exist" containerID="1f53d6b9c019d1f7cec08a1dc03f5c1fc27ca60d23c17b9ed1814e00280b7eaf" Sep 06 12:32:43.238793 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:43.238654 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f53d6b9c019d1f7cec08a1dc03f5c1fc27ca60d23c17b9ed1814e00280b7eaf"} err="failed to get container status \"1f53d6b9c019d1f7cec08a1dc03f5c1fc27ca60d23c17b9ed1814e00280b7eaf\": rpc error: code = NotFound desc = could not find container \"1f53d6b9c019d1f7cec08a1dc03f5c1fc27ca60d23c17b9ed1814e00280b7eaf\": container with ID starting with 1f53d6b9c019d1f7cec08a1dc03f5c1fc27ca60d23c17b9ed1814e00280b7eaf not found: ID does not exist" Sep 06 12:32:43.238793 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:43.238669 2576 scope.go:117] "RemoveContainer" containerID="cb9cbde7ffb936fddd4f03701bd7e840532ed00fbdd8ff2750ac779a891af744" Sep 06 12:32:43.247304 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:43.247290 2576 scope.go:117] "RemoveContainer" containerID="cb9cbde7ffb936fddd4f03701bd7e840532ed00fbdd8ff2750ac779a891af744" Sep 06 12:32:43.247568 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:32:43.247553 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb9cbde7ffb936fddd4f03701bd7e840532ed00fbdd8ff2750ac779a891af744\": container with ID starting with cb9cbde7ffb936fddd4f03701bd7e840532ed00fbdd8ff2750ac779a891af744 not found: ID does not exist" containerID="cb9cbde7ffb936fddd4f03701bd7e840532ed00fbdd8ff2750ac779a891af744" Sep 06 12:32:43.247607 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:43.247575 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb9cbde7ffb936fddd4f03701bd7e840532ed00fbdd8ff2750ac779a891af744"} err="failed to get container status \"cb9cbde7ffb936fddd4f03701bd7e840532ed00fbdd8ff2750ac779a891af744\": rpc error: code = NotFound desc = could not find container \"cb9cbde7ffb936fddd4f03701bd7e840532ed00fbdd8ff2750ac779a891af744\": container with ID starting with cb9cbde7ffb936fddd4f03701bd7e840532ed00fbdd8ff2750ac779a891af744 not found: ID does not exist" Sep 06 12:32:43.247607 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:43.247591 2576 scope.go:117] "RemoveContainer" containerID="971e465ce2d0353e196c7916951611c4c5673a640ae1af5651594f3d200ec627" Sep 06 12:32:43.255159 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:43.255140 2576 scope.go:117] "RemoveContainer" containerID="971e465ce2d0353e196c7916951611c4c5673a640ae1af5651594f3d200ec627" Sep 06 12:32:43.255403 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:32:43.255387 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"971e465ce2d0353e196c7916951611c4c5673a640ae1af5651594f3d200ec627\": container with ID starting with 971e465ce2d0353e196c7916951611c4c5673a640ae1af5651594f3d200ec627 not found: ID does not exist" containerID="971e465ce2d0353e196c7916951611c4c5673a640ae1af5651594f3d200ec627" Sep 06 12:32:43.255474 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:43.255410 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"971e465ce2d0353e196c7916951611c4c5673a640ae1af5651594f3d200ec627"} err="failed to get container status \"971e465ce2d0353e196c7916951611c4c5673a640ae1af5651594f3d200ec627\": rpc error: code = NotFound desc = could not find container \"971e465ce2d0353e196c7916951611c4c5673a640ae1af5651594f3d200ec627\": container with ID starting with 971e465ce2d0353e196c7916951611c4c5673a640ae1af5651594f3d200ec627 not found: ID does not exist" Sep 06 12:32:43.259201 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:43.259179 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-0e6ab1-b887684c4-vjk27"] Sep 06 12:32:43.271247 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:43.271227 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-0e6ab1-b887684c4-vjk27"] Sep 06 12:32:43.281416 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:43.281394 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ff8cf183-96c8c6478-tx5ct"] Sep 06 12:32:43.284834 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:43.284815 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ff8cf183-96c8c6478-tx5ct"] Sep 06 12:32:43.294675 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:43.294639 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ff8cf183-8496596874-h2rjq"] Sep 06 12:32:43.298426 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:43.298405 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ff8cf183-8496596874-h2rjq"] Sep 06 12:32:44.653858 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:44.653827 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0592de14-a464-4ca3-94ae-a2bd0efe3d6c" path="/var/lib/kubelet/pods/0592de14-a464-4ca3-94ae-a2bd0efe3d6c/volumes" Sep 06 12:32:44.654222 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:44.654191 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a25f137-55e3-49d2-bfa3-4781e20b6bf0" path="/var/lib/kubelet/pods/6a25f137-55e3-49d2-bfa3-4781e20b6bf0/volumes" Sep 06 12:32:44.654523 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:32:44.654509 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95611764-8ebf-4f13-986c-612a8b9ffdf6" path="/var/lib/kubelet/pods/95611764-8ebf-4f13-986c-612a8b9ffdf6/volumes" Sep 06 12:33:04.626617 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:04.626588 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98vz4_4103c09f-dd43-419d-92e5-2374af5b5e01/ovn-acl-logging/0.log" Sep 06 12:33:04.628051 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:04.628029 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98vz4_4103c09f-dd43-419d-92e5-2374af5b5e01/ovn-acl-logging/0.log" Sep 06 12:33:31.159678 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.159641 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-1655f5-c85b75486-xb5x8"] Sep 06 12:33:31.160364 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.160296 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0592de14-a464-4ca3-94ae-a2bd0efe3d6c" containerName="payload-pre-processing" Sep 06 12:33:31.160364 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.160313 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0592de14-a464-4ca3-94ae-a2bd0efe3d6c" containerName="payload-pre-processing" Sep 06 12:33:31.160364 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.160337 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95611764-8ebf-4f13-986c-612a8b9ffdf6" containerName="maas-api" Sep 06 12:33:31.160364 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.160343 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="95611764-8ebf-4f13-986c-612a8b9ffdf6" containerName="maas-api" Sep 06 12:33:31.160364 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.160352 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a25f137-55e3-49d2-bfa3-4781e20b6bf0" containerName="maas-api" Sep 06 12:33:31.160364 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.160357 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a25f137-55e3-49d2-bfa3-4781e20b6bf0" containerName="maas-api" Sep 06 12:33:31.160654 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.160428 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0592de14-a464-4ca3-94ae-a2bd0efe3d6c" containerName="payload-pre-processing" Sep 06 12:33:31.160654 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.160439 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a25f137-55e3-49d2-bfa3-4781e20b6bf0" containerName="maas-api" Sep 06 12:33:31.160654 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.160449 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="95611764-8ebf-4f13-986c-612a8b9ffdf6" containerName="maas-api" Sep 06 12:33:31.163681 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.163661 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1655f5-c85b75486-xb5x8" Sep 06 12:33:31.165777 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.165733 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-1655f5-serving-cert\"" Sep 06 12:33:31.174913 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.174888 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-1655f5-c85b75486-xb5x8"] Sep 06 12:33:31.285461 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.285437 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28c434ae-619c-4d1d-ac53-74a62d401b76-maas-api-tls\") pod \"maas-api-e2e-explicit-1655f5-c85b75486-xb5x8\" (UID: \"28c434ae-619c-4d1d-ac53-74a62d401b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1655f5-c85b75486-xb5x8" Sep 06 12:33:31.285616 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.285511 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95gpd\" (UniqueName: \"kubernetes.io/projected/28c434ae-619c-4d1d-ac53-74a62d401b76-kube-api-access-95gpd\") pod \"maas-api-e2e-explicit-1655f5-c85b75486-xb5x8\" (UID: \"28c434ae-619c-4d1d-ac53-74a62d401b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1655f5-c85b75486-xb5x8" Sep 06 12:33:31.285616 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.285545 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28c434ae-619c-4d1d-ac53-74a62d401b76-metrics-certs\") pod \"maas-api-e2e-explicit-1655f5-c85b75486-xb5x8\" (UID: \"28c434ae-619c-4d1d-ac53-74a62d401b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1655f5-c85b75486-xb5x8" Sep 06 12:33:31.285616 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.285584 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28c434ae-619c-4d1d-ac53-74a62d401b76-openshift-service-ca\") pod \"maas-api-e2e-explicit-1655f5-c85b75486-xb5x8\" (UID: \"28c434ae-619c-4d1d-ac53-74a62d401b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1655f5-c85b75486-xb5x8" Sep 06 12:33:31.386001 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.385978 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95gpd\" (UniqueName: \"kubernetes.io/projected/28c434ae-619c-4d1d-ac53-74a62d401b76-kube-api-access-95gpd\") pod \"maas-api-e2e-explicit-1655f5-c85b75486-xb5x8\" (UID: \"28c434ae-619c-4d1d-ac53-74a62d401b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1655f5-c85b75486-xb5x8" Sep 06 12:33:31.386119 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.386012 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28c434ae-619c-4d1d-ac53-74a62d401b76-metrics-certs\") pod \"maas-api-e2e-explicit-1655f5-c85b75486-xb5x8\" (UID: \"28c434ae-619c-4d1d-ac53-74a62d401b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1655f5-c85b75486-xb5x8" Sep 06 12:33:31.386119 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.386052 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28c434ae-619c-4d1d-ac53-74a62d401b76-openshift-service-ca\") pod \"maas-api-e2e-explicit-1655f5-c85b75486-xb5x8\" (UID: \"28c434ae-619c-4d1d-ac53-74a62d401b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1655f5-c85b75486-xb5x8" Sep 06 12:33:31.386119 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.386077 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28c434ae-619c-4d1d-ac53-74a62d401b76-maas-api-tls\") pod \"maas-api-e2e-explicit-1655f5-c85b75486-xb5x8\" (UID: \"28c434ae-619c-4d1d-ac53-74a62d401b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1655f5-c85b75486-xb5x8" Sep 06 12:33:31.386662 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.386637 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28c434ae-619c-4d1d-ac53-74a62d401b76-openshift-service-ca\") pod \"maas-api-e2e-explicit-1655f5-c85b75486-xb5x8\" (UID: \"28c434ae-619c-4d1d-ac53-74a62d401b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1655f5-c85b75486-xb5x8" Sep 06 12:33:31.388632 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.388612 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28c434ae-619c-4d1d-ac53-74a62d401b76-maas-api-tls\") pod \"maas-api-e2e-explicit-1655f5-c85b75486-xb5x8\" (UID: \"28c434ae-619c-4d1d-ac53-74a62d401b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1655f5-c85b75486-xb5x8" Sep 06 12:33:31.388832 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.388813 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28c434ae-619c-4d1d-ac53-74a62d401b76-metrics-certs\") pod \"maas-api-e2e-explicit-1655f5-c85b75486-xb5x8\" (UID: \"28c434ae-619c-4d1d-ac53-74a62d401b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1655f5-c85b75486-xb5x8" Sep 06 12:33:31.394694 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.394670 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95gpd\" (UniqueName: \"kubernetes.io/projected/28c434ae-619c-4d1d-ac53-74a62d401b76-kube-api-access-95gpd\") pod \"maas-api-e2e-explicit-1655f5-c85b75486-xb5x8\" (UID: \"28c434ae-619c-4d1d-ac53-74a62d401b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1655f5-c85b75486-xb5x8" Sep 06 12:33:31.476912 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.476853 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1655f5-c85b75486-xb5x8" Sep 06 12:33:31.753962 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.753936 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-1655f5-c85b75486-xb5x8"] Sep 06 12:33:31.756689 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:33:31.756661 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28c434ae_619c_4d1d_ac53_74a62d401b76.slice/crio-547cff9caf1342d39b3b93f2449c1c31cc261e4469fa796bf9d0049376640a0a WatchSource:0}: Error finding container 547cff9caf1342d39b3b93f2449c1c31cc261e4469fa796bf9d0049376640a0a: Status 404 returned error can't find the container with id 547cff9caf1342d39b3b93f2449c1c31cc261e4469fa796bf9d0049376640a0a Sep 06 12:33:31.792973 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.792945 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-1655f5-5657b449b6-nq9vn"] Sep 06 12:33:31.797662 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.797644 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-1655f5-5657b449b6-nq9vn" Sep 06 12:33:31.800003 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.799980 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-1655f5\"" Sep 06 12:33:31.800118 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.800085 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-1655f5-dockercfg-6fntt\"" Sep 06 12:33:31.806452 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.806430 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-1655f5-5657b449b6-nq9vn"] Sep 06 12:33:31.835346 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.835323 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-1655f5-5d49765fb9-s5mnp"] Sep 06 12:33:31.838796 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.838780 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-1655f5-5d49765fb9-s5mnp" Sep 06 12:33:31.853534 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.853514 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-1655f5-5d49765fb9-s5mnp"] Sep 06 12:33:31.891697 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.891661 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwrpw\" (UniqueName: \"kubernetes.io/projected/e2c695f5-d7bf-40a8-8ced-67cdf9a5b395-kube-api-access-qwrpw\") pod \"payload-pre-processing-e2e-explicit-1655f5-5657b449b6-nq9vn\" (UID: \"e2c695f5-d7bf-40a8-8ced-67cdf9a5b395\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-1655f5-5657b449b6-nq9vn" Sep 06 12:33:31.891901 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.891819 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2c695f5-d7bf-40a8-8ced-67cdf9a5b395-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-1655f5-5657b449b6-nq9vn\" (UID: \"e2c695f5-d7bf-40a8-8ced-67cdf9a5b395\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-1655f5-5657b449b6-nq9vn" Sep 06 12:33:31.891901 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.891860 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/477e5cf8-391a-426e-ae31-7ae89309647e-plugins-config-volume\") pod \"payload-processing-e2e-explicit-1655f5-5d49765fb9-s5mnp\" (UID: \"477e5cf8-391a-426e-ae31-7ae89309647e\") " pod="openshift-ingress/payload-processing-e2e-explicit-1655f5-5d49765fb9-s5mnp" Sep 06 12:33:31.891901 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.891889 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-757rx\" (UniqueName: \"kubernetes.io/projected/477e5cf8-391a-426e-ae31-7ae89309647e-kube-api-access-757rx\") pod \"payload-processing-e2e-explicit-1655f5-5d49765fb9-s5mnp\" (UID: \"477e5cf8-391a-426e-ae31-7ae89309647e\") " pod="openshift-ingress/payload-processing-e2e-explicit-1655f5-5d49765fb9-s5mnp" Sep 06 12:33:31.992402 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.992371 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2c695f5-d7bf-40a8-8ced-67cdf9a5b395-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-1655f5-5657b449b6-nq9vn\" (UID: \"e2c695f5-d7bf-40a8-8ced-67cdf9a5b395\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-1655f5-5657b449b6-nq9vn" Sep 06 12:33:31.992544 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.992411 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/477e5cf8-391a-426e-ae31-7ae89309647e-plugins-config-volume\") pod \"payload-processing-e2e-explicit-1655f5-5d49765fb9-s5mnp\" (UID: \"477e5cf8-391a-426e-ae31-7ae89309647e\") " pod="openshift-ingress/payload-processing-e2e-explicit-1655f5-5d49765fb9-s5mnp" Sep 06 12:33:31.992544 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.992432 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-757rx\" (UniqueName: \"kubernetes.io/projected/477e5cf8-391a-426e-ae31-7ae89309647e-kube-api-access-757rx\") pod \"payload-processing-e2e-explicit-1655f5-5d49765fb9-s5mnp\" (UID: \"477e5cf8-391a-426e-ae31-7ae89309647e\") " pod="openshift-ingress/payload-processing-e2e-explicit-1655f5-5d49765fb9-s5mnp" Sep 06 12:33:31.992544 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.992488 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwrpw\" (UniqueName: \"kubernetes.io/projected/e2c695f5-d7bf-40a8-8ced-67cdf9a5b395-kube-api-access-qwrpw\") pod \"payload-pre-processing-e2e-explicit-1655f5-5657b449b6-nq9vn\" (UID: \"e2c695f5-d7bf-40a8-8ced-67cdf9a5b395\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-1655f5-5657b449b6-nq9vn" Sep 06 12:33:31.993075 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.993058 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2c695f5-d7bf-40a8-8ced-67cdf9a5b395-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-1655f5-5657b449b6-nq9vn\" (UID: \"e2c695f5-d7bf-40a8-8ced-67cdf9a5b395\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-1655f5-5657b449b6-nq9vn" Sep 06 12:33:31.993161 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:31.993144 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/477e5cf8-391a-426e-ae31-7ae89309647e-plugins-config-volume\") pod \"payload-processing-e2e-explicit-1655f5-5d49765fb9-s5mnp\" (UID: \"477e5cf8-391a-426e-ae31-7ae89309647e\") " pod="openshift-ingress/payload-processing-e2e-explicit-1655f5-5d49765fb9-s5mnp" Sep 06 12:33:32.001186 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:32.001156 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwrpw\" (UniqueName: \"kubernetes.io/projected/e2c695f5-d7bf-40a8-8ced-67cdf9a5b395-kube-api-access-qwrpw\") pod \"payload-pre-processing-e2e-explicit-1655f5-5657b449b6-nq9vn\" (UID: \"e2c695f5-d7bf-40a8-8ced-67cdf9a5b395\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-1655f5-5657b449b6-nq9vn" Sep 06 12:33:32.001295 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:32.001278 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-757rx\" (UniqueName: \"kubernetes.io/projected/477e5cf8-391a-426e-ae31-7ae89309647e-kube-api-access-757rx\") pod \"payload-processing-e2e-explicit-1655f5-5d49765fb9-s5mnp\" (UID: \"477e5cf8-391a-426e-ae31-7ae89309647e\") " pod="openshift-ingress/payload-processing-e2e-explicit-1655f5-5d49765fb9-s5mnp" Sep 06 12:33:32.109867 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:32.109791 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-1655f5-5657b449b6-nq9vn" Sep 06 12:33:32.147986 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:32.147952 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-1655f5-5d49765fb9-s5mnp" Sep 06 12:33:32.387944 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:32.387917 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-1655f5-5657b449b6-nq9vn"] Sep 06 12:33:32.389137 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:33:32.389115 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2c695f5_d7bf_40a8_8ced_67cdf9a5b395.slice/crio-f7628d41c5fd76c3a7f5a2fabb07b5b9846a4b7ceadb3e71f032678ec1ba1709 WatchSource:0}: Error finding container f7628d41c5fd76c3a7f5a2fabb07b5b9846a4b7ceadb3e71f032678ec1ba1709: Status 404 returned error can't find the container with id f7628d41c5fd76c3a7f5a2fabb07b5b9846a4b7ceadb3e71f032678ec1ba1709 Sep 06 12:33:32.436735 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:32.436705 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-1655f5-5d49765fb9-s5mnp"] Sep 06 12:33:32.442924 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:33:32.442896 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod477e5cf8_391a_426e_ae31_7ae89309647e.slice/crio-2511842b0003fef230fea620fce8b841cd16bee68084f8aaf3e8c640f41a7637 WatchSource:0}: Error finding container 2511842b0003fef230fea620fce8b841cd16bee68084f8aaf3e8c640f41a7637: Status 404 returned error can't find the container with id 2511842b0003fef230fea620fce8b841cd16bee68084f8aaf3e8c640f41a7637 Sep 06 12:33:32.507523 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:32.507426 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-1655f5-5657b449b6-nq9vn" event={"ID":"e2c695f5-d7bf-40a8-8ced-67cdf9a5b395","Type":"ContainerStarted","Data":"03428ee591c2c177389349baaa2e3f28257d4c56ad10fe0a25205cba63e06c43"} Sep 06 12:33:32.507523 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:32.507498 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-1655f5-5657b449b6-nq9vn" event={"ID":"e2c695f5-d7bf-40a8-8ced-67cdf9a5b395","Type":"ContainerStarted","Data":"f7628d41c5fd76c3a7f5a2fabb07b5b9846a4b7ceadb3e71f032678ec1ba1709"} Sep 06 12:33:32.507712 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:32.507691 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-1655f5-5657b449b6-nq9vn" Sep 06 12:33:32.508969 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:32.508939 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1655f5-c85b75486-xb5x8" event={"ID":"28c434ae-619c-4d1d-ac53-74a62d401b76","Type":"ContainerStarted","Data":"547cff9caf1342d39b3b93f2449c1c31cc261e4469fa796bf9d0049376640a0a"} Sep 06 12:33:32.510545 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:32.510522 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-1655f5-5d49765fb9-s5mnp" event={"ID":"477e5cf8-391a-426e-ae31-7ae89309647e","Type":"ContainerStarted","Data":"2511842b0003fef230fea620fce8b841cd16bee68084f8aaf3e8c640f41a7637"} Sep 06 12:33:32.536947 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:32.536893 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-1655f5-5657b449b6-nq9vn" podStartSLOduration=1.536873428 podStartE2EDuration="1.536873428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:33:32.533984097 +0000 UTC m=+928.389742059" watchObservedRunningTime="2026-09-06 12:33:32.536873428 +0000 UTC m=+928.392631316" Sep 06 12:33:33.518035 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:33.517997 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-1655f5-5d49765fb9-s5mnp" event={"ID":"477e5cf8-391a-426e-ae31-7ae89309647e","Type":"ContainerStarted","Data":"aa8d2b1a550d2a19d16f1f17a9d211942ee7d375469013faa1635dc3eac6c943"} Sep 06 12:33:33.518491 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:33.518069 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-1655f5-5d49765fb9-s5mnp" Sep 06 12:33:33.519678 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:33.519652 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1655f5-c85b75486-xb5x8" event={"ID":"28c434ae-619c-4d1d-ac53-74a62d401b76","Type":"ContainerStarted","Data":"bbc2a67c28430ac521cd97cfdb7bebe668b0a475ad69fd0aee3653d2b61f958f"} Sep 06 12:33:33.519922 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:33.519904 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1655f5-c85b75486-xb5x8" Sep 06 12:33:33.534042 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:33.533999 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-1655f5-5d49765fb9-s5mnp" podStartSLOduration=2.533984585 podStartE2EDuration="2.533984585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:33:33.533058856 +0000 UTC m=+929.388816781" watchObservedRunningTime="2026-09-06 12:33:33.533984585 +0000 UTC m=+929.389742507" Sep 06 12:33:33.554041 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:33.554003 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1655f5-c85b75486-xb5x8" podStartSLOduration=1.867760377 podStartE2EDuration="2.553991086s" totalImagesPullingTime="686.230709ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:33:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:33:33.553293414 +0000 UTC m=+929.409051332" watchObservedRunningTime="2026-09-06 12:33:33.553991086 +0000 UTC m=+929.409748973" Sep 06 12:33:39.534017 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:39.533985 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1655f5-c85b75486-xb5x8" Sep 06 12:33:43.521933 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:43.521895 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-1655f5-5657b449b6-nq9vn" Sep 06 12:33:44.530962 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:33:44.530929 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-1655f5-5d49765fb9-s5mnp" Sep 06 12:34:31.988303 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:31.988102 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-1655f5-c85b75486-xb5x8"] Sep 06 12:34:31.989605 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:31.988920 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1655f5-c85b75486-xb5x8" podUID="28c434ae-619c-4d1d-ac53-74a62d401b76" containerName="maas-api" containerID="cri-o://bbc2a67c28430ac521cd97cfdb7bebe668b0a475ad69fd0aee3653d2b61f958f" gracePeriod=30 Sep 06 12:34:32.036310 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:34:32.036283 2576 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28c434ae_619c_4d1d_ac53_74a62d401b76.slice/crio-conmon-bbc2a67c28430ac521cd97cfdb7bebe668b0a475ad69fd0aee3653d2b61f958f.scope\": RecentStats: unable to find data in memory cache]" Sep 06 12:34:32.112655 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.111993 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-1655f5-5d49765fb9-s5mnp"] Sep 06 12:34:32.112655 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.112449 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-1655f5-5d49765fb9-s5mnp" podUID="477e5cf8-391a-426e-ae31-7ae89309647e" containerName="payload-processing" containerID="cri-o://aa8d2b1a550d2a19d16f1f17a9d211942ee7d375469013faa1635dc3eac6c943" gracePeriod=30 Sep 06 12:34:32.115401 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.115367 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-1655f5-5657b449b6-nq9vn"] Sep 06 12:34:32.115623 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.115581 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-1655f5-5657b449b6-nq9vn" podUID="e2c695f5-d7bf-40a8-8ced-67cdf9a5b395" containerName="payload-pre-processing" containerID="cri-o://03428ee591c2c177389349baaa2e3f28257d4c56ad10fe0a25205cba63e06c43" gracePeriod=30 Sep 06 12:34:32.292904 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.292315 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1655f5-c85b75486-xb5x8" Sep 06 12:34:32.389260 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.389235 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-1655f5-5d49765fb9-s5mnp" Sep 06 12:34:32.427240 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.427201 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95gpd\" (UniqueName: \"kubernetes.io/projected/28c434ae-619c-4d1d-ac53-74a62d401b76-kube-api-access-95gpd\") pod \"28c434ae-619c-4d1d-ac53-74a62d401b76\" (UID: \"28c434ae-619c-4d1d-ac53-74a62d401b76\") " Sep 06 12:34:32.427473 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.427295 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28c434ae-619c-4d1d-ac53-74a62d401b76-maas-api-tls\") pod \"28c434ae-619c-4d1d-ac53-74a62d401b76\" (UID: \"28c434ae-619c-4d1d-ac53-74a62d401b76\") " Sep 06 12:34:32.427473 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.427342 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28c434ae-619c-4d1d-ac53-74a62d401b76-metrics-certs\") pod \"28c434ae-619c-4d1d-ac53-74a62d401b76\" (UID: \"28c434ae-619c-4d1d-ac53-74a62d401b76\") " Sep 06 12:34:32.427473 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.427442 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28c434ae-619c-4d1d-ac53-74a62d401b76-openshift-service-ca\") pod \"28c434ae-619c-4d1d-ac53-74a62d401b76\" (UID: \"28c434ae-619c-4d1d-ac53-74a62d401b76\") " Sep 06 12:34:32.428947 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.428714 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28c434ae-619c-4d1d-ac53-74a62d401b76-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "28c434ae-619c-4d1d-ac53-74a62d401b76" (UID: "28c434ae-619c-4d1d-ac53-74a62d401b76"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:34:32.429132 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.428954 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-1655f5-5657b449b6-nq9vn" Sep 06 12:34:32.430706 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.430673 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28c434ae-619c-4d1d-ac53-74a62d401b76-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "28c434ae-619c-4d1d-ac53-74a62d401b76" (UID: "28c434ae-619c-4d1d-ac53-74a62d401b76"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 12:34:32.430947 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.430919 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28c434ae-619c-4d1d-ac53-74a62d401b76-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "28c434ae-619c-4d1d-ac53-74a62d401b76" (UID: "28c434ae-619c-4d1d-ac53-74a62d401b76"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 12:34:32.430947 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.430936 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28c434ae-619c-4d1d-ac53-74a62d401b76-kube-api-access-95gpd" (OuterVolumeSpecName: "kube-api-access-95gpd") pod "28c434ae-619c-4d1d-ac53-74a62d401b76" (UID: "28c434ae-619c-4d1d-ac53-74a62d401b76"). InnerVolumeSpecName "kube-api-access-95gpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:34:32.529055 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.528978 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/477e5cf8-391a-426e-ae31-7ae89309647e-plugins-config-volume\") pod \"477e5cf8-391a-426e-ae31-7ae89309647e\" (UID: \"477e5cf8-391a-426e-ae31-7ae89309647e\") " Sep 06 12:34:32.529055 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.529024 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2c695f5-d7bf-40a8-8ced-67cdf9a5b395-plugins-config-volume\") pod \"e2c695f5-d7bf-40a8-8ced-67cdf9a5b395\" (UID: \"e2c695f5-d7bf-40a8-8ced-67cdf9a5b395\") " Sep 06 12:34:32.529217 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.529066 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-757rx\" (UniqueName: \"kubernetes.io/projected/477e5cf8-391a-426e-ae31-7ae89309647e-kube-api-access-757rx\") pod \"477e5cf8-391a-426e-ae31-7ae89309647e\" (UID: \"477e5cf8-391a-426e-ae31-7ae89309647e\") " Sep 06 12:34:32.529217 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.529125 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwrpw\" (UniqueName: \"kubernetes.io/projected/e2c695f5-d7bf-40a8-8ced-67cdf9a5b395-kube-api-access-qwrpw\") pod \"e2c695f5-d7bf-40a8-8ced-67cdf9a5b395\" (UID: \"e2c695f5-d7bf-40a8-8ced-67cdf9a5b395\") " Sep 06 12:34:32.529378 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.529352 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/477e5cf8-391a-426e-ae31-7ae89309647e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "477e5cf8-391a-426e-ae31-7ae89309647e" (UID: "477e5cf8-391a-426e-ae31-7ae89309647e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:34:32.529470 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.529448 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2c695f5-d7bf-40a8-8ced-67cdf9a5b395-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e2c695f5-d7bf-40a8-8ced-67cdf9a5b395" (UID: "e2c695f5-d7bf-40a8-8ced-67cdf9a5b395"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:34:32.529523 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.529456 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/477e5cf8-391a-426e-ae31-7ae89309647e-plugins-config-volume\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:34:32.529523 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.529497 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95gpd\" (UniqueName: \"kubernetes.io/projected/28c434ae-619c-4d1d-ac53-74a62d401b76-kube-api-access-95gpd\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:34:32.529523 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.529509 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28c434ae-619c-4d1d-ac53-74a62d401b76-maas-api-tls\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:34:32.529523 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.529524 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28c434ae-619c-4d1d-ac53-74a62d401b76-metrics-certs\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:34:32.529659 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.529532 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28c434ae-619c-4d1d-ac53-74a62d401b76-openshift-service-ca\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:34:32.531333 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.531301 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/477e5cf8-391a-426e-ae31-7ae89309647e-kube-api-access-757rx" (OuterVolumeSpecName: "kube-api-access-757rx") pod "477e5cf8-391a-426e-ae31-7ae89309647e" (UID: "477e5cf8-391a-426e-ae31-7ae89309647e"). InnerVolumeSpecName "kube-api-access-757rx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:34:32.531439 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.531402 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2c695f5-d7bf-40a8-8ced-67cdf9a5b395-kube-api-access-qwrpw" (OuterVolumeSpecName: "kube-api-access-qwrpw") pod "e2c695f5-d7bf-40a8-8ced-67cdf9a5b395" (UID: "e2c695f5-d7bf-40a8-8ced-67cdf9a5b395"). InnerVolumeSpecName "kube-api-access-qwrpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:34:32.630688 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.630667 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-757rx\" (UniqueName: \"kubernetes.io/projected/477e5cf8-391a-426e-ae31-7ae89309647e-kube-api-access-757rx\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:34:32.630688 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.630688 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwrpw\" (UniqueName: \"kubernetes.io/projected/e2c695f5-d7bf-40a8-8ced-67cdf9a5b395-kube-api-access-qwrpw\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:34:32.630849 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.630698 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2c695f5-d7bf-40a8-8ced-67cdf9a5b395-plugins-config-volume\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:34:32.835175 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.835101 2576 generic.go:356] "Generic (PLEG): container finished" podID="e2c695f5-d7bf-40a8-8ced-67cdf9a5b395" containerID="03428ee591c2c177389349baaa2e3f28257d4c56ad10fe0a25205cba63e06c43" exitCode=0 Sep 06 12:34:32.835175 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.835139 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-1655f5-5657b449b6-nq9vn" Sep 06 12:34:32.835363 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.835179 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-1655f5-5657b449b6-nq9vn" event={"ID":"e2c695f5-d7bf-40a8-8ced-67cdf9a5b395","Type":"ContainerDied","Data":"03428ee591c2c177389349baaa2e3f28257d4c56ad10fe0a25205cba63e06c43"} Sep 06 12:34:32.835363 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.835232 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-1655f5-5657b449b6-nq9vn" event={"ID":"e2c695f5-d7bf-40a8-8ced-67cdf9a5b395","Type":"ContainerDied","Data":"f7628d41c5fd76c3a7f5a2fabb07b5b9846a4b7ceadb3e71f032678ec1ba1709"} Sep 06 12:34:32.835363 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.835259 2576 scope.go:117] "RemoveContainer" containerID="03428ee591c2c177389349baaa2e3f28257d4c56ad10fe0a25205cba63e06c43" Sep 06 12:34:32.837082 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.837056 2576 generic.go:356] "Generic (PLEG): container finished" podID="28c434ae-619c-4d1d-ac53-74a62d401b76" containerID="bbc2a67c28430ac521cd97cfdb7bebe668b0a475ad69fd0aee3653d2b61f958f" exitCode=0 Sep 06 12:34:32.837189 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.837131 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1655f5-c85b75486-xb5x8" Sep 06 12:34:32.837300 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.837185 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1655f5-c85b75486-xb5x8" event={"ID":"28c434ae-619c-4d1d-ac53-74a62d401b76","Type":"ContainerDied","Data":"bbc2a67c28430ac521cd97cfdb7bebe668b0a475ad69fd0aee3653d2b61f958f"} Sep 06 12:34:32.837300 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.837229 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1655f5-c85b75486-xb5x8" event={"ID":"28c434ae-619c-4d1d-ac53-74a62d401b76","Type":"ContainerDied","Data":"547cff9caf1342d39b3b93f2449c1c31cc261e4469fa796bf9d0049376640a0a"} Sep 06 12:34:32.838742 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.838716 2576 generic.go:356] "Generic (PLEG): container finished" podID="477e5cf8-391a-426e-ae31-7ae89309647e" containerID="aa8d2b1a550d2a19d16f1f17a9d211942ee7d375469013faa1635dc3eac6c943" exitCode=0 Sep 06 12:34:32.838850 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.838743 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-1655f5-5d49765fb9-s5mnp" event={"ID":"477e5cf8-391a-426e-ae31-7ae89309647e","Type":"ContainerDied","Data":"aa8d2b1a550d2a19d16f1f17a9d211942ee7d375469013faa1635dc3eac6c943"} Sep 06 12:34:32.838850 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.838784 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-1655f5-5d49765fb9-s5mnp" Sep 06 12:34:32.838850 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.838811 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-1655f5-5d49765fb9-s5mnp" event={"ID":"477e5cf8-391a-426e-ae31-7ae89309647e","Type":"ContainerDied","Data":"2511842b0003fef230fea620fce8b841cd16bee68084f8aaf3e8c640f41a7637"} Sep 06 12:34:32.845615 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.845601 2576 scope.go:117] "RemoveContainer" containerID="03428ee591c2c177389349baaa2e3f28257d4c56ad10fe0a25205cba63e06c43" Sep 06 12:34:32.845901 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:34:32.845884 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03428ee591c2c177389349baaa2e3f28257d4c56ad10fe0a25205cba63e06c43\": container with ID starting with 03428ee591c2c177389349baaa2e3f28257d4c56ad10fe0a25205cba63e06c43 not found: ID does not exist" containerID="03428ee591c2c177389349baaa2e3f28257d4c56ad10fe0a25205cba63e06c43" Sep 06 12:34:32.845947 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.845908 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03428ee591c2c177389349baaa2e3f28257d4c56ad10fe0a25205cba63e06c43"} err="failed to get container status \"03428ee591c2c177389349baaa2e3f28257d4c56ad10fe0a25205cba63e06c43\": rpc error: code = NotFound desc = could not find container \"03428ee591c2c177389349baaa2e3f28257d4c56ad10fe0a25205cba63e06c43\": container with ID starting with 03428ee591c2c177389349baaa2e3f28257d4c56ad10fe0a25205cba63e06c43 not found: ID does not exist" Sep 06 12:34:32.845947 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.845922 2576 scope.go:117] "RemoveContainer" containerID="bbc2a67c28430ac521cd97cfdb7bebe668b0a475ad69fd0aee3653d2b61f958f" Sep 06 12:34:32.853624 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.853599 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-1655f5-5657b449b6-nq9vn"] Sep 06 12:34:32.854525 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.854509 2576 scope.go:117] "RemoveContainer" containerID="bbc2a67c28430ac521cd97cfdb7bebe668b0a475ad69fd0aee3653d2b61f958f" Sep 06 12:34:32.854780 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:34:32.854746 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbc2a67c28430ac521cd97cfdb7bebe668b0a475ad69fd0aee3653d2b61f958f\": container with ID starting with bbc2a67c28430ac521cd97cfdb7bebe668b0a475ad69fd0aee3653d2b61f958f not found: ID does not exist" containerID="bbc2a67c28430ac521cd97cfdb7bebe668b0a475ad69fd0aee3653d2b61f958f" Sep 06 12:34:32.854841 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.854785 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbc2a67c28430ac521cd97cfdb7bebe668b0a475ad69fd0aee3653d2b61f958f"} err="failed to get container status \"bbc2a67c28430ac521cd97cfdb7bebe668b0a475ad69fd0aee3653d2b61f958f\": rpc error: code = NotFound desc = could not find container \"bbc2a67c28430ac521cd97cfdb7bebe668b0a475ad69fd0aee3653d2b61f958f\": container with ID starting with bbc2a67c28430ac521cd97cfdb7bebe668b0a475ad69fd0aee3653d2b61f958f not found: ID does not exist" Sep 06 12:34:32.854841 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.854800 2576 scope.go:117] "RemoveContainer" containerID="aa8d2b1a550d2a19d16f1f17a9d211942ee7d375469013faa1635dc3eac6c943" Sep 06 12:34:32.858035 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.858015 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-1655f5-5657b449b6-nq9vn"] Sep 06 12:34:32.862910 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.862894 2576 scope.go:117] "RemoveContainer" containerID="aa8d2b1a550d2a19d16f1f17a9d211942ee7d375469013faa1635dc3eac6c943" Sep 06 12:34:32.863167 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:34:32.863149 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa8d2b1a550d2a19d16f1f17a9d211942ee7d375469013faa1635dc3eac6c943\": container with ID starting with aa8d2b1a550d2a19d16f1f17a9d211942ee7d375469013faa1635dc3eac6c943 not found: ID does not exist" containerID="aa8d2b1a550d2a19d16f1f17a9d211942ee7d375469013faa1635dc3eac6c943" Sep 06 12:34:32.863212 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.863171 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa8d2b1a550d2a19d16f1f17a9d211942ee7d375469013faa1635dc3eac6c943"} err="failed to get container status \"aa8d2b1a550d2a19d16f1f17a9d211942ee7d375469013faa1635dc3eac6c943\": rpc error: code = NotFound desc = could not find container \"aa8d2b1a550d2a19d16f1f17a9d211942ee7d375469013faa1635dc3eac6c943\": container with ID starting with aa8d2b1a550d2a19d16f1f17a9d211942ee7d375469013faa1635dc3eac6c943 not found: ID does not exist" Sep 06 12:34:32.867082 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.867063 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-1655f5-c85b75486-xb5x8"] Sep 06 12:34:32.870898 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.870879 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-1655f5-c85b75486-xb5x8"] Sep 06 12:34:32.880138 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.880110 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-1655f5-5d49765fb9-s5mnp"] Sep 06 12:34:32.881987 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:32.881968 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-1655f5-5d49765fb9-s5mnp"] Sep 06 12:34:34.653778 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:34.653711 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28c434ae-619c-4d1d-ac53-74a62d401b76" path="/var/lib/kubelet/pods/28c434ae-619c-4d1d-ac53-74a62d401b76/volumes" Sep 06 12:34:34.654189 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:34.654132 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="477e5cf8-391a-426e-ae31-7ae89309647e" path="/var/lib/kubelet/pods/477e5cf8-391a-426e-ae31-7ae89309647e/volumes" Sep 06 12:34:34.654471 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:34:34.654455 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2c695f5-d7bf-40a8-8ced-67cdf9a5b395" path="/var/lib/kubelet/pods/e2c695f5-d7bf-40a8-8ced-67cdf9a5b395/volumes" Sep 06 12:35:24.578286 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:24.578255 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8"] Sep 06 12:35:24.578811 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:24.578436 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8" podUID="2a12d7af-a130-4f4e-9796-86dc6d65fb3f" containerName="maas-api" containerID="cri-o://1922a1e21ad9615b1d572e64caf62a3390446ae700b5e9bff1c33563ecffd7e4" gracePeriod=30 Sep 06 12:35:24.637216 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:24.637184 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-36bcab-6447494fbc-mtb22"] Sep 06 12:35:24.637679 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:24.637646 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-36bcab-6447494fbc-mtb22" podUID="316d29e5-cab3-4085-a3d7-0604766545ed" containerName="payload-processing" containerID="cri-o://f220cb256b1c2addc7d13d1333b2c5f0fee97c25f27fe727dd5c584d03a45c15" gracePeriod=30 Sep 06 12:35:24.674552 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:24.674519 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-36bcab-6bd6d9df98-dz7qf"] Sep 06 12:35:24.674838 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:24.674784 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-36bcab-6bd6d9df98-dz7qf" podUID="ff5bc9b8-9277-4869-8846-0313557b5949" containerName="payload-pre-processing" containerID="cri-o://f739537f3a38c0fe8c4a3122574bc18286ae107b8dbe169f618c458c52a6f219" gracePeriod=30 Sep 06 12:35:24.934468 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:24.934398 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-36bcab-6447494fbc-mtb22" Sep 06 12:35:24.966080 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:24.965875 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-36bcab-6bd6d9df98-dz7qf" Sep 06 12:35:25.022935 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.022911 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8" Sep 06 12:35:25.080313 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.080284 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdv9v\" (UniqueName: \"kubernetes.io/projected/ff5bc9b8-9277-4869-8846-0313557b5949-kube-api-access-bdv9v\") pod \"ff5bc9b8-9277-4869-8846-0313557b5949\" (UID: \"ff5bc9b8-9277-4869-8846-0313557b5949\") " Sep 06 12:35:25.080444 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.080375 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff5bc9b8-9277-4869-8846-0313557b5949-plugins-config-volume\") pod \"ff5bc9b8-9277-4869-8846-0313557b5949\" (UID: \"ff5bc9b8-9277-4869-8846-0313557b5949\") " Sep 06 12:35:25.080444 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.080420 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpwpd\" (UniqueName: \"kubernetes.io/projected/316d29e5-cab3-4085-a3d7-0604766545ed-kube-api-access-jpwpd\") pod \"316d29e5-cab3-4085-a3d7-0604766545ed\" (UID: \"316d29e5-cab3-4085-a3d7-0604766545ed\") " Sep 06 12:35:25.080518 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.080473 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/316d29e5-cab3-4085-a3d7-0604766545ed-plugins-config-volume\") pod \"316d29e5-cab3-4085-a3d7-0604766545ed\" (UID: \"316d29e5-cab3-4085-a3d7-0604766545ed\") " Sep 06 12:35:25.080799 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.080744 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff5bc9b8-9277-4869-8846-0313557b5949-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ff5bc9b8-9277-4869-8846-0313557b5949" (UID: "ff5bc9b8-9277-4869-8846-0313557b5949"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:35:25.080901 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.080883 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/316d29e5-cab3-4085-a3d7-0604766545ed-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "316d29e5-cab3-4085-a3d7-0604766545ed" (UID: "316d29e5-cab3-4085-a3d7-0604766545ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:35:25.082599 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.082573 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/316d29e5-cab3-4085-a3d7-0604766545ed-kube-api-access-jpwpd" (OuterVolumeSpecName: "kube-api-access-jpwpd") pod "316d29e5-cab3-4085-a3d7-0604766545ed" (UID: "316d29e5-cab3-4085-a3d7-0604766545ed"). InnerVolumeSpecName "kube-api-access-jpwpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:35:25.082690 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.082619 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff5bc9b8-9277-4869-8846-0313557b5949-kube-api-access-bdv9v" (OuterVolumeSpecName: "kube-api-access-bdv9v") pod "ff5bc9b8-9277-4869-8846-0313557b5949" (UID: "ff5bc9b8-9277-4869-8846-0313557b5949"). InnerVolumeSpecName "kube-api-access-bdv9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:35:25.114417 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.114360 2576 generic.go:356] "Generic (PLEG): container finished" podID="2a12d7af-a130-4f4e-9796-86dc6d65fb3f" containerID="1922a1e21ad9615b1d572e64caf62a3390446ae700b5e9bff1c33563ecffd7e4" exitCode=0 Sep 06 12:35:25.114417 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.114403 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8" event={"ID":"2a12d7af-a130-4f4e-9796-86dc6d65fb3f","Type":"ContainerDied","Data":"1922a1e21ad9615b1d572e64caf62a3390446ae700b5e9bff1c33563ecffd7e4"} Sep 06 12:35:25.114556 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.114409 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8" Sep 06 12:35:25.114556 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.114453 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8" event={"ID":"2a12d7af-a130-4f4e-9796-86dc6d65fb3f","Type":"ContainerDied","Data":"7455f13656573ef4a5ae939808a92536a17080e32ee3ac6a2d8868214d9c2e7d"} Sep 06 12:35:25.114556 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.114477 2576 scope.go:117] "RemoveContainer" containerID="1922a1e21ad9615b1d572e64caf62a3390446ae700b5e9bff1c33563ecffd7e4" Sep 06 12:35:25.115876 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.115855 2576 generic.go:356] "Generic (PLEG): container finished" podID="ff5bc9b8-9277-4869-8846-0313557b5949" containerID="f739537f3a38c0fe8c4a3122574bc18286ae107b8dbe169f618c458c52a6f219" exitCode=0 Sep 06 12:35:25.115972 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.115884 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-36bcab-6bd6d9df98-dz7qf" event={"ID":"ff5bc9b8-9277-4869-8846-0313557b5949","Type":"ContainerDied","Data":"f739537f3a38c0fe8c4a3122574bc18286ae107b8dbe169f618c458c52a6f219"} Sep 06 12:35:25.115972 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.115913 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-36bcab-6bd6d9df98-dz7qf" Sep 06 12:35:25.115972 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.115923 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-36bcab-6bd6d9df98-dz7qf" event={"ID":"ff5bc9b8-9277-4869-8846-0313557b5949","Type":"ContainerDied","Data":"4dce191200259be7ffd4c2872b9dc052bd81510a3627d97bd74684ff4bce0701"} Sep 06 12:35:25.117344 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.117324 2576 generic.go:356] "Generic (PLEG): container finished" podID="316d29e5-cab3-4085-a3d7-0604766545ed" containerID="f220cb256b1c2addc7d13d1333b2c5f0fee97c25f27fe727dd5c584d03a45c15" exitCode=0 Sep 06 12:35:25.117458 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.117353 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-36bcab-6447494fbc-mtb22" event={"ID":"316d29e5-cab3-4085-a3d7-0604766545ed","Type":"ContainerDied","Data":"f220cb256b1c2addc7d13d1333b2c5f0fee97c25f27fe727dd5c584d03a45c15"} Sep 06 12:35:25.117458 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.117404 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-36bcab-6447494fbc-mtb22" Sep 06 12:35:25.117458 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.117435 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-36bcab-6447494fbc-mtb22" event={"ID":"316d29e5-cab3-4085-a3d7-0604766545ed","Type":"ContainerDied","Data":"e16d324bbc8595b507e13fc5e0cb14ca2bd292bb9bfdb04d106a7635f5e229fc"} Sep 06 12:35:25.125831 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.125794 2576 scope.go:117] "RemoveContainer" containerID="1922a1e21ad9615b1d572e64caf62a3390446ae700b5e9bff1c33563ecffd7e4" Sep 06 12:35:25.126168 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:35:25.126140 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1922a1e21ad9615b1d572e64caf62a3390446ae700b5e9bff1c33563ecffd7e4\": container with ID starting with 1922a1e21ad9615b1d572e64caf62a3390446ae700b5e9bff1c33563ecffd7e4 not found: ID does not exist" containerID="1922a1e21ad9615b1d572e64caf62a3390446ae700b5e9bff1c33563ecffd7e4" Sep 06 12:35:25.126328 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.126178 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1922a1e21ad9615b1d572e64caf62a3390446ae700b5e9bff1c33563ecffd7e4"} err="failed to get container status \"1922a1e21ad9615b1d572e64caf62a3390446ae700b5e9bff1c33563ecffd7e4\": rpc error: code = NotFound desc = could not find container \"1922a1e21ad9615b1d572e64caf62a3390446ae700b5e9bff1c33563ecffd7e4\": container with ID starting with 1922a1e21ad9615b1d572e64caf62a3390446ae700b5e9bff1c33563ecffd7e4 not found: ID does not exist" Sep 06 12:35:25.126328 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.126202 2576 scope.go:117] "RemoveContainer" containerID="f739537f3a38c0fe8c4a3122574bc18286ae107b8dbe169f618c458c52a6f219" Sep 06 12:35:25.134649 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.134633 2576 scope.go:117] "RemoveContainer" containerID="f739537f3a38c0fe8c4a3122574bc18286ae107b8dbe169f618c458c52a6f219" Sep 06 12:35:25.134937 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:35:25.134919 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f739537f3a38c0fe8c4a3122574bc18286ae107b8dbe169f618c458c52a6f219\": container with ID starting with f739537f3a38c0fe8c4a3122574bc18286ae107b8dbe169f618c458c52a6f219 not found: ID does not exist" containerID="f739537f3a38c0fe8c4a3122574bc18286ae107b8dbe169f618c458c52a6f219" Sep 06 12:35:25.135024 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.134947 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f739537f3a38c0fe8c4a3122574bc18286ae107b8dbe169f618c458c52a6f219"} err="failed to get container status \"f739537f3a38c0fe8c4a3122574bc18286ae107b8dbe169f618c458c52a6f219\": rpc error: code = NotFound desc = could not find container \"f739537f3a38c0fe8c4a3122574bc18286ae107b8dbe169f618c458c52a6f219\": container with ID starting with f739537f3a38c0fe8c4a3122574bc18286ae107b8dbe169f618c458c52a6f219 not found: ID does not exist" Sep 06 12:35:25.135024 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.134968 2576 scope.go:117] "RemoveContainer" containerID="f220cb256b1c2addc7d13d1333b2c5f0fee97c25f27fe727dd5c584d03a45c15" Sep 06 12:35:25.141009 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.140991 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-36bcab-6bd6d9df98-dz7qf"] Sep 06 12:35:25.143179 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.143157 2576 scope.go:117] "RemoveContainer" containerID="f220cb256b1c2addc7d13d1333b2c5f0fee97c25f27fe727dd5c584d03a45c15" Sep 06 12:35:25.143478 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:35:25.143456 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f220cb256b1c2addc7d13d1333b2c5f0fee97c25f27fe727dd5c584d03a45c15\": container with ID starting with f220cb256b1c2addc7d13d1333b2c5f0fee97c25f27fe727dd5c584d03a45c15 not found: ID does not exist" containerID="f220cb256b1c2addc7d13d1333b2c5f0fee97c25f27fe727dd5c584d03a45c15" Sep 06 12:35:25.143556 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.143484 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f220cb256b1c2addc7d13d1333b2c5f0fee97c25f27fe727dd5c584d03a45c15"} err="failed to get container status \"f220cb256b1c2addc7d13d1333b2c5f0fee97c25f27fe727dd5c584d03a45c15\": rpc error: code = NotFound desc = could not find container \"f220cb256b1c2addc7d13d1333b2c5f0fee97c25f27fe727dd5c584d03a45c15\": container with ID starting with f220cb256b1c2addc7d13d1333b2c5f0fee97c25f27fe727dd5c584d03a45c15 not found: ID does not exist" Sep 06 12:35:25.145083 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.145059 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-36bcab-6bd6d9df98-dz7qf"] Sep 06 12:35:25.156458 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.156334 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-36bcab-6447494fbc-mtb22"] Sep 06 12:35:25.159583 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.158874 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-36bcab-6447494fbc-mtb22"] Sep 06 12:35:25.181361 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.181339 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a12d7af-a130-4f4e-9796-86dc6d65fb3f-openshift-service-ca\") pod \"2a12d7af-a130-4f4e-9796-86dc6d65fb3f\" (UID: \"2a12d7af-a130-4f4e-9796-86dc6d65fb3f\") " Sep 06 12:35:25.181441 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.181387 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a12d7af-a130-4f4e-9796-86dc6d65fb3f-metrics-certs\") pod \"2a12d7af-a130-4f4e-9796-86dc6d65fb3f\" (UID: \"2a12d7af-a130-4f4e-9796-86dc6d65fb3f\") " Sep 06 12:35:25.181441 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.181405 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a12d7af-a130-4f4e-9796-86dc6d65fb3f-maas-api-tls\") pod \"2a12d7af-a130-4f4e-9796-86dc6d65fb3f\" (UID: \"2a12d7af-a130-4f4e-9796-86dc6d65fb3f\") " Sep 06 12:35:25.181441 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.181434 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52bmb\" (UniqueName: \"kubernetes.io/projected/2a12d7af-a130-4f4e-9796-86dc6d65fb3f-kube-api-access-52bmb\") pod \"2a12d7af-a130-4f4e-9796-86dc6d65fb3f\" (UID: \"2a12d7af-a130-4f4e-9796-86dc6d65fb3f\") " Sep 06 12:35:25.181631 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.181618 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdv9v\" (UniqueName: \"kubernetes.io/projected/ff5bc9b8-9277-4869-8846-0313557b5949-kube-api-access-bdv9v\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:35:25.181696 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.181633 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff5bc9b8-9277-4869-8846-0313557b5949-plugins-config-volume\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:35:25.181696 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.181642 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpwpd\" (UniqueName: \"kubernetes.io/projected/316d29e5-cab3-4085-a3d7-0604766545ed-kube-api-access-jpwpd\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:35:25.181696 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.181650 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/316d29e5-cab3-4085-a3d7-0604766545ed-plugins-config-volume\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:35:25.181882 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.181714 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a12d7af-a130-4f4e-9796-86dc6d65fb3f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2a12d7af-a130-4f4e-9796-86dc6d65fb3f" (UID: "2a12d7af-a130-4f4e-9796-86dc6d65fb3f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:35:25.183486 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.183456 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a12d7af-a130-4f4e-9796-86dc6d65fb3f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2a12d7af-a130-4f4e-9796-86dc6d65fb3f" (UID: "2a12d7af-a130-4f4e-9796-86dc6d65fb3f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 12:35:25.183565 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.183498 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a12d7af-a130-4f4e-9796-86dc6d65fb3f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2a12d7af-a130-4f4e-9796-86dc6d65fb3f" (UID: "2a12d7af-a130-4f4e-9796-86dc6d65fb3f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 12:35:25.183565 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.183545 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a12d7af-a130-4f4e-9796-86dc6d65fb3f-kube-api-access-52bmb" (OuterVolumeSpecName: "kube-api-access-52bmb") pod "2a12d7af-a130-4f4e-9796-86dc6d65fb3f" (UID: "2a12d7af-a130-4f4e-9796-86dc6d65fb3f"). InnerVolumeSpecName "kube-api-access-52bmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:35:25.282318 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.282297 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a12d7af-a130-4f4e-9796-86dc6d65fb3f-openshift-service-ca\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:35:25.282318 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.282318 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a12d7af-a130-4f4e-9796-86dc6d65fb3f-metrics-certs\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:35:25.282318 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.282327 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a12d7af-a130-4f4e-9796-86dc6d65fb3f-maas-api-tls\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:35:25.282485 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.282335 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52bmb\" (UniqueName: \"kubernetes.io/projected/2a12d7af-a130-4f4e-9796-86dc6d65fb3f-kube-api-access-52bmb\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:35:25.436164 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.436136 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8"] Sep 06 12:35:25.439873 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:25.439849 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-36bcab-947bcdd46-rbmh8"] Sep 06 12:35:26.654511 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:26.654474 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a12d7af-a130-4f4e-9796-86dc6d65fb3f" path="/var/lib/kubelet/pods/2a12d7af-a130-4f4e-9796-86dc6d65fb3f/volumes" Sep 06 12:35:26.655014 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:26.654995 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="316d29e5-cab3-4085-a3d7-0604766545ed" path="/var/lib/kubelet/pods/316d29e5-cab3-4085-a3d7-0604766545ed/volumes" Sep 06 12:35:26.655344 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:26.655332 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff5bc9b8-9277-4869-8846-0313557b5949" path="/var/lib/kubelet/pods/ff5bc9b8-9277-4869-8846-0313557b5949/volumes" Sep 06 12:35:28.593538 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.593504 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d96501-69744d7f8c-dw76p"] Sep 06 12:35:28.594195 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.593970 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff5bc9b8-9277-4869-8846-0313557b5949" containerName="payload-pre-processing" Sep 06 12:35:28.594195 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.593985 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff5bc9b8-9277-4869-8846-0313557b5949" containerName="payload-pre-processing" Sep 06 12:35:28.594195 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.594012 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="316d29e5-cab3-4085-a3d7-0604766545ed" containerName="payload-processing" Sep 06 12:35:28.594195 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.594018 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="316d29e5-cab3-4085-a3d7-0604766545ed" containerName="payload-processing" Sep 06 12:35:28.594195 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.594025 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a12d7af-a130-4f4e-9796-86dc6d65fb3f" containerName="maas-api" Sep 06 12:35:28.594195 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.594030 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a12d7af-a130-4f4e-9796-86dc6d65fb3f" containerName="maas-api" Sep 06 12:35:28.594195 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.594037 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2c695f5-d7bf-40a8-8ced-67cdf9a5b395" containerName="payload-pre-processing" Sep 06 12:35:28.594195 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.594042 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2c695f5-d7bf-40a8-8ced-67cdf9a5b395" containerName="payload-pre-processing" Sep 06 12:35:28.594195 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.594050 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28c434ae-619c-4d1d-ac53-74a62d401b76" containerName="maas-api" Sep 06 12:35:28.594195 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.594055 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="28c434ae-619c-4d1d-ac53-74a62d401b76" containerName="maas-api" Sep 06 12:35:28.594195 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.594061 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="477e5cf8-391a-426e-ae31-7ae89309647e" containerName="payload-processing" Sep 06 12:35:28.594195 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.594066 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="477e5cf8-391a-426e-ae31-7ae89309647e" containerName="payload-processing" Sep 06 12:35:28.594195 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.594122 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="316d29e5-cab3-4085-a3d7-0604766545ed" containerName="payload-processing" Sep 06 12:35:28.594195 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.594129 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2c695f5-d7bf-40a8-8ced-67cdf9a5b395" containerName="payload-pre-processing" Sep 06 12:35:28.594195 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.594134 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="477e5cf8-391a-426e-ae31-7ae89309647e" containerName="payload-processing" Sep 06 12:35:28.594195 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.594141 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="28c434ae-619c-4d1d-ac53-74a62d401b76" containerName="maas-api" Sep 06 12:35:28.594195 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.594147 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff5bc9b8-9277-4869-8846-0313557b5949" containerName="payload-pre-processing" Sep 06 12:35:28.594195 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.594153 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a12d7af-a130-4f4e-9796-86dc6d65fb3f" containerName="maas-api" Sep 06 12:35:28.598521 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.598504 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d96501-69744d7f8c-dw76p" Sep 06 12:35:28.600795 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.600773 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-d96501-dockercfg-brr99\"" Sep 06 12:35:28.600918 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.600857 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-d96501\"" Sep 06 12:35:28.604584 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.604557 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d96501-69744d7f8c-dw76p"] Sep 06 12:35:28.622395 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.622367 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96501-65b678d9d6-mfltf"] Sep 06 12:35:28.626008 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.625992 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96501-65b678d9d6-mfltf" Sep 06 12:35:28.638961 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.638928 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96501-65b678d9d6-mfltf"] Sep 06 12:35:28.709661 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.709628 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxs9z\" (UniqueName: \"kubernetes.io/projected/1dcdde5c-dd2b-4c98-8daf-56cf9124ea0e-kube-api-access-kxs9z\") pod \"payload-processing-e2e-shared-a-w2-d96501-69744d7f8c-dw76p\" (UID: \"1dcdde5c-dd2b-4c98-8daf-56cf9124ea0e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d96501-69744d7f8c-dw76p" Sep 06 12:35:28.709852 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.709708 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4c48\" (UniqueName: \"kubernetes.io/projected/679235a6-cf73-42f8-8770-8e2bc04afd6c-kube-api-access-k4c48\") pod \"payload-pre-processing-e2e-shared-a-w2-d96501-65b678d9d6-mfltf\" (UID: \"679235a6-cf73-42f8-8770-8e2bc04afd6c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96501-65b678d9d6-mfltf" Sep 06 12:35:28.709852 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.709745 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/679235a6-cf73-42f8-8770-8e2bc04afd6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d96501-65b678d9d6-mfltf\" (UID: \"679235a6-cf73-42f8-8770-8e2bc04afd6c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96501-65b678d9d6-mfltf" Sep 06 12:35:28.709852 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.709840 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dcdde5c-dd2b-4c98-8daf-56cf9124ea0e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d96501-69744d7f8c-dw76p\" (UID: \"1dcdde5c-dd2b-4c98-8daf-56cf9124ea0e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d96501-69744d7f8c-dw76p" Sep 06 12:35:28.811145 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.811107 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/679235a6-cf73-42f8-8770-8e2bc04afd6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d96501-65b678d9d6-mfltf\" (UID: \"679235a6-cf73-42f8-8770-8e2bc04afd6c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96501-65b678d9d6-mfltf" Sep 06 12:35:28.811304 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.811154 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dcdde5c-dd2b-4c98-8daf-56cf9124ea0e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d96501-69744d7f8c-dw76p\" (UID: \"1dcdde5c-dd2b-4c98-8daf-56cf9124ea0e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d96501-69744d7f8c-dw76p" Sep 06 12:35:28.811304 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.811203 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxs9z\" (UniqueName: \"kubernetes.io/projected/1dcdde5c-dd2b-4c98-8daf-56cf9124ea0e-kube-api-access-kxs9z\") pod \"payload-processing-e2e-shared-a-w2-d96501-69744d7f8c-dw76p\" (UID: \"1dcdde5c-dd2b-4c98-8daf-56cf9124ea0e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d96501-69744d7f8c-dw76p" Sep 06 12:35:28.811304 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.811288 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4c48\" (UniqueName: \"kubernetes.io/projected/679235a6-cf73-42f8-8770-8e2bc04afd6c-kube-api-access-k4c48\") pod \"payload-pre-processing-e2e-shared-a-w2-d96501-65b678d9d6-mfltf\" (UID: \"679235a6-cf73-42f8-8770-8e2bc04afd6c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96501-65b678d9d6-mfltf" Sep 06 12:35:28.812088 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.812063 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dcdde5c-dd2b-4c98-8daf-56cf9124ea0e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d96501-69744d7f8c-dw76p\" (UID: \"1dcdde5c-dd2b-4c98-8daf-56cf9124ea0e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d96501-69744d7f8c-dw76p" Sep 06 12:35:28.812181 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.812074 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/679235a6-cf73-42f8-8770-8e2bc04afd6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d96501-65b678d9d6-mfltf\" (UID: \"679235a6-cf73-42f8-8770-8e2bc04afd6c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96501-65b678d9d6-mfltf" Sep 06 12:35:28.820202 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.820175 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxs9z\" (UniqueName: \"kubernetes.io/projected/1dcdde5c-dd2b-4c98-8daf-56cf9124ea0e-kube-api-access-kxs9z\") pod \"payload-processing-e2e-shared-a-w2-d96501-69744d7f8c-dw76p\" (UID: \"1dcdde5c-dd2b-4c98-8daf-56cf9124ea0e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d96501-69744d7f8c-dw76p" Sep 06 12:35:28.820202 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.820183 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4c48\" (UniqueName: \"kubernetes.io/projected/679235a6-cf73-42f8-8770-8e2bc04afd6c-kube-api-access-k4c48\") pod \"payload-pre-processing-e2e-shared-a-w2-d96501-65b678d9d6-mfltf\" (UID: \"679235a6-cf73-42f8-8770-8e2bc04afd6c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96501-65b678d9d6-mfltf" Sep 06 12:35:28.910536 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.910514 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d96501-69744d7f8c-dw76p" Sep 06 12:35:28.936507 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:28.936485 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96501-65b678d9d6-mfltf" Sep 06 12:35:29.183074 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:29.183052 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d96501-69744d7f8c-dw76p"] Sep 06 12:35:29.185873 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:35:29.185823 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dcdde5c_dd2b_4c98_8daf_56cf9124ea0e.slice/crio-766e7842cb6e37c979cdf6d22cdfae09e70cbd24871194b013396b1fb2db78ad WatchSource:0}: Error finding container 766e7842cb6e37c979cdf6d22cdfae09e70cbd24871194b013396b1fb2db78ad: Status 404 returned error can't find the container with id 766e7842cb6e37c979cdf6d22cdfae09e70cbd24871194b013396b1fb2db78ad Sep 06 12:35:29.221327 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:29.221300 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96501-65b678d9d6-mfltf"] Sep 06 12:35:29.224010 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:35:29.223984 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod679235a6_cf73_42f8_8770_8e2bc04afd6c.slice/crio-560d90713d6ee534a2076f2b47c1844656c970e3a624f2aea2d526b02588c854 WatchSource:0}: Error finding container 560d90713d6ee534a2076f2b47c1844656c970e3a624f2aea2d526b02588c854: Status 404 returned error can't find the container with id 560d90713d6ee534a2076f2b47c1844656c970e3a624f2aea2d526b02588c854 Sep 06 12:35:30.152375 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:30.152333 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96501-65b678d9d6-mfltf" event={"ID":"679235a6-cf73-42f8-8770-8e2bc04afd6c","Type":"ContainerStarted","Data":"546ae6ba8b02b90203a4724f6be3700ec845963706083fa608a7cdf4c0f0dd85"} Sep 06 12:35:30.152851 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:30.152399 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96501-65b678d9d6-mfltf" event={"ID":"679235a6-cf73-42f8-8770-8e2bc04afd6c","Type":"ContainerStarted","Data":"560d90713d6ee534a2076f2b47c1844656c970e3a624f2aea2d526b02588c854"} Sep 06 12:35:30.152851 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:30.152430 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96501-65b678d9d6-mfltf" Sep 06 12:35:30.153803 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:30.153784 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d96501-69744d7f8c-dw76p" event={"ID":"1dcdde5c-dd2b-4c98-8daf-56cf9124ea0e","Type":"ContainerStarted","Data":"f13961c33db43c4e92314c471089a985f9d2a99c727cbad42b6a9fa519c39f41"} Sep 06 12:35:30.153865 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:30.153822 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d96501-69744d7f8c-dw76p" event={"ID":"1dcdde5c-dd2b-4c98-8daf-56cf9124ea0e","Type":"ContainerStarted","Data":"766e7842cb6e37c979cdf6d22cdfae09e70cbd24871194b013396b1fb2db78ad"} Sep 06 12:35:30.153903 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:30.153860 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d96501-69744d7f8c-dw76p" Sep 06 12:35:30.168320 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:30.168259 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96501-65b678d9d6-mfltf" podStartSLOduration=2.168245277 podStartE2EDuration="2.168245277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:35:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:35:30.167603204 +0000 UTC m=+1046.023361112" watchObservedRunningTime="2026-09-06 12:35:30.168245277 +0000 UTC m=+1046.024003193" Sep 06 12:35:30.184449 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:30.184406 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d96501-69744d7f8c-dw76p" podStartSLOduration=2.184392579 podStartE2EDuration="2.184392579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:35:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:35:30.183134851 +0000 UTC m=+1046.038892776" watchObservedRunningTime="2026-09-06 12:35:30.184392579 +0000 UTC m=+1046.040150469" Sep 06 12:35:41.159956 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:41.159921 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96501-65b678d9d6-mfltf" Sep 06 12:35:41.160400 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:41.160262 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d96501-69744d7f8c-dw76p" Sep 06 12:35:41.884942 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:41.884904 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh"] Sep 06 12:35:41.885190 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:41.885160 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh" podUID="a4f0223d-3b53-4b7d-ad1d-af0ae3d35738" containerName="maas-api" containerID="cri-o://11773e520d2d16225e6a5d760ced48bf7f399c4c07adcd311cc698bc9d71dc4b" gracePeriod=30 Sep 06 12:35:42.221544 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:42.221520 2576 generic.go:356] "Generic (PLEG): container finished" podID="a4f0223d-3b53-4b7d-ad1d-af0ae3d35738" containerID="11773e520d2d16225e6a5d760ced48bf7f399c4c07adcd311cc698bc9d71dc4b" exitCode=0 Sep 06 12:35:42.221863 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:42.221568 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh" event={"ID":"a4f0223d-3b53-4b7d-ad1d-af0ae3d35738","Type":"ContainerDied","Data":"11773e520d2d16225e6a5d760ced48bf7f399c4c07adcd311cc698bc9d71dc4b"} Sep 06 12:35:42.347132 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:42.347109 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh" Sep 06 12:35:42.530509 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:42.530438 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4f0223d-3b53-4b7d-ad1d-af0ae3d35738-metrics-certs\") pod \"a4f0223d-3b53-4b7d-ad1d-af0ae3d35738\" (UID: \"a4f0223d-3b53-4b7d-ad1d-af0ae3d35738\") " Sep 06 12:35:42.530509 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:42.530479 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4f0223d-3b53-4b7d-ad1d-af0ae3d35738-openshift-service-ca\") pod \"a4f0223d-3b53-4b7d-ad1d-af0ae3d35738\" (UID: \"a4f0223d-3b53-4b7d-ad1d-af0ae3d35738\") " Sep 06 12:35:42.530716 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:42.530522 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4f0223d-3b53-4b7d-ad1d-af0ae3d35738-maas-api-tls\") pod \"a4f0223d-3b53-4b7d-ad1d-af0ae3d35738\" (UID: \"a4f0223d-3b53-4b7d-ad1d-af0ae3d35738\") " Sep 06 12:35:42.530716 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:42.530582 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqn75\" (UniqueName: \"kubernetes.io/projected/a4f0223d-3b53-4b7d-ad1d-af0ae3d35738-kube-api-access-kqn75\") pod \"a4f0223d-3b53-4b7d-ad1d-af0ae3d35738\" (UID: \"a4f0223d-3b53-4b7d-ad1d-af0ae3d35738\") " Sep 06 12:35:42.530908 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:42.530875 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4f0223d-3b53-4b7d-ad1d-af0ae3d35738-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a4f0223d-3b53-4b7d-ad1d-af0ae3d35738" (UID: "a4f0223d-3b53-4b7d-ad1d-af0ae3d35738"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:35:42.532599 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:42.532566 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4f0223d-3b53-4b7d-ad1d-af0ae3d35738-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a4f0223d-3b53-4b7d-ad1d-af0ae3d35738" (UID: "a4f0223d-3b53-4b7d-ad1d-af0ae3d35738"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 12:35:42.532711 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:42.532653 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4f0223d-3b53-4b7d-ad1d-af0ae3d35738-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a4f0223d-3b53-4b7d-ad1d-af0ae3d35738" (UID: "a4f0223d-3b53-4b7d-ad1d-af0ae3d35738"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 12:35:42.532921 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:42.532898 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4f0223d-3b53-4b7d-ad1d-af0ae3d35738-kube-api-access-kqn75" (OuterVolumeSpecName: "kube-api-access-kqn75") pod "a4f0223d-3b53-4b7d-ad1d-af0ae3d35738" (UID: "a4f0223d-3b53-4b7d-ad1d-af0ae3d35738"). InnerVolumeSpecName "kube-api-access-kqn75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:35:42.631796 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:42.631770 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4f0223d-3b53-4b7d-ad1d-af0ae3d35738-metrics-certs\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:35:42.631796 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:42.631795 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4f0223d-3b53-4b7d-ad1d-af0ae3d35738-openshift-service-ca\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:35:42.631936 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:42.631810 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4f0223d-3b53-4b7d-ad1d-af0ae3d35738-maas-api-tls\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:35:42.631936 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:42.631822 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqn75\" (UniqueName: \"kubernetes.io/projected/a4f0223d-3b53-4b7d-ad1d-af0ae3d35738-kube-api-access-kqn75\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:35:43.227577 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:43.227547 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh" Sep 06 12:35:43.227577 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:43.227568 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh" event={"ID":"a4f0223d-3b53-4b7d-ad1d-af0ae3d35738","Type":"ContainerDied","Data":"40f1357a130ad501a2b31f1bc63aa07d9c829bee072a430467f75ca950a5cbf5"} Sep 06 12:35:43.228074 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:43.227627 2576 scope.go:117] "RemoveContainer" containerID="11773e520d2d16225e6a5d760ced48bf7f399c4c07adcd311cc698bc9d71dc4b" Sep 06 12:35:43.244437 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:43.244414 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh"] Sep 06 12:35:43.250000 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:43.249976 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-58053a-6fbd95778b-9rxhh"] Sep 06 12:35:44.653449 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:44.653416 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4f0223d-3b53-4b7d-ad1d-af0ae3d35738" path="/var/lib/kubelet/pods/a4f0223d-3b53-4b7d-ad1d-af0ae3d35738/volumes" Sep 06 12:35:54.849399 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:54.849361 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3feb92-64d8bcfbc8-hx7mq"] Sep 06 12:35:54.849839 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:54.849824 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4f0223d-3b53-4b7d-ad1d-af0ae3d35738" containerName="maas-api" Sep 06 12:35:54.849892 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:54.849841 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4f0223d-3b53-4b7d-ad1d-af0ae3d35738" containerName="maas-api" Sep 06 12:35:54.849967 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:54.849947 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4f0223d-3b53-4b7d-ad1d-af0ae3d35738" containerName="maas-api" Sep 06 12:35:54.854382 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:54.854362 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3feb92-64d8bcfbc8-hx7mq" Sep 06 12:35:54.857564 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:54.857544 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-3feb92\"" Sep 06 12:35:54.857669 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:54.857589 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-3feb92-dockercfg-sgpdp\"" Sep 06 12:35:54.865455 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:54.865431 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3feb92-64d8bcfbc8-hx7mq"] Sep 06 12:35:54.867467 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:54.867445 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3feb92-6ddb5878b5-c6qsd"] Sep 06 12:35:54.871228 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:54.871208 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3feb92-6ddb5878b5-c6qsd" Sep 06 12:35:54.883732 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:54.883710 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3feb92-6ddb5878b5-c6qsd"] Sep 06 12:35:54.945866 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:54.945836 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/014d29cf-4ab3-4600-947b-a6b67306edb2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-3feb92-6ddb5878b5-c6qsd\" (UID: \"014d29cf-4ab3-4600-947b-a6b67306edb2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3feb92-6ddb5878b5-c6qsd" Sep 06 12:35:54.946056 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:54.945871 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb2b858b-eb53-4268-a0e4-e0dde762ae77-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-3feb92-64d8bcfbc8-hx7mq\" (UID: \"eb2b858b-eb53-4268-a0e4-e0dde762ae77\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3feb92-64d8bcfbc8-hx7mq" Sep 06 12:35:54.946056 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:54.945976 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47dpx\" (UniqueName: \"kubernetes.io/projected/eb2b858b-eb53-4268-a0e4-e0dde762ae77-kube-api-access-47dpx\") pod \"payload-pre-processing-e2e-shared-a-w3-3feb92-64d8bcfbc8-hx7mq\" (UID: \"eb2b858b-eb53-4268-a0e4-e0dde762ae77\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3feb92-64d8bcfbc8-hx7mq" Sep 06 12:35:54.946056 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:54.946007 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9hx5\" (UniqueName: \"kubernetes.io/projected/014d29cf-4ab3-4600-947b-a6b67306edb2-kube-api-access-t9hx5\") pod \"payload-processing-e2e-shared-a-w3-3feb92-6ddb5878b5-c6qsd\" (UID: \"014d29cf-4ab3-4600-947b-a6b67306edb2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3feb92-6ddb5878b5-c6qsd" Sep 06 12:35:55.047285 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:55.047248 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/014d29cf-4ab3-4600-947b-a6b67306edb2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-3feb92-6ddb5878b5-c6qsd\" (UID: \"014d29cf-4ab3-4600-947b-a6b67306edb2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3feb92-6ddb5878b5-c6qsd" Sep 06 12:35:55.047285 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:55.047289 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb2b858b-eb53-4268-a0e4-e0dde762ae77-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-3feb92-64d8bcfbc8-hx7mq\" (UID: \"eb2b858b-eb53-4268-a0e4-e0dde762ae77\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3feb92-64d8bcfbc8-hx7mq" Sep 06 12:35:55.047480 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:55.047344 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47dpx\" (UniqueName: \"kubernetes.io/projected/eb2b858b-eb53-4268-a0e4-e0dde762ae77-kube-api-access-47dpx\") pod \"payload-pre-processing-e2e-shared-a-w3-3feb92-64d8bcfbc8-hx7mq\" (UID: \"eb2b858b-eb53-4268-a0e4-e0dde762ae77\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3feb92-64d8bcfbc8-hx7mq" Sep 06 12:35:55.047480 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:55.047361 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9hx5\" (UniqueName: \"kubernetes.io/projected/014d29cf-4ab3-4600-947b-a6b67306edb2-kube-api-access-t9hx5\") pod \"payload-processing-e2e-shared-a-w3-3feb92-6ddb5878b5-c6qsd\" (UID: \"014d29cf-4ab3-4600-947b-a6b67306edb2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3feb92-6ddb5878b5-c6qsd" Sep 06 12:35:55.048053 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:55.048031 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/014d29cf-4ab3-4600-947b-a6b67306edb2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-3feb92-6ddb5878b5-c6qsd\" (UID: \"014d29cf-4ab3-4600-947b-a6b67306edb2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3feb92-6ddb5878b5-c6qsd" Sep 06 12:35:55.048149 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:55.048076 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb2b858b-eb53-4268-a0e4-e0dde762ae77-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-3feb92-64d8bcfbc8-hx7mq\" (UID: \"eb2b858b-eb53-4268-a0e4-e0dde762ae77\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3feb92-64d8bcfbc8-hx7mq" Sep 06 12:35:55.055293 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:55.055269 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9hx5\" (UniqueName: \"kubernetes.io/projected/014d29cf-4ab3-4600-947b-a6b67306edb2-kube-api-access-t9hx5\") pod \"payload-processing-e2e-shared-a-w3-3feb92-6ddb5878b5-c6qsd\" (UID: \"014d29cf-4ab3-4600-947b-a6b67306edb2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3feb92-6ddb5878b5-c6qsd" Sep 06 12:35:55.055701 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:55.055683 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47dpx\" (UniqueName: \"kubernetes.io/projected/eb2b858b-eb53-4268-a0e4-e0dde762ae77-kube-api-access-47dpx\") pod \"payload-pre-processing-e2e-shared-a-w3-3feb92-64d8bcfbc8-hx7mq\" (UID: \"eb2b858b-eb53-4268-a0e4-e0dde762ae77\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3feb92-64d8bcfbc8-hx7mq" Sep 06 12:35:55.165518 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:55.165495 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3feb92-64d8bcfbc8-hx7mq" Sep 06 12:35:55.184802 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:55.184746 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3feb92-6ddb5878b5-c6qsd" Sep 06 12:35:55.445240 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:55.445205 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3feb92-64d8bcfbc8-hx7mq"] Sep 06 12:35:55.447689 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:35:55.447660 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb2b858b_eb53_4268_a0e4_e0dde762ae77.slice/crio-ad06816183e671d18b2f0e839dcdd67d20d78cda726f389dc1c366a5a6a18f92 WatchSource:0}: Error finding container ad06816183e671d18b2f0e839dcdd67d20d78cda726f389dc1c366a5a6a18f92: Status 404 returned error can't find the container with id ad06816183e671d18b2f0e839dcdd67d20d78cda726f389dc1c366a5a6a18f92 Sep 06 12:35:55.467269 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:55.467246 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3feb92-6ddb5878b5-c6qsd"] Sep 06 12:35:55.469077 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:35:55.469051 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod014d29cf_4ab3_4600_947b_a6b67306edb2.slice/crio-a399f6cd586724181ce9b82600211e69069004fbedf7b94da20dc340144f8df7 WatchSource:0}: Error finding container a399f6cd586724181ce9b82600211e69069004fbedf7b94da20dc340144f8df7: Status 404 returned error can't find the container with id a399f6cd586724181ce9b82600211e69069004fbedf7b94da20dc340144f8df7 Sep 06 12:35:56.299145 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:56.299108 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3feb92-6ddb5878b5-c6qsd" event={"ID":"014d29cf-4ab3-4600-947b-a6b67306edb2","Type":"ContainerStarted","Data":"b32646351a601b833fd5f28be3dce3b75e3d7c0c46e55967da3382ed86809f4e"} Sep 06 12:35:56.299650 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:56.299174 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3feb92-6ddb5878b5-c6qsd" event={"ID":"014d29cf-4ab3-4600-947b-a6b67306edb2","Type":"ContainerStarted","Data":"a399f6cd586724181ce9b82600211e69069004fbedf7b94da20dc340144f8df7"} Sep 06 12:35:56.299650 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:56.299234 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3feb92-6ddb5878b5-c6qsd" Sep 06 12:35:56.301056 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:56.301030 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3feb92-64d8bcfbc8-hx7mq" event={"ID":"eb2b858b-eb53-4268-a0e4-e0dde762ae77","Type":"ContainerStarted","Data":"74a82cb9d3ce080972485e02e43ba87300e3dbe0cd7a8c3dedf63f2c870949cd"} Sep 06 12:35:56.301181 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:56.301068 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3feb92-64d8bcfbc8-hx7mq" event={"ID":"eb2b858b-eb53-4268-a0e4-e0dde762ae77","Type":"ContainerStarted","Data":"ad06816183e671d18b2f0e839dcdd67d20d78cda726f389dc1c366a5a6a18f92"} Sep 06 12:35:56.301181 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:56.301123 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3feb92-64d8bcfbc8-hx7mq" Sep 06 12:35:56.315242 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:56.315201 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3feb92-6ddb5878b5-c6qsd" podStartSLOduration=2.315188904 podStartE2EDuration="2.315188904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:35:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:35:56.313966348 +0000 UTC m=+1072.169724268" watchObservedRunningTime="2026-09-06 12:35:56.315188904 +0000 UTC m=+1072.170946795" Sep 06 12:35:56.335140 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:56.335094 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3feb92-64d8bcfbc8-hx7mq" podStartSLOduration=2.3350762720000002 podStartE2EDuration="2.335076272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:35:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:35:56.332385347 +0000 UTC m=+1072.188143274" watchObservedRunningTime="2026-09-06 12:35:56.335076272 +0000 UTC m=+1072.190834160" Sep 06 12:35:57.027693 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:57.027664 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-556bc6-54789d5666-h7gxf"] Sep 06 12:35:57.031593 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:57.031569 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-556bc6-54789d5666-h7gxf" Sep 06 12:35:57.033884 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:57.033860 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-556bc6\"" Sep 06 12:35:57.042184 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:57.042164 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-556bc6-54789d5666-h7gxf"] Sep 06 12:35:57.068002 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:57.067980 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddf74b53-1c59-4d1e-8e4e-55ba5ef2936f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-556bc6-54789d5666-h7gxf\" (UID: \"ddf74b53-1c59-4d1e-8e4e-55ba5ef2936f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-556bc6-54789d5666-h7gxf" Sep 06 12:35:57.068122 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:57.068030 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxt9n\" (UniqueName: \"kubernetes.io/projected/ddf74b53-1c59-4d1e-8e4e-55ba5ef2936f-kube-api-access-zxt9n\") pod \"payload-pre-processing-e2e-shared-b-w2-556bc6-54789d5666-h7gxf\" (UID: \"ddf74b53-1c59-4d1e-8e4e-55ba5ef2936f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-556bc6-54789d5666-h7gxf" Sep 06 12:35:57.073848 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:57.073826 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-556bc6-5bd6db54b9-dzhrj"] Sep 06 12:35:57.077361 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:57.077347 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-556bc6-5bd6db54b9-dzhrj" Sep 06 12:35:57.079491 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:57.079472 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-556bc6-dockercfg-vjtbk\"" Sep 06 12:35:57.093070 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:57.093046 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-556bc6-5bd6db54b9-dzhrj"] Sep 06 12:35:57.168678 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:57.168647 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddf74b53-1c59-4d1e-8e4e-55ba5ef2936f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-556bc6-54789d5666-h7gxf\" (UID: \"ddf74b53-1c59-4d1e-8e4e-55ba5ef2936f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-556bc6-54789d5666-h7gxf" Sep 06 12:35:57.168845 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:57.168717 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxt9n\" (UniqueName: \"kubernetes.io/projected/ddf74b53-1c59-4d1e-8e4e-55ba5ef2936f-kube-api-access-zxt9n\") pod \"payload-pre-processing-e2e-shared-b-w2-556bc6-54789d5666-h7gxf\" (UID: \"ddf74b53-1c59-4d1e-8e4e-55ba5ef2936f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-556bc6-54789d5666-h7gxf" Sep 06 12:35:57.168845 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:57.168790 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdd8d390-2969-4a92-93fc-0a3ad425c583-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-556bc6-5bd6db54b9-dzhrj\" (UID: \"fdd8d390-2969-4a92-93fc-0a3ad425c583\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-556bc6-5bd6db54b9-dzhrj" Sep 06 12:35:57.168966 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:57.168931 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tvqc\" (UniqueName: \"kubernetes.io/projected/fdd8d390-2969-4a92-93fc-0a3ad425c583-kube-api-access-4tvqc\") pod \"payload-processing-e2e-shared-b-w2-556bc6-5bd6db54b9-dzhrj\" (UID: \"fdd8d390-2969-4a92-93fc-0a3ad425c583\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-556bc6-5bd6db54b9-dzhrj" Sep 06 12:35:57.169431 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:57.169414 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddf74b53-1c59-4d1e-8e4e-55ba5ef2936f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-556bc6-54789d5666-h7gxf\" (UID: \"ddf74b53-1c59-4d1e-8e4e-55ba5ef2936f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-556bc6-54789d5666-h7gxf" Sep 06 12:35:57.177591 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:57.177569 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxt9n\" (UniqueName: \"kubernetes.io/projected/ddf74b53-1c59-4d1e-8e4e-55ba5ef2936f-kube-api-access-zxt9n\") pod \"payload-pre-processing-e2e-shared-b-w2-556bc6-54789d5666-h7gxf\" (UID: \"ddf74b53-1c59-4d1e-8e4e-55ba5ef2936f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-556bc6-54789d5666-h7gxf" Sep 06 12:35:57.270484 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:57.270457 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tvqc\" (UniqueName: \"kubernetes.io/projected/fdd8d390-2969-4a92-93fc-0a3ad425c583-kube-api-access-4tvqc\") pod \"payload-processing-e2e-shared-b-w2-556bc6-5bd6db54b9-dzhrj\" (UID: \"fdd8d390-2969-4a92-93fc-0a3ad425c583\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-556bc6-5bd6db54b9-dzhrj" Sep 06 12:35:57.270635 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:57.270538 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdd8d390-2969-4a92-93fc-0a3ad425c583-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-556bc6-5bd6db54b9-dzhrj\" (UID: \"fdd8d390-2969-4a92-93fc-0a3ad425c583\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-556bc6-5bd6db54b9-dzhrj" Sep 06 12:35:57.271246 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:57.271228 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdd8d390-2969-4a92-93fc-0a3ad425c583-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-556bc6-5bd6db54b9-dzhrj\" (UID: \"fdd8d390-2969-4a92-93fc-0a3ad425c583\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-556bc6-5bd6db54b9-dzhrj" Sep 06 12:35:57.278072 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:57.278024 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tvqc\" (UniqueName: \"kubernetes.io/projected/fdd8d390-2969-4a92-93fc-0a3ad425c583-kube-api-access-4tvqc\") pod \"payload-processing-e2e-shared-b-w2-556bc6-5bd6db54b9-dzhrj\" (UID: \"fdd8d390-2969-4a92-93fc-0a3ad425c583\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-556bc6-5bd6db54b9-dzhrj" Sep 06 12:35:57.345122 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:57.345100 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-556bc6-54789d5666-h7gxf" Sep 06 12:35:57.387479 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:57.387453 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-556bc6-5bd6db54b9-dzhrj" Sep 06 12:35:57.618548 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:57.618484 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-556bc6-54789d5666-h7gxf"] Sep 06 12:35:57.621544 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:35:57.621518 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddf74b53_1c59_4d1e_8e4e_55ba5ef2936f.slice/crio-2de01a9750b4167d29fcf2afee9a7961b5c382025b1aeb33df904fb8aa0f841d WatchSource:0}: Error finding container 2de01a9750b4167d29fcf2afee9a7961b5c382025b1aeb33df904fb8aa0f841d: Status 404 returned error can't find the container with id 2de01a9750b4167d29fcf2afee9a7961b5c382025b1aeb33df904fb8aa0f841d Sep 06 12:35:57.671112 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:57.671089 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-556bc6-5bd6db54b9-dzhrj"] Sep 06 12:35:57.671852 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:35:57.671832 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdd8d390_2969_4a92_93fc_0a3ad425c583.slice/crio-4e4d5ccb721f320c02037fa0fe97ea07ce21b5b8c5eb19f8db92f8a6334c5737 WatchSource:0}: Error finding container 4e4d5ccb721f320c02037fa0fe97ea07ce21b5b8c5eb19f8db92f8a6334c5737: Status 404 returned error can't find the container with id 4e4d5ccb721f320c02037fa0fe97ea07ce21b5b8c5eb19f8db92f8a6334c5737 Sep 06 12:35:58.313203 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:58.313169 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-556bc6-54789d5666-h7gxf" event={"ID":"ddf74b53-1c59-4d1e-8e4e-55ba5ef2936f","Type":"ContainerStarted","Data":"34fbe77c378bba4e31834b9d8d1d13b49c3421d3c700966f3251c296e89bce20"} Sep 06 12:35:58.313473 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:58.313443 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-556bc6-54789d5666-h7gxf" Sep 06 12:35:58.313608 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:58.313483 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-556bc6-54789d5666-h7gxf" event={"ID":"ddf74b53-1c59-4d1e-8e4e-55ba5ef2936f","Type":"ContainerStarted","Data":"2de01a9750b4167d29fcf2afee9a7961b5c382025b1aeb33df904fb8aa0f841d"} Sep 06 12:35:58.314976 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:58.314954 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-556bc6-5bd6db54b9-dzhrj" event={"ID":"fdd8d390-2969-4a92-93fc-0a3ad425c583","Type":"ContainerStarted","Data":"5280eb9fe81ab53050476b9f564ee29f0e20f27add8e18ab17f3a3082d5e77e4"} Sep 06 12:35:58.315079 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:58.315002 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-556bc6-5bd6db54b9-dzhrj" event={"ID":"fdd8d390-2969-4a92-93fc-0a3ad425c583","Type":"ContainerStarted","Data":"4e4d5ccb721f320c02037fa0fe97ea07ce21b5b8c5eb19f8db92f8a6334c5737"} Sep 06 12:35:58.315079 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:58.315037 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-556bc6-5bd6db54b9-dzhrj" Sep 06 12:35:58.329931 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:58.329890 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-556bc6-54789d5666-h7gxf" podStartSLOduration=1.329878322 podStartE2EDuration="1.329878322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:35:58.328805276 +0000 UTC m=+1074.184563188" watchObservedRunningTime="2026-09-06 12:35:58.329878322 +0000 UTC m=+1074.185636249" Sep 06 12:35:58.343396 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:35:58.343120 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-556bc6-5bd6db54b9-dzhrj" podStartSLOduration=1.343103675 podStartE2EDuration="1.343103675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:35:58.34286166 +0000 UTC m=+1074.198619573" watchObservedRunningTime="2026-09-06 12:35:58.343103675 +0000 UTC m=+1074.198861562" Sep 06 12:36:07.308934 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:07.308894 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3feb92-64d8bcfbc8-hx7mq" Sep 06 12:36:07.309408 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:07.309027 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3feb92-6ddb5878b5-c6qsd" Sep 06 12:36:09.322495 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:09.322467 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-556bc6-5bd6db54b9-dzhrj" Sep 06 12:36:09.322875 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:09.322549 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-556bc6-54789d5666-h7gxf" Sep 06 12:36:23.040208 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:23.040171 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e6515e-557dcd8dcf-xzsdw"] Sep 06 12:36:23.045625 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:23.045604 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e6515e-557dcd8dcf-xzsdw" Sep 06 12:36:23.048239 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:23.048218 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-e6515e-dockercfg-wb958\"" Sep 06 12:36:23.048342 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:23.048249 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-e6515e\"" Sep 06 12:36:23.053384 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:23.053358 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e6515e-557dcd8dcf-xzsdw"] Sep 06 12:36:23.208654 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:23.208615 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7k6c\" (UniqueName: \"kubernetes.io/projected/84081fd7-e184-44e2-9e76-df6ec0014650-kube-api-access-k7k6c\") pod \"payload-processing-e2e-shared-b-w3-e6515e-557dcd8dcf-xzsdw\" (UID: \"84081fd7-e184-44e2-9e76-df6ec0014650\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e6515e-557dcd8dcf-xzsdw" Sep 06 12:36:23.208859 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:23.208668 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84081fd7-e184-44e2-9e76-df6ec0014650-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-e6515e-557dcd8dcf-xzsdw\" (UID: \"84081fd7-e184-44e2-9e76-df6ec0014650\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e6515e-557dcd8dcf-xzsdw" Sep 06 12:36:23.309918 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:23.309844 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7k6c\" (UniqueName: \"kubernetes.io/projected/84081fd7-e184-44e2-9e76-df6ec0014650-kube-api-access-k7k6c\") pod \"payload-processing-e2e-shared-b-w3-e6515e-557dcd8dcf-xzsdw\" (UID: \"84081fd7-e184-44e2-9e76-df6ec0014650\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e6515e-557dcd8dcf-xzsdw" Sep 06 12:36:23.309918 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:23.309900 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84081fd7-e184-44e2-9e76-df6ec0014650-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-e6515e-557dcd8dcf-xzsdw\" (UID: \"84081fd7-e184-44e2-9e76-df6ec0014650\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e6515e-557dcd8dcf-xzsdw" Sep 06 12:36:23.310553 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:23.310534 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84081fd7-e184-44e2-9e76-df6ec0014650-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-e6515e-557dcd8dcf-xzsdw\" (UID: \"84081fd7-e184-44e2-9e76-df6ec0014650\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e6515e-557dcd8dcf-xzsdw" Sep 06 12:36:23.317729 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:23.317704 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7k6c\" (UniqueName: \"kubernetes.io/projected/84081fd7-e184-44e2-9e76-df6ec0014650-kube-api-access-k7k6c\") pod \"payload-processing-e2e-shared-b-w3-e6515e-557dcd8dcf-xzsdw\" (UID: \"84081fd7-e184-44e2-9e76-df6ec0014650\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e6515e-557dcd8dcf-xzsdw" Sep 06 12:36:23.358725 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:23.358703 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e6515e-557dcd8dcf-xzsdw" Sep 06 12:36:23.637878 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:23.637857 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e6515e-557dcd8dcf-xzsdw"] Sep 06 12:36:23.640258 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:36:23.640219 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84081fd7_e184_44e2_9e76_df6ec0014650.slice/crio-398e0208d7609ccfeeeb858aa6eb5d266b6ade717b54be6f55bc70c31f80856c WatchSource:0}: Error finding container 398e0208d7609ccfeeeb858aa6eb5d266b6ade717b54be6f55bc70c31f80856c: Status 404 returned error can't find the container with id 398e0208d7609ccfeeeb858aa6eb5d266b6ade717b54be6f55bc70c31f80856c Sep 06 12:36:24.458614 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:24.458578 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e6515e-557dcd8dcf-xzsdw" event={"ID":"84081fd7-e184-44e2-9e76-df6ec0014650","Type":"ContainerStarted","Data":"6fdf0e5d29914c65637f9801dcfc5b279c7886b7c285290c556fd482d91887e8"} Sep 06 12:36:24.459088 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:24.458630 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e6515e-557dcd8dcf-xzsdw" event={"ID":"84081fd7-e184-44e2-9e76-df6ec0014650","Type":"ContainerStarted","Data":"398e0208d7609ccfeeeb858aa6eb5d266b6ade717b54be6f55bc70c31f80856c"} Sep 06 12:36:24.459088 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:24.458663 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e6515e-557dcd8dcf-xzsdw" Sep 06 12:36:24.474356 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:24.474308 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e6515e-557dcd8dcf-xzsdw" podStartSLOduration=1.474294742 podStartE2EDuration="1.474294742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:36:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:36:24.474246999 +0000 UTC m=+1100.330004886" watchObservedRunningTime="2026-09-06 12:36:24.474294742 +0000 UTC m=+1100.330052628" Sep 06 12:36:35.465818 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:35.465784 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e6515e-557dcd8dcf-xzsdw" Sep 06 12:36:38.966510 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:38.966466 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-48920d6f-5cd779f8b-f79vt"] Sep 06 12:36:38.970638 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:38.970613 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-48920d6f-5cd779f8b-f79vt" Sep 06 12:36:38.972891 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:38.972875 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-48920d6f\"" Sep 06 12:36:38.981517 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:38.981497 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-48920d6f-5cd779f8b-f79vt"] Sep 06 12:36:39.013887 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:39.013861 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-48920d6f-5bf6844c49-7dlkx"] Sep 06 12:36:39.017444 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:39.017430 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-48920d6f-5bf6844c49-7dlkx" Sep 06 12:36:39.019795 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:39.019767 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-48920d6f-dockercfg-mcwzz\"" Sep 06 12:36:39.028435 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:39.028411 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-48920d6f-5bf6844c49-7dlkx"] Sep 06 12:36:39.050729 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:39.050700 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rksx5\" (UniqueName: \"kubernetes.io/projected/006442ee-f84e-4d7d-99bf-0a487aba0b79-kube-api-access-rksx5\") pod \"payload-processing-e2e-mt-48920d6f-5bf6844c49-7dlkx\" (UID: \"006442ee-f84e-4d7d-99bf-0a487aba0b79\") " pod="openshift-ingress/payload-processing-e2e-mt-48920d6f-5bf6844c49-7dlkx" Sep 06 12:36:39.050847 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:39.050746 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/006442ee-f84e-4d7d-99bf-0a487aba0b79-plugins-config-volume\") pod \"payload-processing-e2e-mt-48920d6f-5bf6844c49-7dlkx\" (UID: \"006442ee-f84e-4d7d-99bf-0a487aba0b79\") " pod="openshift-ingress/payload-processing-e2e-mt-48920d6f-5bf6844c49-7dlkx" Sep 06 12:36:39.050891 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:39.050846 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cde4c450-f480-4ae4-b912-c7ce440f9be8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-48920d6f-5cd779f8b-f79vt\" (UID: \"cde4c450-f480-4ae4-b912-c7ce440f9be8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-48920d6f-5cd779f8b-f79vt" Sep 06 12:36:39.050927 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:39.050894 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk5cs\" (UniqueName: \"kubernetes.io/projected/cde4c450-f480-4ae4-b912-c7ce440f9be8-kube-api-access-tk5cs\") pod \"payload-pre-processing-e2e-mt-48920d6f-5cd779f8b-f79vt\" (UID: \"cde4c450-f480-4ae4-b912-c7ce440f9be8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-48920d6f-5cd779f8b-f79vt" Sep 06 12:36:39.151873 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:39.151838 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rksx5\" (UniqueName: \"kubernetes.io/projected/006442ee-f84e-4d7d-99bf-0a487aba0b79-kube-api-access-rksx5\") pod \"payload-processing-e2e-mt-48920d6f-5bf6844c49-7dlkx\" (UID: \"006442ee-f84e-4d7d-99bf-0a487aba0b79\") " pod="openshift-ingress/payload-processing-e2e-mt-48920d6f-5bf6844c49-7dlkx" Sep 06 12:36:39.151873 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:39.151873 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/006442ee-f84e-4d7d-99bf-0a487aba0b79-plugins-config-volume\") pod \"payload-processing-e2e-mt-48920d6f-5bf6844c49-7dlkx\" (UID: \"006442ee-f84e-4d7d-99bf-0a487aba0b79\") " pod="openshift-ingress/payload-processing-e2e-mt-48920d6f-5bf6844c49-7dlkx" Sep 06 12:36:39.152103 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:39.151917 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cde4c450-f480-4ae4-b912-c7ce440f9be8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-48920d6f-5cd779f8b-f79vt\" (UID: \"cde4c450-f480-4ae4-b912-c7ce440f9be8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-48920d6f-5cd779f8b-f79vt" Sep 06 12:36:39.152103 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:39.151946 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk5cs\" (UniqueName: \"kubernetes.io/projected/cde4c450-f480-4ae4-b912-c7ce440f9be8-kube-api-access-tk5cs\") pod \"payload-pre-processing-e2e-mt-48920d6f-5cd779f8b-f79vt\" (UID: \"cde4c450-f480-4ae4-b912-c7ce440f9be8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-48920d6f-5cd779f8b-f79vt" Sep 06 12:36:39.152690 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:39.152665 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cde4c450-f480-4ae4-b912-c7ce440f9be8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-48920d6f-5cd779f8b-f79vt\" (UID: \"cde4c450-f480-4ae4-b912-c7ce440f9be8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-48920d6f-5cd779f8b-f79vt" Sep 06 12:36:39.152805 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:39.152787 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/006442ee-f84e-4d7d-99bf-0a487aba0b79-plugins-config-volume\") pod \"payload-processing-e2e-mt-48920d6f-5bf6844c49-7dlkx\" (UID: \"006442ee-f84e-4d7d-99bf-0a487aba0b79\") " pod="openshift-ingress/payload-processing-e2e-mt-48920d6f-5bf6844c49-7dlkx" Sep 06 12:36:39.160855 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:39.160830 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk5cs\" (UniqueName: \"kubernetes.io/projected/cde4c450-f480-4ae4-b912-c7ce440f9be8-kube-api-access-tk5cs\") pod \"payload-pre-processing-e2e-mt-48920d6f-5cd779f8b-f79vt\" (UID: \"cde4c450-f480-4ae4-b912-c7ce440f9be8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-48920d6f-5cd779f8b-f79vt" Sep 06 12:36:39.161127 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:39.161088 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rksx5\" (UniqueName: \"kubernetes.io/projected/006442ee-f84e-4d7d-99bf-0a487aba0b79-kube-api-access-rksx5\") pod \"payload-processing-e2e-mt-48920d6f-5bf6844c49-7dlkx\" (UID: \"006442ee-f84e-4d7d-99bf-0a487aba0b79\") " pod="openshift-ingress/payload-processing-e2e-mt-48920d6f-5bf6844c49-7dlkx" Sep 06 12:36:39.281814 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:39.281719 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-48920d6f-5cd779f8b-f79vt" Sep 06 12:36:39.328832 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:39.328732 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-48920d6f-5bf6844c49-7dlkx" Sep 06 12:36:39.553010 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:39.552981 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-48920d6f-5cd779f8b-f79vt"] Sep 06 12:36:39.554711 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:36:39.554686 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde4c450_f480_4ae4_b912_c7ce440f9be8.slice/crio-fbdcd852e2d6d7bdbac380a3e57fb6204780364da6ea4cf423e0b414d38a3b3a WatchSource:0}: Error finding container fbdcd852e2d6d7bdbac380a3e57fb6204780364da6ea4cf423e0b414d38a3b3a: Status 404 returned error can't find the container with id fbdcd852e2d6d7bdbac380a3e57fb6204780364da6ea4cf423e0b414d38a3b3a Sep 06 12:36:39.619177 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:39.619156 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-48920d6f-5bf6844c49-7dlkx"] Sep 06 12:36:39.620665 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:36:39.620635 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod006442ee_f84e_4d7d_99bf_0a487aba0b79.slice/crio-adf95f02174a0fa38c4cc658fca7df98232d8c7d585cc350a71c8567a811f873 WatchSource:0}: Error finding container adf95f02174a0fa38c4cc658fca7df98232d8c7d585cc350a71c8567a811f873: Status 404 returned error can't find the container with id adf95f02174a0fa38c4cc658fca7df98232d8c7d585cc350a71c8567a811f873 Sep 06 12:36:40.562601 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:40.562562 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-48920d6f-5bf6844c49-7dlkx" event={"ID":"006442ee-f84e-4d7d-99bf-0a487aba0b79","Type":"ContainerStarted","Data":"dfb59895210f13c82390bca650f138ce2d2c2c283b56a9e24882de258de5babb"} Sep 06 12:36:40.563091 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:40.562623 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-48920d6f-5bf6844c49-7dlkx" event={"ID":"006442ee-f84e-4d7d-99bf-0a487aba0b79","Type":"ContainerStarted","Data":"adf95f02174a0fa38c4cc658fca7df98232d8c7d585cc350a71c8567a811f873"} Sep 06 12:36:40.563091 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:40.562673 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-48920d6f-5bf6844c49-7dlkx" Sep 06 12:36:40.564375 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:40.564356 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-48920d6f-5cd779f8b-f79vt" event={"ID":"cde4c450-f480-4ae4-b912-c7ce440f9be8","Type":"ContainerStarted","Data":"711c26615cff05e77c903864ccb63ffae75ddcb438ce38d52141feac96b202cf"} Sep 06 12:36:40.564493 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:40.564389 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-48920d6f-5cd779f8b-f79vt" event={"ID":"cde4c450-f480-4ae4-b912-c7ce440f9be8","Type":"ContainerStarted","Data":"fbdcd852e2d6d7bdbac380a3e57fb6204780364da6ea4cf423e0b414d38a3b3a"} Sep 06 12:36:40.564493 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:40.564418 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-48920d6f-5cd779f8b-f79vt" Sep 06 12:36:40.578451 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:40.578407 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-48920d6f-5bf6844c49-7dlkx" podStartSLOduration=2.578394458 podStartE2EDuration="2.578394458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:36:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:36:40.577424842 +0000 UTC m=+1116.433182770" watchObservedRunningTime="2026-09-06 12:36:40.578394458 +0000 UTC m=+1116.434152344" Sep 06 12:36:40.592277 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:40.592231 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-48920d6f-5cd779f8b-f79vt" podStartSLOduration=2.592217556 podStartE2EDuration="2.592217556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:36:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:36:40.591336213 +0000 UTC m=+1116.447094126" watchObservedRunningTime="2026-09-06 12:36:40.592217556 +0000 UTC m=+1116.447975497" Sep 06 12:36:51.572966 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:51.572928 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-48920d6f-5bf6844c49-7dlkx" Sep 06 12:36:51.573414 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:36:51.573004 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-48920d6f-5cd779f8b-f79vt" Sep 06 12:38:04.667087 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:04.667061 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98vz4_4103c09f-dd43-419d-92e5-2374af5b5e01/ovn-acl-logging/0.log" Sep 06 12:38:04.671050 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:04.671029 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98vz4_4103c09f-dd43-419d-92e5-2374af5b5e01/ovn-acl-logging/0.log" Sep 06 12:38:22.364982 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:22.364945 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-48920d6f-5bf6844c49-7dlkx"] Sep 06 12:38:22.365494 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:22.365217 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-48920d6f-5bf6844c49-7dlkx" podUID="006442ee-f84e-4d7d-99bf-0a487aba0b79" containerName="payload-processing" containerID="cri-o://dfb59895210f13c82390bca650f138ce2d2c2c283b56a9e24882de258de5babb" gracePeriod=30 Sep 06 12:38:22.371849 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:22.371817 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-48920d6f-5cd779f8b-f79vt"] Sep 06 12:38:22.372088 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:22.372055 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-48920d6f-5cd779f8b-f79vt" podUID="cde4c450-f480-4ae4-b912-c7ce440f9be8" containerName="payload-pre-processing" containerID="cri-o://711c26615cff05e77c903864ccb63ffae75ddcb438ce38d52141feac96b202cf" gracePeriod=30 Sep 06 12:38:22.648456 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:22.648432 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-48920d6f-5bf6844c49-7dlkx" Sep 06 12:38:22.661265 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:22.661245 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-48920d6f-5cd779f8b-f79vt" Sep 06 12:38:22.746326 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:22.746295 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cde4c450-f480-4ae4-b912-c7ce440f9be8-plugins-config-volume\") pod \"cde4c450-f480-4ae4-b912-c7ce440f9be8\" (UID: \"cde4c450-f480-4ae4-b912-c7ce440f9be8\") " Sep 06 12:38:22.746490 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:22.746362 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rksx5\" (UniqueName: \"kubernetes.io/projected/006442ee-f84e-4d7d-99bf-0a487aba0b79-kube-api-access-rksx5\") pod \"006442ee-f84e-4d7d-99bf-0a487aba0b79\" (UID: \"006442ee-f84e-4d7d-99bf-0a487aba0b79\") " Sep 06 12:38:22.746490 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:22.746390 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tk5cs\" (UniqueName: \"kubernetes.io/projected/cde4c450-f480-4ae4-b912-c7ce440f9be8-kube-api-access-tk5cs\") pod \"cde4c450-f480-4ae4-b912-c7ce440f9be8\" (UID: \"cde4c450-f480-4ae4-b912-c7ce440f9be8\") " Sep 06 12:38:22.746490 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:22.746446 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/006442ee-f84e-4d7d-99bf-0a487aba0b79-plugins-config-volume\") pod \"006442ee-f84e-4d7d-99bf-0a487aba0b79\" (UID: \"006442ee-f84e-4d7d-99bf-0a487aba0b79\") " Sep 06 12:38:22.746883 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:22.746852 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cde4c450-f480-4ae4-b912-c7ce440f9be8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cde4c450-f480-4ae4-b912-c7ce440f9be8" (UID: "cde4c450-f480-4ae4-b912-c7ce440f9be8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:38:22.747018 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:22.746859 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/006442ee-f84e-4d7d-99bf-0a487aba0b79-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "006442ee-f84e-4d7d-99bf-0a487aba0b79" (UID: "006442ee-f84e-4d7d-99bf-0a487aba0b79"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:38:22.748586 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:22.748562 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde4c450-f480-4ae4-b912-c7ce440f9be8-kube-api-access-tk5cs" (OuterVolumeSpecName: "kube-api-access-tk5cs") pod "cde4c450-f480-4ae4-b912-c7ce440f9be8" (UID: "cde4c450-f480-4ae4-b912-c7ce440f9be8"). InnerVolumeSpecName "kube-api-access-tk5cs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:38:22.748650 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:22.748598 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/006442ee-f84e-4d7d-99bf-0a487aba0b79-kube-api-access-rksx5" (OuterVolumeSpecName: "kube-api-access-rksx5") pod "006442ee-f84e-4d7d-99bf-0a487aba0b79" (UID: "006442ee-f84e-4d7d-99bf-0a487aba0b79"). InnerVolumeSpecName "kube-api-access-rksx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:38:22.847970 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:22.847949 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/006442ee-f84e-4d7d-99bf-0a487aba0b79-plugins-config-volume\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:38:22.847970 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:22.847970 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cde4c450-f480-4ae4-b912-c7ce440f9be8-plugins-config-volume\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:38:22.848103 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:22.847979 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rksx5\" (UniqueName: \"kubernetes.io/projected/006442ee-f84e-4d7d-99bf-0a487aba0b79-kube-api-access-rksx5\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:38:22.848103 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:22.847988 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk5cs\" (UniqueName: \"kubernetes.io/projected/cde4c450-f480-4ae4-b912-c7ce440f9be8-kube-api-access-tk5cs\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:38:23.121351 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:23.121321 2576 generic.go:356] "Generic (PLEG): container finished" podID="006442ee-f84e-4d7d-99bf-0a487aba0b79" containerID="dfb59895210f13c82390bca650f138ce2d2c2c283b56a9e24882de258de5babb" exitCode=0 Sep 06 12:38:23.121471 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:23.121371 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-48920d6f-5bf6844c49-7dlkx" Sep 06 12:38:23.121471 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:23.121403 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-48920d6f-5bf6844c49-7dlkx" event={"ID":"006442ee-f84e-4d7d-99bf-0a487aba0b79","Type":"ContainerDied","Data":"dfb59895210f13c82390bca650f138ce2d2c2c283b56a9e24882de258de5babb"} Sep 06 12:38:23.121471 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:23.121463 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-48920d6f-5bf6844c49-7dlkx" event={"ID":"006442ee-f84e-4d7d-99bf-0a487aba0b79","Type":"ContainerDied","Data":"adf95f02174a0fa38c4cc658fca7df98232d8c7d585cc350a71c8567a811f873"} Sep 06 12:38:23.121603 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:23.121492 2576 scope.go:117] "RemoveContainer" containerID="dfb59895210f13c82390bca650f138ce2d2c2c283b56a9e24882de258de5babb" Sep 06 12:38:23.122990 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:23.122962 2576 generic.go:356] "Generic (PLEG): container finished" podID="cde4c450-f480-4ae4-b912-c7ce440f9be8" containerID="711c26615cff05e77c903864ccb63ffae75ddcb438ce38d52141feac96b202cf" exitCode=0 Sep 06 12:38:23.123109 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:23.123023 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-48920d6f-5cd779f8b-f79vt" Sep 06 12:38:23.123109 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:23.123053 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-48920d6f-5cd779f8b-f79vt" event={"ID":"cde4c450-f480-4ae4-b912-c7ce440f9be8","Type":"ContainerDied","Data":"711c26615cff05e77c903864ccb63ffae75ddcb438ce38d52141feac96b202cf"} Sep 06 12:38:23.123210 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:23.123112 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-48920d6f-5cd779f8b-f79vt" event={"ID":"cde4c450-f480-4ae4-b912-c7ce440f9be8","Type":"ContainerDied","Data":"fbdcd852e2d6d7bdbac380a3e57fb6204780364da6ea4cf423e0b414d38a3b3a"} Sep 06 12:38:23.131377 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:23.131356 2576 scope.go:117] "RemoveContainer" containerID="dfb59895210f13c82390bca650f138ce2d2c2c283b56a9e24882de258de5babb" Sep 06 12:38:23.131790 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:38:23.131643 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfb59895210f13c82390bca650f138ce2d2c2c283b56a9e24882de258de5babb\": container with ID starting with dfb59895210f13c82390bca650f138ce2d2c2c283b56a9e24882de258de5babb not found: ID does not exist" containerID="dfb59895210f13c82390bca650f138ce2d2c2c283b56a9e24882de258de5babb" Sep 06 12:38:23.131790 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:23.131669 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfb59895210f13c82390bca650f138ce2d2c2c283b56a9e24882de258de5babb"} err="failed to get container status \"dfb59895210f13c82390bca650f138ce2d2c2c283b56a9e24882de258de5babb\": rpc error: code = NotFound desc = could not find container \"dfb59895210f13c82390bca650f138ce2d2c2c283b56a9e24882de258de5babb\": container with ID starting with dfb59895210f13c82390bca650f138ce2d2c2c283b56a9e24882de258de5babb not found: ID does not exist" Sep 06 12:38:23.131790 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:23.131685 2576 scope.go:117] "RemoveContainer" containerID="711c26615cff05e77c903864ccb63ffae75ddcb438ce38d52141feac96b202cf" Sep 06 12:38:23.140522 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:23.140507 2576 scope.go:117] "RemoveContainer" containerID="711c26615cff05e77c903864ccb63ffae75ddcb438ce38d52141feac96b202cf" Sep 06 12:38:23.140845 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:38:23.140810 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"711c26615cff05e77c903864ccb63ffae75ddcb438ce38d52141feac96b202cf\": container with ID starting with 711c26615cff05e77c903864ccb63ffae75ddcb438ce38d52141feac96b202cf not found: ID does not exist" containerID="711c26615cff05e77c903864ccb63ffae75ddcb438ce38d52141feac96b202cf" Sep 06 12:38:23.140898 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:23.140851 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"711c26615cff05e77c903864ccb63ffae75ddcb438ce38d52141feac96b202cf"} err="failed to get container status \"711c26615cff05e77c903864ccb63ffae75ddcb438ce38d52141feac96b202cf\": rpc error: code = NotFound desc = could not find container \"711c26615cff05e77c903864ccb63ffae75ddcb438ce38d52141feac96b202cf\": container with ID starting with 711c26615cff05e77c903864ccb63ffae75ddcb438ce38d52141feac96b202cf not found: ID does not exist" Sep 06 12:38:23.149162 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:23.149141 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-48920d6f-5cd779f8b-f79vt"] Sep 06 12:38:23.151145 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:23.151126 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-48920d6f-5cd779f8b-f79vt"] Sep 06 12:38:23.162919 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:23.162897 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-48920d6f-5bf6844c49-7dlkx"] Sep 06 12:38:23.166673 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:23.166651 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-48920d6f-5bf6844c49-7dlkx"] Sep 06 12:38:24.655417 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:24.655384 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="006442ee-f84e-4d7d-99bf-0a487aba0b79" path="/var/lib/kubelet/pods/006442ee-f84e-4d7d-99bf-0a487aba0b79/volumes" Sep 06 12:38:24.655820 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:38:24.655724 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cde4c450-f480-4ae4-b912-c7ce440f9be8" path="/var/lib/kubelet/pods/cde4c450-f480-4ae4-b912-c7ce440f9be8/volumes" Sep 06 12:39:56.048719 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:39:56.048688 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a898e1-7694986778-96tls"] Sep 06 12:39:56.049220 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:39:56.049150 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="006442ee-f84e-4d7d-99bf-0a487aba0b79" containerName="payload-processing" Sep 06 12:39:56.049220 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:39:56.049163 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="006442ee-f84e-4d7d-99bf-0a487aba0b79" containerName="payload-processing" Sep 06 12:39:56.049220 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:39:56.049181 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cde4c450-f480-4ae4-b912-c7ce440f9be8" containerName="payload-pre-processing" Sep 06 12:39:56.049220 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:39:56.049186 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cde4c450-f480-4ae4-b912-c7ce440f9be8" containerName="payload-pre-processing" Sep 06 12:39:56.049375 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:39:56.049252 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="006442ee-f84e-4d7d-99bf-0a487aba0b79" containerName="payload-processing" Sep 06 12:39:56.049375 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:39:56.049269 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cde4c450-f480-4ae4-b912-c7ce440f9be8" containerName="payload-pre-processing" Sep 06 12:39:56.052100 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:39:56.052085 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a898e1-7694986778-96tls" Sep 06 12:39:56.054742 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:39:56.054722 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a898e1\"" Sep 06 12:39:56.055049 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:39:56.055032 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-a898e1-dockercfg-bqptj\"" Sep 06 12:39:56.061872 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:39:56.061842 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppmk2\" (UniqueName: \"kubernetes.io/projected/2461899c-c473-4831-86c9-d9294de1c488-kube-api-access-ppmk2\") pod \"payload-pre-processing-e2e-api-a-a898e1-7694986778-96tls\" (UID: \"2461899c-c473-4831-86c9-d9294de1c488\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a898e1-7694986778-96tls" Sep 06 12:39:56.061964 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:39:56.061890 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2461899c-c473-4831-86c9-d9294de1c488-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a898e1-7694986778-96tls\" (UID: \"2461899c-c473-4831-86c9-d9294de1c488\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a898e1-7694986778-96tls" Sep 06 12:39:56.072435 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:39:56.072414 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a898e1-7694986778-96tls"] Sep 06 12:39:56.099627 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:39:56.099603 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a898e1-77b7d88c6c-5v7wl"] Sep 06 12:39:56.103505 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:39:56.103488 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a898e1-77b7d88c6c-5v7wl" Sep 06 12:39:56.122120 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:39:56.122087 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a898e1-77b7d88c6c-5v7wl"] Sep 06 12:39:56.163212 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:39:56.163171 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2461899c-c473-4831-86c9-d9294de1c488-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a898e1-7694986778-96tls\" (UID: \"2461899c-c473-4831-86c9-d9294de1c488\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a898e1-7694986778-96tls" Sep 06 12:39:56.163212 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:39:56.163212 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e49e3895-ebee-48ec-ab1b-1bca838c4db0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a898e1-77b7d88c6c-5v7wl\" (UID: \"e49e3895-ebee-48ec-ab1b-1bca838c4db0\") " pod="openshift-ingress/payload-processing-e2e-api-a-a898e1-77b7d88c6c-5v7wl" Sep 06 12:39:56.163404 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:39:56.163293 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z544h\" (UniqueName: \"kubernetes.io/projected/e49e3895-ebee-48ec-ab1b-1bca838c4db0-kube-api-access-z544h\") pod \"payload-processing-e2e-api-a-a898e1-77b7d88c6c-5v7wl\" (UID: \"e49e3895-ebee-48ec-ab1b-1bca838c4db0\") " pod="openshift-ingress/payload-processing-e2e-api-a-a898e1-77b7d88c6c-5v7wl" Sep 06 12:39:56.163404 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:39:56.163374 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppmk2\" (UniqueName: \"kubernetes.io/projected/2461899c-c473-4831-86c9-d9294de1c488-kube-api-access-ppmk2\") pod \"payload-pre-processing-e2e-api-a-a898e1-7694986778-96tls\" (UID: \"2461899c-c473-4831-86c9-d9294de1c488\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a898e1-7694986778-96tls" Sep 06 12:39:56.164037 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:39:56.164018 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2461899c-c473-4831-86c9-d9294de1c488-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a898e1-7694986778-96tls\" (UID: \"2461899c-c473-4831-86c9-d9294de1c488\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a898e1-7694986778-96tls" Sep 06 12:39:56.177664 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:39:56.177636 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppmk2\" (UniqueName: \"kubernetes.io/projected/2461899c-c473-4831-86c9-d9294de1c488-kube-api-access-ppmk2\") pod \"payload-pre-processing-e2e-api-a-a898e1-7694986778-96tls\" (UID: \"2461899c-c473-4831-86c9-d9294de1c488\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a898e1-7694986778-96tls" Sep 06 12:39:56.264776 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:39:56.264733 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z544h\" (UniqueName: \"kubernetes.io/projected/e49e3895-ebee-48ec-ab1b-1bca838c4db0-kube-api-access-z544h\") pod \"payload-processing-e2e-api-a-a898e1-77b7d88c6c-5v7wl\" (UID: \"e49e3895-ebee-48ec-ab1b-1bca838c4db0\") " pod="openshift-ingress/payload-processing-e2e-api-a-a898e1-77b7d88c6c-5v7wl" Sep 06 12:39:56.264883 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:39:56.264818 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e49e3895-ebee-48ec-ab1b-1bca838c4db0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a898e1-77b7d88c6c-5v7wl\" (UID: \"e49e3895-ebee-48ec-ab1b-1bca838c4db0\") " pod="openshift-ingress/payload-processing-e2e-api-a-a898e1-77b7d88c6c-5v7wl" Sep 06 12:39:56.265630 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:39:56.265598 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e49e3895-ebee-48ec-ab1b-1bca838c4db0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a898e1-77b7d88c6c-5v7wl\" (UID: \"e49e3895-ebee-48ec-ab1b-1bca838c4db0\") " pod="openshift-ingress/payload-processing-e2e-api-a-a898e1-77b7d88c6c-5v7wl" Sep 06 12:39:56.273267 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:39:56.273243 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z544h\" (UniqueName: \"kubernetes.io/projected/e49e3895-ebee-48ec-ab1b-1bca838c4db0-kube-api-access-z544h\") pod \"payload-processing-e2e-api-a-a898e1-77b7d88c6c-5v7wl\" (UID: \"e49e3895-ebee-48ec-ab1b-1bca838c4db0\") " pod="openshift-ingress/payload-processing-e2e-api-a-a898e1-77b7d88c6c-5v7wl" Sep 06 12:39:56.363138 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:39:56.363072 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a898e1-7694986778-96tls" Sep 06 12:39:56.413782 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:39:56.413717 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a898e1-77b7d88c6c-5v7wl" Sep 06 12:39:56.632732 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:39:56.632709 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a898e1-7694986778-96tls"] Sep 06 12:39:56.634710 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:39:56.634685 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2461899c_c473_4831_86c9_d9294de1c488.slice/crio-624f9aa2f274667131b230fb7c54bd4cba51b43510468b68439f6534fb706d22 WatchSource:0}: Error finding container 624f9aa2f274667131b230fb7c54bd4cba51b43510468b68439f6534fb706d22: Status 404 returned error can't find the container with id 624f9aa2f274667131b230fb7c54bd4cba51b43510468b68439f6534fb706d22 Sep 06 12:39:56.699436 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:39:56.699414 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a898e1-77b7d88c6c-5v7wl"] Sep 06 12:39:56.701606 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:39:56.701568 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode49e3895_ebee_48ec_ab1b_1bca838c4db0.slice/crio-23058cf9db7f944ed2549cc61c0698e7bf380a3902f0caad46556f258498588d WatchSource:0}: Error finding container 23058cf9db7f944ed2549cc61c0698e7bf380a3902f0caad46556f258498588d: Status 404 returned error can't find the container with id 23058cf9db7f944ed2549cc61c0698e7bf380a3902f0caad46556f258498588d Sep 06 12:39:57.633403 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:39:57.633364 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a898e1-77b7d88c6c-5v7wl" event={"ID":"e49e3895-ebee-48ec-ab1b-1bca838c4db0","Type":"ContainerStarted","Data":"7a9c48671a355fc7a861f36c9d70cd0060f5f1e2cad42ee05ff47c56687f72a7"} Sep 06 12:39:57.633897 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:39:57.633430 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a898e1-77b7d88c6c-5v7wl" event={"ID":"e49e3895-ebee-48ec-ab1b-1bca838c4db0","Type":"ContainerStarted","Data":"23058cf9db7f944ed2549cc61c0698e7bf380a3902f0caad46556f258498588d"} Sep 06 12:39:57.633897 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:39:57.633560 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-a898e1-77b7d88c6c-5v7wl" Sep 06 12:39:57.635504 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:39:57.635484 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a898e1-7694986778-96tls" event={"ID":"2461899c-c473-4831-86c9-d9294de1c488","Type":"ContainerStarted","Data":"b19279ee05c1f9634d3e61cac844be37ca4617d1b8f284f29ad1f156eba37a73"} Sep 06 12:39:57.635664 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:39:57.635517 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a898e1-7694986778-96tls" event={"ID":"2461899c-c473-4831-86c9-d9294de1c488","Type":"ContainerStarted","Data":"624f9aa2f274667131b230fb7c54bd4cba51b43510468b68439f6534fb706d22"} Sep 06 12:39:57.635664 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:39:57.635606 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a898e1-7694986778-96tls" Sep 06 12:39:57.650020 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:39:57.649975 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-a898e1-77b7d88c6c-5v7wl" podStartSLOduration=1.649962866 podStartE2EDuration="1.649962866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:39:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:39:57.647735593 +0000 UTC m=+1313.503493481" watchObservedRunningTime="2026-09-06 12:39:57.649962866 +0000 UTC m=+1313.505720753" Sep 06 12:39:57.664833 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:39:57.664788 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a898e1-7694986778-96tls" podStartSLOduration=1.664776266 podStartE2EDuration="1.664776266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:39:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:39:57.662152676 +0000 UTC m=+1313.517910579" watchObservedRunningTime="2026-09-06 12:39:57.664776266 +0000 UTC m=+1313.520534150" Sep 06 12:40:08.642656 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:40:08.642628 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a898e1-7694986778-96tls" Sep 06 12:40:08.643142 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:40:08.642704 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-a898e1-77b7d88c6c-5v7wl" Sep 06 12:40:24.717834 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:40:24.717800 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-969fdf-596bdcbfcc-6qjqs"] Sep 06 12:40:24.721353 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:40:24.721338 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-969fdf-596bdcbfcc-6qjqs" Sep 06 12:40:24.723995 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:40:24.723974 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-969fdf-dockercfg-ptj8z\"" Sep 06 12:40:24.724204 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:40:24.724175 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-969fdf\"" Sep 06 12:40:24.734881 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:40:24.734861 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-969fdf-596bdcbfcc-6qjqs"] Sep 06 12:40:24.815130 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:40:24.815103 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a49117b9-9819-4d46-a236-5362d40baa64-plugins-config-volume\") pod \"payload-processing-e2e-api-b-969fdf-596bdcbfcc-6qjqs\" (UID: \"a49117b9-9819-4d46-a236-5362d40baa64\") " pod="openshift-ingress/payload-processing-e2e-api-b-969fdf-596bdcbfcc-6qjqs" Sep 06 12:40:24.815247 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:40:24.815137 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r7r4\" (UniqueName: \"kubernetes.io/projected/a49117b9-9819-4d46-a236-5362d40baa64-kube-api-access-8r7r4\") pod \"payload-processing-e2e-api-b-969fdf-596bdcbfcc-6qjqs\" (UID: \"a49117b9-9819-4d46-a236-5362d40baa64\") " pod="openshift-ingress/payload-processing-e2e-api-b-969fdf-596bdcbfcc-6qjqs" Sep 06 12:40:24.916446 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:40:24.916414 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a49117b9-9819-4d46-a236-5362d40baa64-plugins-config-volume\") pod \"payload-processing-e2e-api-b-969fdf-596bdcbfcc-6qjqs\" (UID: \"a49117b9-9819-4d46-a236-5362d40baa64\") " pod="openshift-ingress/payload-processing-e2e-api-b-969fdf-596bdcbfcc-6qjqs" Sep 06 12:40:24.916620 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:40:24.916453 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r7r4\" (UniqueName: \"kubernetes.io/projected/a49117b9-9819-4d46-a236-5362d40baa64-kube-api-access-8r7r4\") pod \"payload-processing-e2e-api-b-969fdf-596bdcbfcc-6qjqs\" (UID: \"a49117b9-9819-4d46-a236-5362d40baa64\") " pod="openshift-ingress/payload-processing-e2e-api-b-969fdf-596bdcbfcc-6qjqs" Sep 06 12:40:24.917197 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:40:24.917171 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a49117b9-9819-4d46-a236-5362d40baa64-plugins-config-volume\") pod \"payload-processing-e2e-api-b-969fdf-596bdcbfcc-6qjqs\" (UID: \"a49117b9-9819-4d46-a236-5362d40baa64\") " pod="openshift-ingress/payload-processing-e2e-api-b-969fdf-596bdcbfcc-6qjqs" Sep 06 12:40:24.930308 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:40:24.930287 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r7r4\" (UniqueName: \"kubernetes.io/projected/a49117b9-9819-4d46-a236-5362d40baa64-kube-api-access-8r7r4\") pod \"payload-processing-e2e-api-b-969fdf-596bdcbfcc-6qjqs\" (UID: \"a49117b9-9819-4d46-a236-5362d40baa64\") " pod="openshift-ingress/payload-processing-e2e-api-b-969fdf-596bdcbfcc-6qjqs" Sep 06 12:40:25.032365 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:40:25.032300 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-969fdf-596bdcbfcc-6qjqs" Sep 06 12:40:25.305305 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:40:25.305211 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-969fdf-596bdcbfcc-6qjqs"] Sep 06 12:40:25.308324 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:40:25.308285 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda49117b9_9819_4d46_a236_5362d40baa64.slice/crio-6a94405a598f01f4bb8534d9bc10e8b6c182325400ecfe70932f884213a0b0dd WatchSource:0}: Error finding container 6a94405a598f01f4bb8534d9bc10e8b6c182325400ecfe70932f884213a0b0dd: Status 404 returned error can't find the container with id 6a94405a598f01f4bb8534d9bc10e8b6c182325400ecfe70932f884213a0b0dd Sep 06 12:40:25.787516 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:40:25.787474 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-969fdf-596bdcbfcc-6qjqs" event={"ID":"a49117b9-9819-4d46-a236-5362d40baa64","Type":"ContainerStarted","Data":"a67be1401bc3469abe171211e7a1f6f3013489725e3ef381a7629c1191be3aaa"} Sep 06 12:40:25.787937 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:40:25.787534 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-969fdf-596bdcbfcc-6qjqs" event={"ID":"a49117b9-9819-4d46-a236-5362d40baa64","Type":"ContainerStarted","Data":"6a94405a598f01f4bb8534d9bc10e8b6c182325400ecfe70932f884213a0b0dd"} Sep 06 12:40:25.787937 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:40:25.787584 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-969fdf-596bdcbfcc-6qjqs" Sep 06 12:40:25.804812 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:40:25.804743 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-969fdf-596bdcbfcc-6qjqs" podStartSLOduration=1.804725506 podStartE2EDuration="1.804725506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:40:25.802524781 +0000 UTC m=+1341.658282686" watchObservedRunningTime="2026-09-06 12:40:25.804725506 +0000 UTC m=+1341.660483394" Sep 06 12:40:36.794232 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:40:36.794194 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-969fdf-596bdcbfcc-6qjqs" Sep 06 12:41:06.049698 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:06.049657 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a898e1-77b7d88c6c-5v7wl"] Sep 06 12:41:06.050178 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:06.049944 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-a898e1-77b7d88c6c-5v7wl" podUID="e49e3895-ebee-48ec-ab1b-1bca838c4db0" containerName="payload-processing" containerID="cri-o://7a9c48671a355fc7a861f36c9d70cd0060f5f1e2cad42ee05ff47c56687f72a7" gracePeriod=30 Sep 06 12:41:06.057598 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:06.057568 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a898e1-7694986778-96tls"] Sep 06 12:41:06.057836 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:06.057808 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a898e1-7694986778-96tls" podUID="2461899c-c473-4831-86c9-d9294de1c488" containerName="payload-pre-processing" containerID="cri-o://b19279ee05c1f9634d3e61cac844be37ca4617d1b8f284f29ad1f156eba37a73" gracePeriod=30 Sep 06 12:41:06.323019 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:06.322930 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a898e1-77b7d88c6c-5v7wl" Sep 06 12:41:06.353720 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:06.353696 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a898e1-7694986778-96tls" Sep 06 12:41:06.397103 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:06.397068 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e49e3895-ebee-48ec-ab1b-1bca838c4db0-plugins-config-volume\") pod \"e49e3895-ebee-48ec-ab1b-1bca838c4db0\" (UID: \"e49e3895-ebee-48ec-ab1b-1bca838c4db0\") " Sep 06 12:41:06.397294 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:06.397133 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z544h\" (UniqueName: \"kubernetes.io/projected/e49e3895-ebee-48ec-ab1b-1bca838c4db0-kube-api-access-z544h\") pod \"e49e3895-ebee-48ec-ab1b-1bca838c4db0\" (UID: \"e49e3895-ebee-48ec-ab1b-1bca838c4db0\") " Sep 06 12:41:06.397717 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:06.397686 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e49e3895-ebee-48ec-ab1b-1bca838c4db0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e49e3895-ebee-48ec-ab1b-1bca838c4db0" (UID: "e49e3895-ebee-48ec-ab1b-1bca838c4db0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:41:06.400003 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:06.399977 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e49e3895-ebee-48ec-ab1b-1bca838c4db0-kube-api-access-z544h" (OuterVolumeSpecName: "kube-api-access-z544h") pod "e49e3895-ebee-48ec-ab1b-1bca838c4db0" (UID: "e49e3895-ebee-48ec-ab1b-1bca838c4db0"). InnerVolumeSpecName "kube-api-access-z544h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:41:06.498091 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:06.498070 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2461899c-c473-4831-86c9-d9294de1c488-plugins-config-volume\") pod \"2461899c-c473-4831-86c9-d9294de1c488\" (UID: \"2461899c-c473-4831-86c9-d9294de1c488\") " Sep 06 12:41:06.498192 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:06.498108 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppmk2\" (UniqueName: \"kubernetes.io/projected/2461899c-c473-4831-86c9-d9294de1c488-kube-api-access-ppmk2\") pod \"2461899c-c473-4831-86c9-d9294de1c488\" (UID: \"2461899c-c473-4831-86c9-d9294de1c488\") " Sep 06 12:41:06.498461 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:06.498433 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2461899c-c473-4831-86c9-d9294de1c488-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2461899c-c473-4831-86c9-d9294de1c488" (UID: "2461899c-c473-4831-86c9-d9294de1c488"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:41:06.498561 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:06.498473 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e49e3895-ebee-48ec-ab1b-1bca838c4db0-plugins-config-volume\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:41:06.498561 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:06.498487 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z544h\" (UniqueName: \"kubernetes.io/projected/e49e3895-ebee-48ec-ab1b-1bca838c4db0-kube-api-access-z544h\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:41:06.500230 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:06.500208 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2461899c-c473-4831-86c9-d9294de1c488-kube-api-access-ppmk2" (OuterVolumeSpecName: "kube-api-access-ppmk2") pod "2461899c-c473-4831-86c9-d9294de1c488" (UID: "2461899c-c473-4831-86c9-d9294de1c488"). InnerVolumeSpecName "kube-api-access-ppmk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:41:06.599495 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:06.599442 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2461899c-c473-4831-86c9-d9294de1c488-plugins-config-volume\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:41:06.599495 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:06.599460 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppmk2\" (UniqueName: \"kubernetes.io/projected/2461899c-c473-4831-86c9-d9294de1c488-kube-api-access-ppmk2\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:41:07.022426 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:07.022391 2576 generic.go:356] "Generic (PLEG): container finished" podID="2461899c-c473-4831-86c9-d9294de1c488" containerID="b19279ee05c1f9634d3e61cac844be37ca4617d1b8f284f29ad1f156eba37a73" exitCode=0 Sep 06 12:41:07.022613 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:07.022441 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a898e1-7694986778-96tls" Sep 06 12:41:07.022613 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:07.022482 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a898e1-7694986778-96tls" event={"ID":"2461899c-c473-4831-86c9-d9294de1c488","Type":"ContainerDied","Data":"b19279ee05c1f9634d3e61cac844be37ca4617d1b8f284f29ad1f156eba37a73"} Sep 06 12:41:07.022613 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:07.022561 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a898e1-7694986778-96tls" event={"ID":"2461899c-c473-4831-86c9-d9294de1c488","Type":"ContainerDied","Data":"624f9aa2f274667131b230fb7c54bd4cba51b43510468b68439f6534fb706d22"} Sep 06 12:41:07.022613 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:07.022585 2576 scope.go:117] "RemoveContainer" containerID="b19279ee05c1f9634d3e61cac844be37ca4617d1b8f284f29ad1f156eba37a73" Sep 06 12:41:07.024069 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:07.024046 2576 generic.go:356] "Generic (PLEG): container finished" podID="e49e3895-ebee-48ec-ab1b-1bca838c4db0" containerID="7a9c48671a355fc7a861f36c9d70cd0060f5f1e2cad42ee05ff47c56687f72a7" exitCode=0 Sep 06 12:41:07.024338 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:07.024094 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a898e1-77b7d88c6c-5v7wl" Sep 06 12:41:07.024338 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:07.024103 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a898e1-77b7d88c6c-5v7wl" event={"ID":"e49e3895-ebee-48ec-ab1b-1bca838c4db0","Type":"ContainerDied","Data":"7a9c48671a355fc7a861f36c9d70cd0060f5f1e2cad42ee05ff47c56687f72a7"} Sep 06 12:41:07.024338 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:07.024153 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a898e1-77b7d88c6c-5v7wl" event={"ID":"e49e3895-ebee-48ec-ab1b-1bca838c4db0","Type":"ContainerDied","Data":"23058cf9db7f944ed2549cc61c0698e7bf380a3902f0caad46556f258498588d"} Sep 06 12:41:07.034318 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:07.034288 2576 scope.go:117] "RemoveContainer" containerID="b19279ee05c1f9634d3e61cac844be37ca4617d1b8f284f29ad1f156eba37a73" Sep 06 12:41:07.034577 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:41:07.034558 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b19279ee05c1f9634d3e61cac844be37ca4617d1b8f284f29ad1f156eba37a73\": container with ID starting with b19279ee05c1f9634d3e61cac844be37ca4617d1b8f284f29ad1f156eba37a73 not found: ID does not exist" containerID="b19279ee05c1f9634d3e61cac844be37ca4617d1b8f284f29ad1f156eba37a73" Sep 06 12:41:07.034637 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:07.034585 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b19279ee05c1f9634d3e61cac844be37ca4617d1b8f284f29ad1f156eba37a73"} err="failed to get container status \"b19279ee05c1f9634d3e61cac844be37ca4617d1b8f284f29ad1f156eba37a73\": rpc error: code = NotFound desc = could not find container \"b19279ee05c1f9634d3e61cac844be37ca4617d1b8f284f29ad1f156eba37a73\": container with ID starting with b19279ee05c1f9634d3e61cac844be37ca4617d1b8f284f29ad1f156eba37a73 not found: ID does not exist" Sep 06 12:41:07.034637 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:07.034602 2576 scope.go:117] "RemoveContainer" containerID="7a9c48671a355fc7a861f36c9d70cd0060f5f1e2cad42ee05ff47c56687f72a7" Sep 06 12:41:07.041389 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:07.041369 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a898e1-77b7d88c6c-5v7wl"] Sep 06 12:41:07.043362 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:07.043348 2576 scope.go:117] "RemoveContainer" containerID="7a9c48671a355fc7a861f36c9d70cd0060f5f1e2cad42ee05ff47c56687f72a7" Sep 06 12:41:07.043655 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:41:07.043638 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a9c48671a355fc7a861f36c9d70cd0060f5f1e2cad42ee05ff47c56687f72a7\": container with ID starting with 7a9c48671a355fc7a861f36c9d70cd0060f5f1e2cad42ee05ff47c56687f72a7 not found: ID does not exist" containerID="7a9c48671a355fc7a861f36c9d70cd0060f5f1e2cad42ee05ff47c56687f72a7" Sep 06 12:41:07.043811 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:07.043661 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a9c48671a355fc7a861f36c9d70cd0060f5f1e2cad42ee05ff47c56687f72a7"} err="failed to get container status \"7a9c48671a355fc7a861f36c9d70cd0060f5f1e2cad42ee05ff47c56687f72a7\": rpc error: code = NotFound desc = could not find container \"7a9c48671a355fc7a861f36c9d70cd0060f5f1e2cad42ee05ff47c56687f72a7\": container with ID starting with 7a9c48671a355fc7a861f36c9d70cd0060f5f1e2cad42ee05ff47c56687f72a7 not found: ID does not exist" Sep 06 12:41:07.046780 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:07.046743 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a898e1-77b7d88c6c-5v7wl"] Sep 06 12:41:07.056023 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:07.056003 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a898e1-7694986778-96tls"] Sep 06 12:41:07.059370 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:07.059351 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a898e1-7694986778-96tls"] Sep 06 12:41:08.653475 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:08.653438 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2461899c-c473-4831-86c9-d9294de1c488" path="/var/lib/kubelet/pods/2461899c-c473-4831-86c9-d9294de1c488/volumes" Sep 06 12:41:08.653965 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:08.653810 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e49e3895-ebee-48ec-ab1b-1bca838c4db0" path="/var/lib/kubelet/pods/e49e3895-ebee-48ec-ab1b-1bca838c4db0/volumes" Sep 06 12:41:28.916425 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:28.916394 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-969fdf-596bdcbfcc-6qjqs"] Sep 06 12:41:28.916923 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:28.916636 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-969fdf-596bdcbfcc-6qjqs" podUID="a49117b9-9819-4d46-a236-5362d40baa64" containerName="payload-processing" containerID="cri-o://a67be1401bc3469abe171211e7a1f6f3013489725e3ef381a7629c1191be3aaa" gracePeriod=30 Sep 06 12:41:29.152479 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:29.152450 2576 generic.go:356] "Generic (PLEG): container finished" podID="a49117b9-9819-4d46-a236-5362d40baa64" containerID="a67be1401bc3469abe171211e7a1f6f3013489725e3ef381a7629c1191be3aaa" exitCode=0 Sep 06 12:41:29.152591 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:29.152521 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-969fdf-596bdcbfcc-6qjqs" event={"ID":"a49117b9-9819-4d46-a236-5362d40baa64","Type":"ContainerDied","Data":"a67be1401bc3469abe171211e7a1f6f3013489725e3ef381a7629c1191be3aaa"} Sep 06 12:41:29.174534 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:29.174473 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-969fdf-596bdcbfcc-6qjqs" Sep 06 12:41:29.295503 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:29.295476 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8r7r4\" (UniqueName: \"kubernetes.io/projected/a49117b9-9819-4d46-a236-5362d40baa64-kube-api-access-8r7r4\") pod \"a49117b9-9819-4d46-a236-5362d40baa64\" (UID: \"a49117b9-9819-4d46-a236-5362d40baa64\") " Sep 06 12:41:29.295656 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:29.295514 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a49117b9-9819-4d46-a236-5362d40baa64-plugins-config-volume\") pod \"a49117b9-9819-4d46-a236-5362d40baa64\" (UID: \"a49117b9-9819-4d46-a236-5362d40baa64\") " Sep 06 12:41:29.295940 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:29.295909 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a49117b9-9819-4d46-a236-5362d40baa64-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a49117b9-9819-4d46-a236-5362d40baa64" (UID: "a49117b9-9819-4d46-a236-5362d40baa64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:41:29.296080 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:29.296025 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a49117b9-9819-4d46-a236-5362d40baa64-plugins-config-volume\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:41:29.297936 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:29.297907 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a49117b9-9819-4d46-a236-5362d40baa64-kube-api-access-8r7r4" (OuterVolumeSpecName: "kube-api-access-8r7r4") pod "a49117b9-9819-4d46-a236-5362d40baa64" (UID: "a49117b9-9819-4d46-a236-5362d40baa64"). InnerVolumeSpecName "kube-api-access-8r7r4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:41:29.396922 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:29.396893 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r7r4\" (UniqueName: \"kubernetes.io/projected/a49117b9-9819-4d46-a236-5362d40baa64-kube-api-access-8r7r4\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:41:30.160459 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:30.160427 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-969fdf-596bdcbfcc-6qjqs" Sep 06 12:41:30.160459 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:30.160412 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-969fdf-596bdcbfcc-6qjqs" event={"ID":"a49117b9-9819-4d46-a236-5362d40baa64","Type":"ContainerDied","Data":"6a94405a598f01f4bb8534d9bc10e8b6c182325400ecfe70932f884213a0b0dd"} Sep 06 12:41:30.161057 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:30.160521 2576 scope.go:117] "RemoveContainer" containerID="a67be1401bc3469abe171211e7a1f6f3013489725e3ef381a7629c1191be3aaa" Sep 06 12:41:30.189293 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:30.189268 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-969fdf-596bdcbfcc-6qjqs"] Sep 06 12:41:30.195708 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:30.195685 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-969fdf-596bdcbfcc-6qjqs"] Sep 06 12:41:30.653248 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:30.653215 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a49117b9-9819-4d46-a236-5362d40baa64" path="/var/lib/kubelet/pods/a49117b9-9819-4d46-a236-5362d40baa64/volumes" Sep 06 12:41:46.693400 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:46.693365 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d96501-69744d7f8c-dw76p"] Sep 06 12:41:46.694000 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:46.693615 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d96501-69744d7f8c-dw76p" podUID="1dcdde5c-dd2b-4c98-8daf-56cf9124ea0e" containerName="payload-processing" containerID="cri-o://f13961c33db43c4e92314c471089a985f9d2a99c727cbad42b6a9fa519c39f41" gracePeriod=30 Sep 06 12:41:46.701964 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:46.701935 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96501-65b678d9d6-mfltf"] Sep 06 12:41:46.702191 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:46.702143 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96501-65b678d9d6-mfltf" podUID="679235a6-cf73-42f8-8770-8e2bc04afd6c" containerName="payload-pre-processing" containerID="cri-o://546ae6ba8b02b90203a4724f6be3700ec845963706083fa608a7cdf4c0f0dd85" gracePeriod=30 Sep 06 12:41:46.997330 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:46.997303 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96501-65b678d9d6-mfltf" Sep 06 12:41:47.000544 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:47.000523 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d96501-69744d7f8c-dw76p" Sep 06 12:41:47.044283 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:47.044260 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/679235a6-cf73-42f8-8770-8e2bc04afd6c-plugins-config-volume\") pod \"679235a6-cf73-42f8-8770-8e2bc04afd6c\" (UID: \"679235a6-cf73-42f8-8770-8e2bc04afd6c\") " Sep 06 12:41:47.044409 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:47.044360 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxs9z\" (UniqueName: \"kubernetes.io/projected/1dcdde5c-dd2b-4c98-8daf-56cf9124ea0e-kube-api-access-kxs9z\") pod \"1dcdde5c-dd2b-4c98-8daf-56cf9124ea0e\" (UID: \"1dcdde5c-dd2b-4c98-8daf-56cf9124ea0e\") " Sep 06 12:41:47.044409 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:47.044381 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dcdde5c-dd2b-4c98-8daf-56cf9124ea0e-plugins-config-volume\") pod \"1dcdde5c-dd2b-4c98-8daf-56cf9124ea0e\" (UID: \"1dcdde5c-dd2b-4c98-8daf-56cf9124ea0e\") " Sep 06 12:41:47.044526 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:47.044427 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4c48\" (UniqueName: \"kubernetes.io/projected/679235a6-cf73-42f8-8770-8e2bc04afd6c-kube-api-access-k4c48\") pod \"679235a6-cf73-42f8-8770-8e2bc04afd6c\" (UID: \"679235a6-cf73-42f8-8770-8e2bc04afd6c\") " Sep 06 12:41:47.044743 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:47.044715 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/679235a6-cf73-42f8-8770-8e2bc04afd6c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "679235a6-cf73-42f8-8770-8e2bc04afd6c" (UID: "679235a6-cf73-42f8-8770-8e2bc04afd6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:41:47.044743 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:47.044735 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dcdde5c-dd2b-4c98-8daf-56cf9124ea0e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1dcdde5c-dd2b-4c98-8daf-56cf9124ea0e" (UID: "1dcdde5c-dd2b-4c98-8daf-56cf9124ea0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:41:47.046635 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:47.046607 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/679235a6-cf73-42f8-8770-8e2bc04afd6c-kube-api-access-k4c48" (OuterVolumeSpecName: "kube-api-access-k4c48") pod "679235a6-cf73-42f8-8770-8e2bc04afd6c" (UID: "679235a6-cf73-42f8-8770-8e2bc04afd6c"). InnerVolumeSpecName "kube-api-access-k4c48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:41:47.046716 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:47.046639 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dcdde5c-dd2b-4c98-8daf-56cf9124ea0e-kube-api-access-kxs9z" (OuterVolumeSpecName: "kube-api-access-kxs9z") pod "1dcdde5c-dd2b-4c98-8daf-56cf9124ea0e" (UID: "1dcdde5c-dd2b-4c98-8daf-56cf9124ea0e"). InnerVolumeSpecName "kube-api-access-kxs9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:41:47.145837 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:47.145811 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxs9z\" (UniqueName: \"kubernetes.io/projected/1dcdde5c-dd2b-4c98-8daf-56cf9124ea0e-kube-api-access-kxs9z\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:41:47.145837 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:47.145834 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dcdde5c-dd2b-4c98-8daf-56cf9124ea0e-plugins-config-volume\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:41:47.146009 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:47.145844 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4c48\" (UniqueName: \"kubernetes.io/projected/679235a6-cf73-42f8-8770-8e2bc04afd6c-kube-api-access-k4c48\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:41:47.146009 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:47.145852 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/679235a6-cf73-42f8-8770-8e2bc04afd6c-plugins-config-volume\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:41:47.265361 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:47.265283 2576 generic.go:356] "Generic (PLEG): container finished" podID="1dcdde5c-dd2b-4c98-8daf-56cf9124ea0e" containerID="f13961c33db43c4e92314c471089a985f9d2a99c727cbad42b6a9fa519c39f41" exitCode=0 Sep 06 12:41:47.265361 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:47.265341 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d96501-69744d7f8c-dw76p" Sep 06 12:41:47.265543 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:47.265360 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d96501-69744d7f8c-dw76p" event={"ID":"1dcdde5c-dd2b-4c98-8daf-56cf9124ea0e","Type":"ContainerDied","Data":"f13961c33db43c4e92314c471089a985f9d2a99c727cbad42b6a9fa519c39f41"} Sep 06 12:41:47.265543 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:47.265437 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d96501-69744d7f8c-dw76p" event={"ID":"1dcdde5c-dd2b-4c98-8daf-56cf9124ea0e","Type":"ContainerDied","Data":"766e7842cb6e37c979cdf6d22cdfae09e70cbd24871194b013396b1fb2db78ad"} Sep 06 12:41:47.265543 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:47.265459 2576 scope.go:117] "RemoveContainer" containerID="f13961c33db43c4e92314c471089a985f9d2a99c727cbad42b6a9fa519c39f41" Sep 06 12:41:47.266936 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:47.266912 2576 generic.go:356] "Generic (PLEG): container finished" podID="679235a6-cf73-42f8-8770-8e2bc04afd6c" containerID="546ae6ba8b02b90203a4724f6be3700ec845963706083fa608a7cdf4c0f0dd85" exitCode=0 Sep 06 12:41:47.267030 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:47.266985 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96501-65b678d9d6-mfltf" event={"ID":"679235a6-cf73-42f8-8770-8e2bc04afd6c","Type":"ContainerDied","Data":"546ae6ba8b02b90203a4724f6be3700ec845963706083fa608a7cdf4c0f0dd85"} Sep 06 12:41:47.267030 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:47.267021 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96501-65b678d9d6-mfltf" event={"ID":"679235a6-cf73-42f8-8770-8e2bc04afd6c","Type":"ContainerDied","Data":"560d90713d6ee534a2076f2b47c1844656c970e3a624f2aea2d526b02588c854"} Sep 06 12:41:47.267141 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:47.267127 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96501-65b678d9d6-mfltf" Sep 06 12:41:47.275915 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:47.275895 2576 scope.go:117] "RemoveContainer" containerID="f13961c33db43c4e92314c471089a985f9d2a99c727cbad42b6a9fa519c39f41" Sep 06 12:41:47.276224 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:41:47.276205 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f13961c33db43c4e92314c471089a985f9d2a99c727cbad42b6a9fa519c39f41\": container with ID starting with f13961c33db43c4e92314c471089a985f9d2a99c727cbad42b6a9fa519c39f41 not found: ID does not exist" containerID="f13961c33db43c4e92314c471089a985f9d2a99c727cbad42b6a9fa519c39f41" Sep 06 12:41:47.276315 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:47.276237 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f13961c33db43c4e92314c471089a985f9d2a99c727cbad42b6a9fa519c39f41"} err="failed to get container status \"f13961c33db43c4e92314c471089a985f9d2a99c727cbad42b6a9fa519c39f41\": rpc error: code = NotFound desc = could not find container \"f13961c33db43c4e92314c471089a985f9d2a99c727cbad42b6a9fa519c39f41\": container with ID starting with f13961c33db43c4e92314c471089a985f9d2a99c727cbad42b6a9fa519c39f41 not found: ID does not exist" Sep 06 12:41:47.276315 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:47.276260 2576 scope.go:117] "RemoveContainer" containerID="546ae6ba8b02b90203a4724f6be3700ec845963706083fa608a7cdf4c0f0dd85" Sep 06 12:41:47.284444 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:47.284428 2576 scope.go:117] "RemoveContainer" containerID="546ae6ba8b02b90203a4724f6be3700ec845963706083fa608a7cdf4c0f0dd85" Sep 06 12:41:47.284727 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:41:47.284707 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"546ae6ba8b02b90203a4724f6be3700ec845963706083fa608a7cdf4c0f0dd85\": container with ID starting with 546ae6ba8b02b90203a4724f6be3700ec845963706083fa608a7cdf4c0f0dd85 not found: ID does not exist" containerID="546ae6ba8b02b90203a4724f6be3700ec845963706083fa608a7cdf4c0f0dd85" Sep 06 12:41:47.284811 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:47.284735 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"546ae6ba8b02b90203a4724f6be3700ec845963706083fa608a7cdf4c0f0dd85"} err="failed to get container status \"546ae6ba8b02b90203a4724f6be3700ec845963706083fa608a7cdf4c0f0dd85\": rpc error: code = NotFound desc = could not find container \"546ae6ba8b02b90203a4724f6be3700ec845963706083fa608a7cdf4c0f0dd85\": container with ID starting with 546ae6ba8b02b90203a4724f6be3700ec845963706083fa608a7cdf4c0f0dd85 not found: ID does not exist" Sep 06 12:41:47.287621 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:47.287602 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d96501-69744d7f8c-dw76p"] Sep 06 12:41:47.291228 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:47.291202 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d96501-69744d7f8c-dw76p"] Sep 06 12:41:47.301522 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:47.301499 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96501-65b678d9d6-mfltf"] Sep 06 12:41:47.304742 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:47.304722 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96501-65b678d9d6-mfltf"] Sep 06 12:41:48.655333 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:48.655286 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dcdde5c-dd2b-4c98-8daf-56cf9124ea0e" path="/var/lib/kubelet/pods/1dcdde5c-dd2b-4c98-8daf-56cf9124ea0e/volumes" Sep 06 12:41:48.655736 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:41:48.655659 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="679235a6-cf73-42f8-8770-8e2bc04afd6c" path="/var/lib/kubelet/pods/679235a6-cf73-42f8-8770-8e2bc04afd6c/volumes" Sep 06 12:42:03.974507 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:42:03.974472 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-556bc6-5bd6db54b9-dzhrj"] Sep 06 12:42:03.974993 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:42:03.974714 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-556bc6-5bd6db54b9-dzhrj" podUID="fdd8d390-2969-4a92-93fc-0a3ad425c583" containerName="payload-processing" containerID="cri-o://5280eb9fe81ab53050476b9f564ee29f0e20f27add8e18ab17f3a3082d5e77e4" gracePeriod=30 Sep 06 12:42:03.978192 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:42:03.978167 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-556bc6-54789d5666-h7gxf"] Sep 06 12:42:03.978430 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:42:03.978375 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-556bc6-54789d5666-h7gxf" podUID="ddf74b53-1c59-4d1e-8e4e-55ba5ef2936f" containerName="payload-pre-processing" containerID="cri-o://34fbe77c378bba4e31834b9d8d1d13b49c3421d3c700966f3251c296e89bce20" gracePeriod=30 Sep 06 12:42:04.260476 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:42:04.260454 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-556bc6-54789d5666-h7gxf" Sep 06 12:42:04.272481 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:42:04.272462 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-556bc6-5bd6db54b9-dzhrj" Sep 06 12:42:04.367813 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:42:04.367783 2576 generic.go:356] "Generic (PLEG): container finished" podID="ddf74b53-1c59-4d1e-8e4e-55ba5ef2936f" containerID="34fbe77c378bba4e31834b9d8d1d13b49c3421d3c700966f3251c296e89bce20" exitCode=0 Sep 06 12:42:04.367964 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:42:04.367873 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-556bc6-54789d5666-h7gxf" Sep 06 12:42:04.367964 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:42:04.367878 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-556bc6-54789d5666-h7gxf" event={"ID":"ddf74b53-1c59-4d1e-8e4e-55ba5ef2936f","Type":"ContainerDied","Data":"34fbe77c378bba4e31834b9d8d1d13b49c3421d3c700966f3251c296e89bce20"} Sep 06 12:42:04.368065 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:42:04.367991 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-556bc6-54789d5666-h7gxf" event={"ID":"ddf74b53-1c59-4d1e-8e4e-55ba5ef2936f","Type":"ContainerDied","Data":"2de01a9750b4167d29fcf2afee9a7961b5c382025b1aeb33df904fb8aa0f841d"} Sep 06 12:42:04.368065 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:42:04.368022 2576 scope.go:117] "RemoveContainer" containerID="34fbe77c378bba4e31834b9d8d1d13b49c3421d3c700966f3251c296e89bce20" Sep 06 12:42:04.369385 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:42:04.369363 2576 generic.go:356] "Generic (PLEG): container finished" podID="fdd8d390-2969-4a92-93fc-0a3ad425c583" containerID="5280eb9fe81ab53050476b9f564ee29f0e20f27add8e18ab17f3a3082d5e77e4" exitCode=0 Sep 06 12:42:04.369486 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:42:04.369415 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-556bc6-5bd6db54b9-dzhrj" Sep 06 12:42:04.369544 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:42:04.369487 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-556bc6-5bd6db54b9-dzhrj" event={"ID":"fdd8d390-2969-4a92-93fc-0a3ad425c583","Type":"ContainerDied","Data":"5280eb9fe81ab53050476b9f564ee29f0e20f27add8e18ab17f3a3082d5e77e4"} Sep 06 12:42:04.369544 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:42:04.369531 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-556bc6-5bd6db54b9-dzhrj" event={"ID":"fdd8d390-2969-4a92-93fc-0a3ad425c583","Type":"ContainerDied","Data":"4e4d5ccb721f320c02037fa0fe97ea07ce21b5b8c5eb19f8db92f8a6334c5737"} Sep 06 12:42:04.376988 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:42:04.376972 2576 scope.go:117] "RemoveContainer" containerID="34fbe77c378bba4e31834b9d8d1d13b49c3421d3c700966f3251c296e89bce20" Sep 06 12:42:04.377284 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:42:04.377267 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34fbe77c378bba4e31834b9d8d1d13b49c3421d3c700966f3251c296e89bce20\": container with ID starting with 34fbe77c378bba4e31834b9d8d1d13b49c3421d3c700966f3251c296e89bce20 not found: ID does not exist" containerID="34fbe77c378bba4e31834b9d8d1d13b49c3421d3c700966f3251c296e89bce20" Sep 06 12:42:04.377328 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:42:04.377294 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34fbe77c378bba4e31834b9d8d1d13b49c3421d3c700966f3251c296e89bce20"} err="failed to get container status \"34fbe77c378bba4e31834b9d8d1d13b49c3421d3c700966f3251c296e89bce20\": rpc error: code = NotFound desc = could not find container \"34fbe77c378bba4e31834b9d8d1d13b49c3421d3c700966f3251c296e89bce20\": container with ID starting with 34fbe77c378bba4e31834b9d8d1d13b49c3421d3c700966f3251c296e89bce20 not found: ID does not exist" Sep 06 12:42:04.377328 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:42:04.377310 2576 scope.go:117] "RemoveContainer" containerID="5280eb9fe81ab53050476b9f564ee29f0e20f27add8e18ab17f3a3082d5e77e4" Sep 06 12:42:04.385220 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:42:04.385205 2576 scope.go:117] "RemoveContainer" containerID="5280eb9fe81ab53050476b9f564ee29f0e20f27add8e18ab17f3a3082d5e77e4" Sep 06 12:42:04.385511 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:42:04.385493 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5280eb9fe81ab53050476b9f564ee29f0e20f27add8e18ab17f3a3082d5e77e4\": container with ID starting with 5280eb9fe81ab53050476b9f564ee29f0e20f27add8e18ab17f3a3082d5e77e4 not found: ID does not exist" containerID="5280eb9fe81ab53050476b9f564ee29f0e20f27add8e18ab17f3a3082d5e77e4" Sep 06 12:42:04.385567 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:42:04.385517 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5280eb9fe81ab53050476b9f564ee29f0e20f27add8e18ab17f3a3082d5e77e4"} err="failed to get container status \"5280eb9fe81ab53050476b9f564ee29f0e20f27add8e18ab17f3a3082d5e77e4\": rpc error: code = NotFound desc = could not find container \"5280eb9fe81ab53050476b9f564ee29f0e20f27add8e18ab17f3a3082d5e77e4\": container with ID starting with 5280eb9fe81ab53050476b9f564ee29f0e20f27add8e18ab17f3a3082d5e77e4 not found: ID does not exist" Sep 06 12:42:04.395910 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:42:04.395890 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdd8d390-2969-4a92-93fc-0a3ad425c583-plugins-config-volume\") pod \"fdd8d390-2969-4a92-93fc-0a3ad425c583\" (UID: \"fdd8d390-2969-4a92-93fc-0a3ad425c583\") " Sep 06 12:42:04.395988 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:42:04.395934 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxt9n\" (UniqueName: \"kubernetes.io/projected/ddf74b53-1c59-4d1e-8e4e-55ba5ef2936f-kube-api-access-zxt9n\") pod \"ddf74b53-1c59-4d1e-8e4e-55ba5ef2936f\" (UID: \"ddf74b53-1c59-4d1e-8e4e-55ba5ef2936f\") " Sep 06 12:42:04.395988 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:42:04.395953 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddf74b53-1c59-4d1e-8e4e-55ba5ef2936f-plugins-config-volume\") pod \"ddf74b53-1c59-4d1e-8e4e-55ba5ef2936f\" (UID: \"ddf74b53-1c59-4d1e-8e4e-55ba5ef2936f\") " Sep 06 12:42:04.395988 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:42:04.395977 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tvqc\" (UniqueName: \"kubernetes.io/projected/fdd8d390-2969-4a92-93fc-0a3ad425c583-kube-api-access-4tvqc\") pod \"fdd8d390-2969-4a92-93fc-0a3ad425c583\" (UID: \"fdd8d390-2969-4a92-93fc-0a3ad425c583\") " Sep 06 12:42:04.396257 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:42:04.396226 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdd8d390-2969-4a92-93fc-0a3ad425c583-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fdd8d390-2969-4a92-93fc-0a3ad425c583" (UID: "fdd8d390-2969-4a92-93fc-0a3ad425c583"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:42:04.396322 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:42:04.396310 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdd8d390-2969-4a92-93fc-0a3ad425c583-plugins-config-volume\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:42:04.396358 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:42:04.396324 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddf74b53-1c59-4d1e-8e4e-55ba5ef2936f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ddf74b53-1c59-4d1e-8e4e-55ba5ef2936f" (UID: "ddf74b53-1c59-4d1e-8e4e-55ba5ef2936f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:42:04.398038 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:42:04.398020 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdd8d390-2969-4a92-93fc-0a3ad425c583-kube-api-access-4tvqc" (OuterVolumeSpecName: "kube-api-access-4tvqc") pod "fdd8d390-2969-4a92-93fc-0a3ad425c583" (UID: "fdd8d390-2969-4a92-93fc-0a3ad425c583"). InnerVolumeSpecName "kube-api-access-4tvqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:42:04.398105 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:42:04.398030 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddf74b53-1c59-4d1e-8e4e-55ba5ef2936f-kube-api-access-zxt9n" (OuterVolumeSpecName: "kube-api-access-zxt9n") pod "ddf74b53-1c59-4d1e-8e4e-55ba5ef2936f" (UID: "ddf74b53-1c59-4d1e-8e4e-55ba5ef2936f"). InnerVolumeSpecName "kube-api-access-zxt9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:42:04.497081 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:42:04.497015 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxt9n\" (UniqueName: \"kubernetes.io/projected/ddf74b53-1c59-4d1e-8e4e-55ba5ef2936f-kube-api-access-zxt9n\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:42:04.497081 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:42:04.497040 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddf74b53-1c59-4d1e-8e4e-55ba5ef2936f-plugins-config-volume\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:42:04.497081 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:42:04.497054 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tvqc\" (UniqueName: \"kubernetes.io/projected/fdd8d390-2969-4a92-93fc-0a3ad425c583-kube-api-access-4tvqc\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:42:04.683971 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:42:04.683939 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-556bc6-54789d5666-h7gxf"] Sep 06 12:42:04.687965 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:42:04.687942 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-556bc6-54789d5666-h7gxf"] Sep 06 12:42:04.696516 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:42:04.696497 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-556bc6-5bd6db54b9-dzhrj"] Sep 06 12:42:04.699935 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:42:04.699913 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-556bc6-5bd6db54b9-dzhrj"] Sep 06 12:42:06.654354 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:42:06.654318 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddf74b53-1c59-4d1e-8e4e-55ba5ef2936f" path="/var/lib/kubelet/pods/ddf74b53-1c59-4d1e-8e4e-55ba5ef2936f/volumes" Sep 06 12:42:06.654785 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:42:06.654674 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdd8d390-2969-4a92-93fc-0a3ad425c583" path="/var/lib/kubelet/pods/fdd8d390-2969-4a92-93fc-0a3ad425c583/volumes" Sep 06 12:43:04.704648 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:04.704622 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98vz4_4103c09f-dd43-419d-92e5-2374af5b5e01/ovn-acl-logging/0.log" Sep 06 12:43:04.710695 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:04.710675 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98vz4_4103c09f-dd43-419d-92e5-2374af5b5e01/ovn-acl-logging/0.log" Sep 06 12:43:07.002091 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.002054 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst"] Sep 06 12:43:07.002573 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.002477 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dcdde5c-dd2b-4c98-8daf-56cf9124ea0e" containerName="payload-processing" Sep 06 12:43:07.002573 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.002488 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dcdde5c-dd2b-4c98-8daf-56cf9124ea0e" containerName="payload-processing" Sep 06 12:43:07.002573 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.002511 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e49e3895-ebee-48ec-ab1b-1bca838c4db0" containerName="payload-processing" Sep 06 12:43:07.002573 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.002518 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e49e3895-ebee-48ec-ab1b-1bca838c4db0" containerName="payload-processing" Sep 06 12:43:07.002573 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.002525 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddf74b53-1c59-4d1e-8e4e-55ba5ef2936f" containerName="payload-pre-processing" Sep 06 12:43:07.002573 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.002531 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddf74b53-1c59-4d1e-8e4e-55ba5ef2936f" containerName="payload-pre-processing" Sep 06 12:43:07.002573 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.002538 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="679235a6-cf73-42f8-8770-8e2bc04afd6c" containerName="payload-pre-processing" Sep 06 12:43:07.002573 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.002543 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="679235a6-cf73-42f8-8770-8e2bc04afd6c" containerName="payload-pre-processing" Sep 06 12:43:07.002573 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.002556 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2461899c-c473-4831-86c9-d9294de1c488" containerName="payload-pre-processing" Sep 06 12:43:07.002573 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.002562 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2461899c-c473-4831-86c9-d9294de1c488" containerName="payload-pre-processing" Sep 06 12:43:07.002573 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.002568 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a49117b9-9819-4d46-a236-5362d40baa64" containerName="payload-processing" Sep 06 12:43:07.002573 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.002573 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a49117b9-9819-4d46-a236-5362d40baa64" containerName="payload-processing" Sep 06 12:43:07.002573 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.002579 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdd8d390-2969-4a92-93fc-0a3ad425c583" containerName="payload-processing" Sep 06 12:43:07.002573 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.002584 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdd8d390-2969-4a92-93fc-0a3ad425c583" containerName="payload-processing" Sep 06 12:43:07.003376 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.002647 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddf74b53-1c59-4d1e-8e4e-55ba5ef2936f" containerName="payload-pre-processing" Sep 06 12:43:07.003376 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.002654 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdd8d390-2969-4a92-93fc-0a3ad425c583" containerName="payload-processing" Sep 06 12:43:07.003376 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.002659 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2461899c-c473-4831-86c9-d9294de1c488" containerName="payload-pre-processing" Sep 06 12:43:07.003376 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.002666 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="679235a6-cf73-42f8-8770-8e2bc04afd6c" containerName="payload-pre-processing" Sep 06 12:43:07.003376 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.002672 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a49117b9-9819-4d46-a236-5362d40baa64" containerName="payload-processing" Sep 06 12:43:07.003376 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.002678 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e49e3895-ebee-48ec-ab1b-1bca838c4db0" containerName="payload-processing" Sep 06 12:43:07.003376 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.002684 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dcdde5c-dd2b-4c98-8daf-56cf9124ea0e" containerName="payload-processing" Sep 06 12:43:07.013712 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.013690 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst" Sep 06 12:43:07.016581 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.016550 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-4f1951-serving-cert\"" Sep 06 12:43:07.018640 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.018616 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst"] Sep 06 12:43:07.127687 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.127659 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b533bd8-7770-4a73-9c49-a768f1273b2b-metrics-certs\") pod \"maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst\" (UID: \"7b533bd8-7770-4a73-9c49-a768f1273b2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst" Sep 06 12:43:07.127831 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.127701 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b533bd8-7770-4a73-9c49-a768f1273b2b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst\" (UID: \"7b533bd8-7770-4a73-9c49-a768f1273b2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst" Sep 06 12:43:07.127881 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.127849 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hq9j\" (UniqueName: \"kubernetes.io/projected/7b533bd8-7770-4a73-9c49-a768f1273b2b-kube-api-access-6hq9j\") pod \"maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst\" (UID: \"7b533bd8-7770-4a73-9c49-a768f1273b2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst" Sep 06 12:43:07.127927 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.127890 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b533bd8-7770-4a73-9c49-a768f1273b2b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst\" (UID: \"7b533bd8-7770-4a73-9c49-a768f1273b2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst" Sep 06 12:43:07.229231 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.229201 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hq9j\" (UniqueName: \"kubernetes.io/projected/7b533bd8-7770-4a73-9c49-a768f1273b2b-kube-api-access-6hq9j\") pod \"maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst\" (UID: \"7b533bd8-7770-4a73-9c49-a768f1273b2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst" Sep 06 12:43:07.229231 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.229235 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b533bd8-7770-4a73-9c49-a768f1273b2b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst\" (UID: \"7b533bd8-7770-4a73-9c49-a768f1273b2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst" Sep 06 12:43:07.229460 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.229255 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b533bd8-7770-4a73-9c49-a768f1273b2b-metrics-certs\") pod \"maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst\" (UID: \"7b533bd8-7770-4a73-9c49-a768f1273b2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst" Sep 06 12:43:07.229460 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.229389 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b533bd8-7770-4a73-9c49-a768f1273b2b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst\" (UID: \"7b533bd8-7770-4a73-9c49-a768f1273b2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst" Sep 06 12:43:07.229637 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:43:07.229609 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f1951-serving-cert: secret "maas-api-e2e-ipp-a-4f1951-serving-cert" not found Sep 06 12:43:07.229720 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:43:07.229709 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b533bd8-7770-4a73-9c49-a768f1273b2b-maas-api-tls podName:7b533bd8-7770-4a73-9c49-a768f1273b2b nodeName:}" failed. No retries permitted until 2026-09-06 12:43:07.729686113 +0000 UTC m=+1503.585443996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7b533bd8-7770-4a73-9c49-a768f1273b2b-maas-api-tls") pod "maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst" (UID: "7b533bd8-7770-4a73-9c49-a768f1273b2b") : secret "maas-api-e2e-ipp-a-4f1951-serving-cert" not found Sep 06 12:43:07.230060 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.230042 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b533bd8-7770-4a73-9c49-a768f1273b2b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst\" (UID: \"7b533bd8-7770-4a73-9c49-a768f1273b2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst" Sep 06 12:43:07.232041 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.232024 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b533bd8-7770-4a73-9c49-a768f1273b2b-metrics-certs\") pod \"maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst\" (UID: \"7b533bd8-7770-4a73-9c49-a768f1273b2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst" Sep 06 12:43:07.240157 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.240127 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hq9j\" (UniqueName: \"kubernetes.io/projected/7b533bd8-7770-4a73-9c49-a768f1273b2b-kube-api-access-6hq9j\") pod \"maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst\" (UID: \"7b533bd8-7770-4a73-9c49-a768f1273b2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst" Sep 06 12:43:07.734597 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.734567 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b533bd8-7770-4a73-9c49-a768f1273b2b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst\" (UID: \"7b533bd8-7770-4a73-9c49-a768f1273b2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst" Sep 06 12:43:07.737078 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.737049 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b533bd8-7770-4a73-9c49-a768f1273b2b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst\" (UID: \"7b533bd8-7770-4a73-9c49-a768f1273b2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst" Sep 06 12:43:07.803279 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.803250 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4f1951-744f4455f7-p949b"] Sep 06 12:43:07.807051 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.807033 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4f1951-744f4455f7-p949b" Sep 06 12:43:07.809860 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.809841 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4f1951-dockercfg-8jpk2\"" Sep 06 12:43:07.810310 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.810295 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4f1951\"" Sep 06 12:43:07.824301 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.824273 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4f1951-744f4455f7-p949b"] Sep 06 12:43:07.860231 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.860204 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4f1951-b654889c4-fvct2"] Sep 06 12:43:07.863654 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.863635 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f1951-b654889c4-fvct2" Sep 06 12:43:07.874471 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.874449 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4f1951-b654889c4-fvct2"] Sep 06 12:43:07.927249 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.927218 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst" Sep 06 12:43:07.937438 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.937415 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc9xd\" (UniqueName: \"kubernetes.io/projected/591382be-edad-43f5-af3b-1ebc658eb823-kube-api-access-vc9xd\") pod \"payload-processing-e2e-ipp-a-4f1951-744f4455f7-p949b\" (UID: \"591382be-edad-43f5-af3b-1ebc658eb823\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4f1951-744f4455f7-p949b" Sep 06 12:43:07.937571 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:07.937488 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/591382be-edad-43f5-af3b-1ebc658eb823-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4f1951-744f4455f7-p949b\" (UID: \"591382be-edad-43f5-af3b-1ebc658eb823\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4f1951-744f4455f7-p949b" Sep 06 12:43:08.038183 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:08.038156 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/591382be-edad-43f5-af3b-1ebc658eb823-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4f1951-744f4455f7-p949b\" (UID: \"591382be-edad-43f5-af3b-1ebc658eb823\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4f1951-744f4455f7-p949b" Sep 06 12:43:08.038560 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:08.038190 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee118fe-a2e9-4753-9bfc-6d2d8f0da070-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4f1951-b654889c4-fvct2\" (UID: \"7ee118fe-a2e9-4753-9bfc-6d2d8f0da070\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f1951-b654889c4-fvct2" Sep 06 12:43:08.038560 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:08.038345 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbd8b\" (UniqueName: \"kubernetes.io/projected/7ee118fe-a2e9-4753-9bfc-6d2d8f0da070-kube-api-access-rbd8b\") pod \"payload-pre-processing-e2e-ipp-a-4f1951-b654889c4-fvct2\" (UID: \"7ee118fe-a2e9-4753-9bfc-6d2d8f0da070\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f1951-b654889c4-fvct2" Sep 06 12:43:08.038560 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:08.038380 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc9xd\" (UniqueName: \"kubernetes.io/projected/591382be-edad-43f5-af3b-1ebc658eb823-kube-api-access-vc9xd\") pod \"payload-processing-e2e-ipp-a-4f1951-744f4455f7-p949b\" (UID: \"591382be-edad-43f5-af3b-1ebc658eb823\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4f1951-744f4455f7-p949b" Sep 06 12:43:08.038963 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:08.038944 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/591382be-edad-43f5-af3b-1ebc658eb823-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4f1951-744f4455f7-p949b\" (UID: \"591382be-edad-43f5-af3b-1ebc658eb823\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4f1951-744f4455f7-p949b" Sep 06 12:43:08.047574 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:08.047548 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc9xd\" (UniqueName: \"kubernetes.io/projected/591382be-edad-43f5-af3b-1ebc658eb823-kube-api-access-vc9xd\") pod \"payload-processing-e2e-ipp-a-4f1951-744f4455f7-p949b\" (UID: \"591382be-edad-43f5-af3b-1ebc658eb823\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4f1951-744f4455f7-p949b" Sep 06 12:43:08.117876 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:08.117852 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4f1951-744f4455f7-p949b" Sep 06 12:43:08.139142 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:08.139115 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbd8b\" (UniqueName: \"kubernetes.io/projected/7ee118fe-a2e9-4753-9bfc-6d2d8f0da070-kube-api-access-rbd8b\") pod \"payload-pre-processing-e2e-ipp-a-4f1951-b654889c4-fvct2\" (UID: \"7ee118fe-a2e9-4753-9bfc-6d2d8f0da070\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f1951-b654889c4-fvct2" Sep 06 12:43:08.139246 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:08.139201 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee118fe-a2e9-4753-9bfc-6d2d8f0da070-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4f1951-b654889c4-fvct2\" (UID: \"7ee118fe-a2e9-4753-9bfc-6d2d8f0da070\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f1951-b654889c4-fvct2" Sep 06 12:43:08.139877 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:08.139858 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee118fe-a2e9-4753-9bfc-6d2d8f0da070-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4f1951-b654889c4-fvct2\" (UID: \"7ee118fe-a2e9-4753-9bfc-6d2d8f0da070\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f1951-b654889c4-fvct2" Sep 06 12:43:08.148128 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:08.148091 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbd8b\" (UniqueName: \"kubernetes.io/projected/7ee118fe-a2e9-4753-9bfc-6d2d8f0da070-kube-api-access-rbd8b\") pod \"payload-pre-processing-e2e-ipp-a-4f1951-b654889c4-fvct2\" (UID: \"7ee118fe-a2e9-4753-9bfc-6d2d8f0da070\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f1951-b654889c4-fvct2" Sep 06 12:43:08.177307 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:08.174054 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f1951-b654889c4-fvct2" Sep 06 12:43:08.209027 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:08.208998 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst"] Sep 06 12:43:08.212420 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:43:08.212395 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b533bd8_7770_4a73_9c49_a768f1273b2b.slice/crio-99da6c167fee0553716c496efbf33de5722abe9fcc9203957bcf584701554be5 WatchSource:0}: Error finding container 99da6c167fee0553716c496efbf33de5722abe9fcc9203957bcf584701554be5: Status 404 returned error can't find the container with id 99da6c167fee0553716c496efbf33de5722abe9fcc9203957bcf584701554be5 Sep 06 12:43:08.214783 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:08.214743 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 06 12:43:08.391441 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:08.391418 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4f1951-744f4455f7-p949b"] Sep 06 12:43:08.393741 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:43:08.393716 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod591382be_edad_43f5_af3b_1ebc658eb823.slice/crio-935508acf32bbd1b6298a51bd21145f1a138fb4c996db8cd020674332abb17e7 WatchSource:0}: Error finding container 935508acf32bbd1b6298a51bd21145f1a138fb4c996db8cd020674332abb17e7: Status 404 returned error can't find the container with id 935508acf32bbd1b6298a51bd21145f1a138fb4c996db8cd020674332abb17e7 Sep 06 12:43:08.462080 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:08.462056 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4f1951-b654889c4-fvct2"] Sep 06 12:43:08.463624 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:43:08.463596 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee118fe_a2e9_4753_9bfc_6d2d8f0da070.slice/crio-dcfc54f6d522b76503197656e36a5b01e9ea3eea54a82fa22bad964bea49dcfd WatchSource:0}: Error finding container dcfc54f6d522b76503197656e36a5b01e9ea3eea54a82fa22bad964bea49dcfd: Status 404 returned error can't find the container with id dcfc54f6d522b76503197656e36a5b01e9ea3eea54a82fa22bad964bea49dcfd Sep 06 12:43:08.721995 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:08.721960 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst" event={"ID":"7b533bd8-7770-4a73-9c49-a768f1273b2b","Type":"ContainerStarted","Data":"99da6c167fee0553716c496efbf33de5722abe9fcc9203957bcf584701554be5"} Sep 06 12:43:08.724102 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:08.724070 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f1951-b654889c4-fvct2" event={"ID":"7ee118fe-a2e9-4753-9bfc-6d2d8f0da070","Type":"ContainerStarted","Data":"dadde2fbe89a3a2de216d62db9329e5d616e38417cb9b92470cf257d89cabe12"} Sep 06 12:43:08.724240 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:08.724139 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f1951-b654889c4-fvct2" event={"ID":"7ee118fe-a2e9-4753-9bfc-6d2d8f0da070","Type":"ContainerStarted","Data":"dcfc54f6d522b76503197656e36a5b01e9ea3eea54a82fa22bad964bea49dcfd"} Sep 06 12:43:08.724240 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:08.724175 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f1951-b654889c4-fvct2" Sep 06 12:43:08.725813 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:08.725786 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4f1951-744f4455f7-p949b" event={"ID":"591382be-edad-43f5-af3b-1ebc658eb823","Type":"ContainerStarted","Data":"f552988df2821a35d4f7bf8c9ab5179e9485061ac51ed15457b0b44a9211f6da"} Sep 06 12:43:08.725930 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:08.725834 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4f1951-744f4455f7-p949b" event={"ID":"591382be-edad-43f5-af3b-1ebc658eb823","Type":"ContainerStarted","Data":"935508acf32bbd1b6298a51bd21145f1a138fb4c996db8cd020674332abb17e7"} Sep 06 12:43:08.725930 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:08.725857 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4f1951-744f4455f7-p949b" Sep 06 12:43:08.743320 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:08.743093 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f1951-b654889c4-fvct2" podStartSLOduration=1.7430753650000002 podStartE2EDuration="1.743075365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:43:08.74210384 +0000 UTC m=+1504.597861753" watchObservedRunningTime="2026-09-06 12:43:08.743075365 +0000 UTC m=+1504.598833269" Sep 06 12:43:08.762724 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:08.762290 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-4f1951-744f4455f7-p949b" podStartSLOduration=1.762266025 podStartE2EDuration="1.762266025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:43:08.761835568 +0000 UTC m=+1504.617593481" watchObservedRunningTime="2026-09-06 12:43:08.762266025 +0000 UTC m=+1504.618023925" Sep 06 12:43:09.732486 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:09.732448 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst" event={"ID":"7b533bd8-7770-4a73-9c49-a768f1273b2b","Type":"ContainerStarted","Data":"f4318170c4c49b01bdbb4d014f08976a595e3d687017e58f780ab2d9f7cd12d1"} Sep 06 12:43:09.732981 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:09.732565 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst" Sep 06 12:43:09.753016 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:09.752965 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst" podStartSLOduration=3.275128257 podStartE2EDuration="3.752948791s" totalImagesPullingTime="477.820534ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:43:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:43:09.75099966 +0000 UTC m=+1505.606757546" watchObservedRunningTime="2026-09-06 12:43:09.752948791 +0000 UTC m=+1505.608706678" Sep 06 12:43:15.743660 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:15.743625 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst" Sep 06 12:43:19.735095 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:19.735064 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4f1951-744f4455f7-p949b" Sep 06 12:43:19.735530 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:43:19.735291 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f1951-b654889c4-fvct2" Sep 06 12:44:04.787200 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:44:04.787165 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-jlvzm_2a0a23e6-8a48-4825-b2d7-e6bfd31fbe1f/payload-processing/0.log" Sep 06 12:44:55.749013 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:44:55.748924 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-4f1951-744f4455f7-p949b_591382be-edad-43f5-af3b-1ebc658eb823/payload-processing/0.log" Sep 06 12:44:55.859828 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:44:55.859793 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-jlvzm_2a0a23e6-8a48-4825-b2d7-e6bfd31fbe1f/payload-processing/0.log" Sep 06 12:45:00.137940 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:00.137889 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e6515e-29811645-rf8nv"] Sep 06 12:45:00.141863 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:00.141828 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29811645-n24zn"] Sep 06 12:45:00.142037 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:00.141970 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e6515e-29811645-rf8nv" Sep 06 12:45:00.144446 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:00.144407 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhtt2\"" Sep 06 12:45:00.145957 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:00.145934 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811645-n24zn" Sep 06 12:45:00.156583 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:00.156560 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e6515e-29811645-rf8nv" Sep 06 12:45:00.159961 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:00.159885 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29811645-n24zn"] Sep 06 12:45:00.164162 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:00.164113 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e6515e-29811645-rf8nv"] Sep 06 12:45:00.168588 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:00.168527 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811645-n24zn" Sep 06 12:45:00.450983 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:00.450953 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e6515e-29811645-rf8nv"] Sep 06 12:45:00.453642 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:45:00.453613 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf2508bc_b378_4bd4_8306_36ed5bcdee3d.slice/crio-2f62cf32c429b803c0475d7159d2b2f4367f21bc8229a725cae6cfa2515c806f WatchSource:0}: Error finding container 2f62cf32c429b803c0475d7159d2b2f4367f21bc8229a725cae6cfa2515c806f: Status 404 returned error can't find the container with id 2f62cf32c429b803c0475d7159d2b2f4367f21bc8229a725cae6cfa2515c806f Sep 06 12:45:00.454673 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:00.454647 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29811645-n24zn"] Sep 06 12:45:00.456952 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:45:00.456930 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f0b970c_996d_457e_b26b_66e6b997b591.slice/crio-6368dafebe7cac07c1952d10625bd2e0d425819f27ad3f2088c34f85e759c0d9 WatchSource:0}: Error finding container 6368dafebe7cac07c1952d10625bd2e0d425819f27ad3f2088c34f85e759c0d9: Status 404 returned error can't find the container with id 6368dafebe7cac07c1952d10625bd2e0d425819f27ad3f2088c34f85e759c0d9 Sep 06 12:45:01.266797 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:01.266740 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c82458-5b6bdf5b6b-4jq27"] Sep 06 12:45:01.271621 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:01.271599 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c82458-5b6bdf5b6b-4jq27" Sep 06 12:45:01.274107 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:01.274086 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-c82458\"" Sep 06 12:45:01.274218 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:01.274086 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-c82458-dockercfg-w8scz\"" Sep 06 12:45:01.281816 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:01.281795 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c82458-5b6bdf5b6b-4jq27"] Sep 06 12:45:01.289893 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:01.289872 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c82458-55d79fc9c9-44fm8"] Sep 06 12:45:01.293583 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:01.293559 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c82458-55d79fc9c9-44fm8" Sep 06 12:45:01.304711 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:01.304688 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c82458-55d79fc9c9-44fm8"] Sep 06 12:45:01.313159 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:01.313139 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aee8d14-7567-41a1-aed4-6469d5e3782d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-c82458-55d79fc9c9-44fm8\" (UID: \"1aee8d14-7567-41a1-aed4-6469d5e3782d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c82458-55d79fc9c9-44fm8" Sep 06 12:45:01.313253 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:01.313183 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spbvm\" (UniqueName: \"kubernetes.io/projected/1aee8d14-7567-41a1-aed4-6469d5e3782d-kube-api-access-spbvm\") pod \"payload-processing-e2e-ipp-cleanup-c82458-55d79fc9c9-44fm8\" (UID: \"1aee8d14-7567-41a1-aed4-6469d5e3782d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c82458-55d79fc9c9-44fm8" Sep 06 12:45:01.313253 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:01.313206 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8929\" (UniqueName: \"kubernetes.io/projected/d0e5f0f1-b1f6-4f2a-8ffd-c4afb58b211e-kube-api-access-q8929\") pod \"payload-pre-processing-e2e-ipp-cleanup-c82458-5b6bdf5b6b-4jq27\" (UID: \"d0e5f0f1-b1f6-4f2a-8ffd-c4afb58b211e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c82458-5b6bdf5b6b-4jq27" Sep 06 12:45:01.313348 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:01.313275 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0e5f0f1-b1f6-4f2a-8ffd-c4afb58b211e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-c82458-5b6bdf5b6b-4jq27\" (UID: \"d0e5f0f1-b1f6-4f2a-8ffd-c4afb58b211e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c82458-5b6bdf5b6b-4jq27" Sep 06 12:45:01.335845 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:01.335813 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811645-n24zn" event={"ID":"2f0b970c-996d-457e-b26b-66e6b997b591","Type":"ContainerStarted","Data":"6368dafebe7cac07c1952d10625bd2e0d425819f27ad3f2088c34f85e759c0d9"} Sep 06 12:45:01.337165 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:01.337143 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e6515e-29811645-rf8nv" event={"ID":"cf2508bc-b378-4bd4-8306-36ed5bcdee3d","Type":"ContainerStarted","Data":"2f62cf32c429b803c0475d7159d2b2f4367f21bc8229a725cae6cfa2515c806f"} Sep 06 12:45:01.414452 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:01.414409 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spbvm\" (UniqueName: \"kubernetes.io/projected/1aee8d14-7567-41a1-aed4-6469d5e3782d-kube-api-access-spbvm\") pod \"payload-processing-e2e-ipp-cleanup-c82458-55d79fc9c9-44fm8\" (UID: \"1aee8d14-7567-41a1-aed4-6469d5e3782d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c82458-55d79fc9c9-44fm8" Sep 06 12:45:01.414452 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:01.414452 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8929\" (UniqueName: \"kubernetes.io/projected/d0e5f0f1-b1f6-4f2a-8ffd-c4afb58b211e-kube-api-access-q8929\") pod \"payload-pre-processing-e2e-ipp-cleanup-c82458-5b6bdf5b6b-4jq27\" (UID: \"d0e5f0f1-b1f6-4f2a-8ffd-c4afb58b211e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c82458-5b6bdf5b6b-4jq27" Sep 06 12:45:01.414699 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:01.414492 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0e5f0f1-b1f6-4f2a-8ffd-c4afb58b211e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-c82458-5b6bdf5b6b-4jq27\" (UID: \"d0e5f0f1-b1f6-4f2a-8ffd-c4afb58b211e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c82458-5b6bdf5b6b-4jq27" Sep 06 12:45:01.414699 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:01.414590 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aee8d14-7567-41a1-aed4-6469d5e3782d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-c82458-55d79fc9c9-44fm8\" (UID: \"1aee8d14-7567-41a1-aed4-6469d5e3782d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c82458-55d79fc9c9-44fm8" Sep 06 12:45:01.416037 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:01.415484 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0e5f0f1-b1f6-4f2a-8ffd-c4afb58b211e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-c82458-5b6bdf5b6b-4jq27\" (UID: \"d0e5f0f1-b1f6-4f2a-8ffd-c4afb58b211e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c82458-5b6bdf5b6b-4jq27" Sep 06 12:45:01.416037 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:01.415587 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aee8d14-7567-41a1-aed4-6469d5e3782d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-c82458-55d79fc9c9-44fm8\" (UID: \"1aee8d14-7567-41a1-aed4-6469d5e3782d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c82458-55d79fc9c9-44fm8" Sep 06 12:45:01.427023 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:01.426989 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8929\" (UniqueName: \"kubernetes.io/projected/d0e5f0f1-b1f6-4f2a-8ffd-c4afb58b211e-kube-api-access-q8929\") pod \"payload-pre-processing-e2e-ipp-cleanup-c82458-5b6bdf5b6b-4jq27\" (UID: \"d0e5f0f1-b1f6-4f2a-8ffd-c4afb58b211e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c82458-5b6bdf5b6b-4jq27" Sep 06 12:45:01.427693 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:01.427657 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spbvm\" (UniqueName: \"kubernetes.io/projected/1aee8d14-7567-41a1-aed4-6469d5e3782d-kube-api-access-spbvm\") pod \"payload-processing-e2e-ipp-cleanup-c82458-55d79fc9c9-44fm8\" (UID: \"1aee8d14-7567-41a1-aed4-6469d5e3782d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c82458-55d79fc9c9-44fm8" Sep 06 12:45:01.585129 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:01.585037 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c82458-5b6bdf5b6b-4jq27" Sep 06 12:45:01.604017 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:01.603981 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c82458-55d79fc9c9-44fm8" Sep 06 12:45:01.887123 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:01.887094 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c82458-5b6bdf5b6b-4jq27"] Sep 06 12:45:01.889481 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:45:01.889420 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0e5f0f1_b1f6_4f2a_8ffd_c4afb58b211e.slice/crio-1097fce4386ca8136ea304a1a73c06c0ce9aefa252a25ec526a5890f4b684ace WatchSource:0}: Error finding container 1097fce4386ca8136ea304a1a73c06c0ce9aefa252a25ec526a5890f4b684ace: Status 404 returned error can't find the container with id 1097fce4386ca8136ea304a1a73c06c0ce9aefa252a25ec526a5890f4b684ace Sep 06 12:45:01.907713 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:01.907685 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c82458-55d79fc9c9-44fm8"] Sep 06 12:45:01.910657 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:45:01.910627 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aee8d14_7567_41a1_aed4_6469d5e3782d.slice/crio-68de80a0a5172c06c775df23ce29b8c29f16dd2008e93523867566325c8db7ce WatchSource:0}: Error finding container 68de80a0a5172c06c775df23ce29b8c29f16dd2008e93523867566325c8db7ce: Status 404 returned error can't find the container with id 68de80a0a5172c06c775df23ce29b8c29f16dd2008e93523867566325c8db7ce Sep 06 12:45:02.351187 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:02.351144 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c82458-5b6bdf5b6b-4jq27" event={"ID":"d0e5f0f1-b1f6-4f2a-8ffd-c4afb58b211e","Type":"ContainerStarted","Data":"50dcce74aecebececa74f371e67930a0aebc0df2088d75f8de4051b656e624b0"} Sep 06 12:45:02.351659 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:02.351227 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c82458-5b6bdf5b6b-4jq27" event={"ID":"d0e5f0f1-b1f6-4f2a-8ffd-c4afb58b211e","Type":"ContainerStarted","Data":"1097fce4386ca8136ea304a1a73c06c0ce9aefa252a25ec526a5890f4b684ace"} Sep 06 12:45:02.351745 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:02.351700 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c82458-5b6bdf5b6b-4jq27" Sep 06 12:45:02.353914 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:02.353886 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c82458-55d79fc9c9-44fm8" event={"ID":"1aee8d14-7567-41a1-aed4-6469d5e3782d","Type":"ContainerStarted","Data":"c739e847c49276d2601ba8b1278a5f08a843d8297358ad8ab5839e4234ca225d"} Sep 06 12:45:02.354047 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:02.353933 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c82458-55d79fc9c9-44fm8" event={"ID":"1aee8d14-7567-41a1-aed4-6469d5e3782d","Type":"ContainerStarted","Data":"68de80a0a5172c06c775df23ce29b8c29f16dd2008e93523867566325c8db7ce"} Sep 06 12:45:02.354047 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:02.353991 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c82458-55d79fc9c9-44fm8" Sep 06 12:45:02.367867 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:02.367822 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c82458-5b6bdf5b6b-4jq27" podStartSLOduration=1.367805549 podStartE2EDuration="1.367805549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:45:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:45:02.36632855 +0000 UTC m=+1618.222086439" watchObservedRunningTime="2026-09-06 12:45:02.367805549 +0000 UTC m=+1618.223563437" Sep 06 12:45:02.382343 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:02.382295 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c82458-55d79fc9c9-44fm8" podStartSLOduration=1.382279179 podStartE2EDuration="1.382279179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:45:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:45:02.381246319 +0000 UTC m=+1618.237004233" watchObservedRunningTime="2026-09-06 12:45:02.382279179 +0000 UTC m=+1618.238037067" Sep 06 12:45:03.361965 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:03.361932 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811645-n24zn" event={"ID":"2f0b970c-996d-457e-b26b-66e6b997b591","Type":"ContainerStarted","Data":"3e512d0635b35eb619d9aaa4dca021adf6c48d6d4d6e6ab186e2effb4d13f9b3"} Sep 06 12:45:03.377414 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:03.377363 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811645-n24zn" podStartSLOduration=0.843965676 podStartE2EDuration="3.377340762s" totalImagesPullingTime="2.533375086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:45:03.375703544 +0000 UTC m=+1619.231461456" watchObservedRunningTime="2026-09-06 12:45:03.377340762 +0000 UTC m=+1619.233098651" Sep 06 12:45:04.369289 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:04.369220 2576 generic.go:356] "Generic (PLEG): container finished" podID="2f0b970c-996d-457e-b26b-66e6b997b591" containerID="3e512d0635b35eb619d9aaa4dca021adf6c48d6d4d6e6ab186e2effb4d13f9b3" exitCode=0 Sep 06 12:45:04.369636 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:04.369304 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811645-n24zn" event={"ID":"2f0b970c-996d-457e-b26b-66e6b997b591","Type":"ContainerDied","Data":"3e512d0635b35eb619d9aaa4dca021adf6c48d6d4d6e6ab186e2effb4d13f9b3"} Sep 06 12:45:04.370842 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:04.370816 2576 generic.go:356] "Generic (PLEG): container finished" podID="cf2508bc-b378-4bd4-8306-36ed5bcdee3d" containerID="0b7d7fddbe46e305438b275459d74102bfd7f5b40e69e2fe834d125955c06a31" exitCode=0 Sep 06 12:45:04.370945 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:04.370901 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e6515e-29811645-rf8nv" event={"ID":"cf2508bc-b378-4bd4-8306-36ed5bcdee3d","Type":"ContainerDied","Data":"0b7d7fddbe46e305438b275459d74102bfd7f5b40e69e2fe834d125955c06a31"} Sep 06 12:45:05.514468 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:05.514444 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811645-n24zn" Sep 06 12:45:05.574937 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:05.574915 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e6515e-29811645-rf8nv" Sep 06 12:45:06.384469 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:06.384437 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811645-n24zn" Sep 06 12:45:06.384469 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:06.384449 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811645-n24zn" event={"ID":"2f0b970c-996d-457e-b26b-66e6b997b591","Type":"ContainerDied","Data":"6368dafebe7cac07c1952d10625bd2e0d425819f27ad3f2088c34f85e759c0d9"} Sep 06 12:45:06.384779 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:06.384502 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6368dafebe7cac07c1952d10625bd2e0d425819f27ad3f2088c34f85e759c0d9" Sep 06 12:45:06.386118 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:06.386095 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e6515e-29811645-rf8nv" Sep 06 12:45:06.386240 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:06.386126 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e6515e-29811645-rf8nv" event={"ID":"cf2508bc-b378-4bd4-8306-36ed5bcdee3d","Type":"ContainerDied","Data":"2f62cf32c429b803c0475d7159d2b2f4367f21bc8229a725cae6cfa2515c806f"} Sep 06 12:45:06.386240 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:06.386163 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f62cf32c429b803c0475d7159d2b2f4367f21bc8229a725cae6cfa2515c806f" Sep 06 12:45:13.364677 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:13.364639 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c82458-55d79fc9c9-44fm8" Sep 06 12:45:14.373709 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:14.373668 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c82458-5b6bdf5b6b-4jq27" Sep 06 12:45:31.100818 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:31.100784 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c82458-55d79fc9c9-44fm8"] Sep 06 12:45:31.101423 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:31.101076 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c82458-55d79fc9c9-44fm8" podUID="1aee8d14-7567-41a1-aed4-6469d5e3782d" containerName="payload-processing" containerID="cri-o://c739e847c49276d2601ba8b1278a5f08a843d8297358ad8ab5839e4234ca225d" gracePeriod=30 Sep 06 12:45:31.109659 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:31.108822 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c82458-5b6bdf5b6b-4jq27"] Sep 06 12:45:31.109659 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:31.109028 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c82458-5b6bdf5b6b-4jq27" podUID="d0e5f0f1-b1f6-4f2a-8ffd-c4afb58b211e" containerName="payload-pre-processing" containerID="cri-o://50dcce74aecebececa74f371e67930a0aebc0df2088d75f8de4051b656e624b0" gracePeriod=30 Sep 06 12:45:31.360639 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:31.360581 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c82458-55d79fc9c9-44fm8" Sep 06 12:45:31.392300 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:31.392275 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c82458-5b6bdf5b6b-4jq27" Sep 06 12:45:31.474482 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:31.474453 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aee8d14-7567-41a1-aed4-6469d5e3782d-plugins-config-volume\") pod \"1aee8d14-7567-41a1-aed4-6469d5e3782d\" (UID: \"1aee8d14-7567-41a1-aed4-6469d5e3782d\") " Sep 06 12:45:31.474654 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:31.474507 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0e5f0f1-b1f6-4f2a-8ffd-c4afb58b211e-plugins-config-volume\") pod \"d0e5f0f1-b1f6-4f2a-8ffd-c4afb58b211e\" (UID: \"d0e5f0f1-b1f6-4f2a-8ffd-c4afb58b211e\") " Sep 06 12:45:31.474654 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:31.474533 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spbvm\" (UniqueName: \"kubernetes.io/projected/1aee8d14-7567-41a1-aed4-6469d5e3782d-kube-api-access-spbvm\") pod \"1aee8d14-7567-41a1-aed4-6469d5e3782d\" (UID: \"1aee8d14-7567-41a1-aed4-6469d5e3782d\") " Sep 06 12:45:31.474654 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:31.474645 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8929\" (UniqueName: \"kubernetes.io/projected/d0e5f0f1-b1f6-4f2a-8ffd-c4afb58b211e-kube-api-access-q8929\") pod \"d0e5f0f1-b1f6-4f2a-8ffd-c4afb58b211e\" (UID: \"d0e5f0f1-b1f6-4f2a-8ffd-c4afb58b211e\") " Sep 06 12:45:31.474952 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:31.474912 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aee8d14-7567-41a1-aed4-6469d5e3782d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1aee8d14-7567-41a1-aed4-6469d5e3782d" (UID: "1aee8d14-7567-41a1-aed4-6469d5e3782d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:45:31.474952 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:31.474911 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0e5f0f1-b1f6-4f2a-8ffd-c4afb58b211e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d0e5f0f1-b1f6-4f2a-8ffd-c4afb58b211e" (UID: "d0e5f0f1-b1f6-4f2a-8ffd-c4afb58b211e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:45:31.475112 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:31.475097 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aee8d14-7567-41a1-aed4-6469d5e3782d-plugins-config-volume\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:45:31.475162 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:31.475114 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0e5f0f1-b1f6-4f2a-8ffd-c4afb58b211e-plugins-config-volume\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:45:31.476812 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:31.476789 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0e5f0f1-b1f6-4f2a-8ffd-c4afb58b211e-kube-api-access-q8929" (OuterVolumeSpecName: "kube-api-access-q8929") pod "d0e5f0f1-b1f6-4f2a-8ffd-c4afb58b211e" (UID: "d0e5f0f1-b1f6-4f2a-8ffd-c4afb58b211e"). InnerVolumeSpecName "kube-api-access-q8929". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:45:31.476916 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:31.476898 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aee8d14-7567-41a1-aed4-6469d5e3782d-kube-api-access-spbvm" (OuterVolumeSpecName: "kube-api-access-spbvm") pod "1aee8d14-7567-41a1-aed4-6469d5e3782d" (UID: "1aee8d14-7567-41a1-aed4-6469d5e3782d"). InnerVolumeSpecName "kube-api-access-spbvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:45:31.530354 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:31.530324 2576 generic.go:356] "Generic (PLEG): container finished" podID="d0e5f0f1-b1f6-4f2a-8ffd-c4afb58b211e" containerID="50dcce74aecebececa74f371e67930a0aebc0df2088d75f8de4051b656e624b0" exitCode=0 Sep 06 12:45:31.530451 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:31.530364 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c82458-5b6bdf5b6b-4jq27" Sep 06 12:45:31.530451 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:31.530420 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c82458-5b6bdf5b6b-4jq27" event={"ID":"d0e5f0f1-b1f6-4f2a-8ffd-c4afb58b211e","Type":"ContainerDied","Data":"50dcce74aecebececa74f371e67930a0aebc0df2088d75f8de4051b656e624b0"} Sep 06 12:45:31.530555 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:31.530468 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c82458-5b6bdf5b6b-4jq27" event={"ID":"d0e5f0f1-b1f6-4f2a-8ffd-c4afb58b211e","Type":"ContainerDied","Data":"1097fce4386ca8136ea304a1a73c06c0ce9aefa252a25ec526a5890f4b684ace"} Sep 06 12:45:31.530555 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:31.530490 2576 scope.go:117] "RemoveContainer" containerID="50dcce74aecebececa74f371e67930a0aebc0df2088d75f8de4051b656e624b0" Sep 06 12:45:31.531964 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:31.531939 2576 generic.go:356] "Generic (PLEG): container finished" podID="1aee8d14-7567-41a1-aed4-6469d5e3782d" containerID="c739e847c49276d2601ba8b1278a5f08a843d8297358ad8ab5839e4234ca225d" exitCode=0 Sep 06 12:45:31.532060 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:31.531977 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c82458-55d79fc9c9-44fm8" event={"ID":"1aee8d14-7567-41a1-aed4-6469d5e3782d","Type":"ContainerDied","Data":"c739e847c49276d2601ba8b1278a5f08a843d8297358ad8ab5839e4234ca225d"} Sep 06 12:45:31.532060 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:31.531982 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c82458-55d79fc9c9-44fm8" Sep 06 12:45:31.532060 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:31.532016 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c82458-55d79fc9c9-44fm8" event={"ID":"1aee8d14-7567-41a1-aed4-6469d5e3782d","Type":"ContainerDied","Data":"68de80a0a5172c06c775df23ce29b8c29f16dd2008e93523867566325c8db7ce"} Sep 06 12:45:31.539885 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:31.539866 2576 scope.go:117] "RemoveContainer" containerID="50dcce74aecebececa74f371e67930a0aebc0df2088d75f8de4051b656e624b0" Sep 06 12:45:31.540150 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:45:31.540119 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50dcce74aecebececa74f371e67930a0aebc0df2088d75f8de4051b656e624b0\": container with ID starting with 50dcce74aecebececa74f371e67930a0aebc0df2088d75f8de4051b656e624b0 not found: ID does not exist" containerID="50dcce74aecebececa74f371e67930a0aebc0df2088d75f8de4051b656e624b0" Sep 06 12:45:31.540251 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:31.540154 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50dcce74aecebececa74f371e67930a0aebc0df2088d75f8de4051b656e624b0"} err="failed to get container status \"50dcce74aecebececa74f371e67930a0aebc0df2088d75f8de4051b656e624b0\": rpc error: code = NotFound desc = could not find container \"50dcce74aecebececa74f371e67930a0aebc0df2088d75f8de4051b656e624b0\": container with ID starting with 50dcce74aecebececa74f371e67930a0aebc0df2088d75f8de4051b656e624b0 not found: ID does not exist" Sep 06 12:45:31.540251 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:31.540170 2576 scope.go:117] "RemoveContainer" containerID="c739e847c49276d2601ba8b1278a5f08a843d8297358ad8ab5839e4234ca225d" Sep 06 12:45:31.550462 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:31.550439 2576 scope.go:117] "RemoveContainer" containerID="c739e847c49276d2601ba8b1278a5f08a843d8297358ad8ab5839e4234ca225d" Sep 06 12:45:31.550744 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:45:31.550723 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c739e847c49276d2601ba8b1278a5f08a843d8297358ad8ab5839e4234ca225d\": container with ID starting with c739e847c49276d2601ba8b1278a5f08a843d8297358ad8ab5839e4234ca225d not found: ID does not exist" containerID="c739e847c49276d2601ba8b1278a5f08a843d8297358ad8ab5839e4234ca225d" Sep 06 12:45:31.550857 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:31.550773 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c739e847c49276d2601ba8b1278a5f08a843d8297358ad8ab5839e4234ca225d"} err="failed to get container status \"c739e847c49276d2601ba8b1278a5f08a843d8297358ad8ab5839e4234ca225d\": rpc error: code = NotFound desc = could not find container \"c739e847c49276d2601ba8b1278a5f08a843d8297358ad8ab5839e4234ca225d\": container with ID starting with c739e847c49276d2601ba8b1278a5f08a843d8297358ad8ab5839e4234ca225d not found: ID does not exist" Sep 06 12:45:31.562790 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:31.562746 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c82458-55d79fc9c9-44fm8"] Sep 06 12:45:31.567398 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:31.567378 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c82458-55d79fc9c9-44fm8"] Sep 06 12:45:31.575966 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:31.575946 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c82458-5b6bdf5b6b-4jq27"] Sep 06 12:45:31.576376 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:31.576358 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spbvm\" (UniqueName: \"kubernetes.io/projected/1aee8d14-7567-41a1-aed4-6469d5e3782d-kube-api-access-spbvm\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:45:31.576442 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:31.576380 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8929\" (UniqueName: \"kubernetes.io/projected/d0e5f0f1-b1f6-4f2a-8ffd-c4afb58b211e-kube-api-access-q8929\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:45:31.580087 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:31.580065 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c82458-5b6bdf5b6b-4jq27"] Sep 06 12:45:32.653428 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:32.653389 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1aee8d14-7567-41a1-aed4-6469d5e3782d" path="/var/lib/kubelet/pods/1aee8d14-7567-41a1-aed4-6469d5e3782d/volumes" Sep 06 12:45:32.653957 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:32.653821 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0e5f0f1-b1f6-4f2a-8ffd-c4afb58b211e" path="/var/lib/kubelet/pods/d0e5f0f1-b1f6-4f2a-8ffd-c4afb58b211e/volumes" Sep 06 12:45:56.358453 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.358320 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst"] Sep 06 12:45:56.358936 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.358587 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst" podUID="7b533bd8-7770-4a73-9c49-a768f1273b2b" containerName="maas-api" containerID="cri-o://f4318170c4c49b01bdbb4d014f08976a595e3d687017e58f780ab2d9f7cd12d1" gracePeriod=30 Sep 06 12:45:56.412515 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.412481 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4f1951-744f4455f7-p949b"] Sep 06 12:45:56.413319 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.412806 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-4f1951-744f4455f7-p949b" podUID="591382be-edad-43f5-af3b-1ebc658eb823" containerName="payload-processing" containerID="cri-o://f552988df2821a35d4f7bf8c9ab5179e9485061ac51ed15457b0b44a9211f6da" gracePeriod=30 Sep 06 12:45:56.420862 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.420836 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4f1951-b654889c4-fvct2"] Sep 06 12:45:56.421335 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.421190 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f1951-b654889c4-fvct2" podUID="7ee118fe-a2e9-4753-9bfc-6d2d8f0da070" containerName="payload-pre-processing" containerID="cri-o://dadde2fbe89a3a2de216d62db9329e5d616e38417cb9b92470cf257d89cabe12" gracePeriod=30 Sep 06 12:45:56.684183 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.684152 2576 generic.go:356] "Generic (PLEG): container finished" podID="7ee118fe-a2e9-4753-9bfc-6d2d8f0da070" containerID="dadde2fbe89a3a2de216d62db9329e5d616e38417cb9b92470cf257d89cabe12" exitCode=0 Sep 06 12:45:56.684333 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.684241 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f1951-b654889c4-fvct2" event={"ID":"7ee118fe-a2e9-4753-9bfc-6d2d8f0da070","Type":"ContainerDied","Data":"dadde2fbe89a3a2de216d62db9329e5d616e38417cb9b92470cf257d89cabe12"} Sep 06 12:45:56.686442 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.686319 2576 generic.go:356] "Generic (PLEG): container finished" podID="591382be-edad-43f5-af3b-1ebc658eb823" containerID="f552988df2821a35d4f7bf8c9ab5179e9485061ac51ed15457b0b44a9211f6da" exitCode=0 Sep 06 12:45:56.686442 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.686364 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4f1951-744f4455f7-p949b" event={"ID":"591382be-edad-43f5-af3b-1ebc658eb823","Type":"ContainerDied","Data":"f552988df2821a35d4f7bf8c9ab5179e9485061ac51ed15457b0b44a9211f6da"} Sep 06 12:45:56.690423 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.688166 2576 generic.go:356] "Generic (PLEG): container finished" podID="7b533bd8-7770-4a73-9c49-a768f1273b2b" containerID="f4318170c4c49b01bdbb4d014f08976a595e3d687017e58f780ab2d9f7cd12d1" exitCode=0 Sep 06 12:45:56.690423 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.688328 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst" event={"ID":"7b533bd8-7770-4a73-9c49-a768f1273b2b","Type":"ContainerDied","Data":"f4318170c4c49b01bdbb4d014f08976a595e3d687017e58f780ab2d9f7cd12d1"} Sep 06 12:45:56.690423 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.688370 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst" event={"ID":"7b533bd8-7770-4a73-9c49-a768f1273b2b","Type":"ContainerDied","Data":"99da6c167fee0553716c496efbf33de5722abe9fcc9203957bcf584701554be5"} Sep 06 12:45:56.690423 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.688393 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99da6c167fee0553716c496efbf33de5722abe9fcc9203957bcf584701554be5" Sep 06 12:45:56.700803 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.700640 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst" Sep 06 12:45:56.705023 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.705005 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4f1951-744f4455f7-p949b" Sep 06 12:45:56.718675 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.718654 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f1951-b654889c4-fvct2" Sep 06 12:45:56.795730 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.795699 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbd8b\" (UniqueName: \"kubernetes.io/projected/7ee118fe-a2e9-4753-9bfc-6d2d8f0da070-kube-api-access-rbd8b\") pod \"7ee118fe-a2e9-4753-9bfc-6d2d8f0da070\" (UID: \"7ee118fe-a2e9-4753-9bfc-6d2d8f0da070\") " Sep 06 12:45:56.795730 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.795731 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/591382be-edad-43f5-af3b-1ebc658eb823-plugins-config-volume\") pod \"591382be-edad-43f5-af3b-1ebc658eb823\" (UID: \"591382be-edad-43f5-af3b-1ebc658eb823\") " Sep 06 12:45:56.795950 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.795796 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b533bd8-7770-4a73-9c49-a768f1273b2b-metrics-certs\") pod \"7b533bd8-7770-4a73-9c49-a768f1273b2b\" (UID: \"7b533bd8-7770-4a73-9c49-a768f1273b2b\") " Sep 06 12:45:56.795950 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.795845 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee118fe-a2e9-4753-9bfc-6d2d8f0da070-plugins-config-volume\") pod \"7ee118fe-a2e9-4753-9bfc-6d2d8f0da070\" (UID: \"7ee118fe-a2e9-4753-9bfc-6d2d8f0da070\") " Sep 06 12:45:56.795950 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.795902 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vc9xd\" (UniqueName: \"kubernetes.io/projected/591382be-edad-43f5-af3b-1ebc658eb823-kube-api-access-vc9xd\") pod \"591382be-edad-43f5-af3b-1ebc658eb823\" (UID: \"591382be-edad-43f5-af3b-1ebc658eb823\") " Sep 06 12:45:56.795950 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.795944 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hq9j\" (UniqueName: \"kubernetes.io/projected/7b533bd8-7770-4a73-9c49-a768f1273b2b-kube-api-access-6hq9j\") pod \"7b533bd8-7770-4a73-9c49-a768f1273b2b\" (UID: \"7b533bd8-7770-4a73-9c49-a768f1273b2b\") " Sep 06 12:45:56.796167 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.795996 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b533bd8-7770-4a73-9c49-a768f1273b2b-maas-api-tls\") pod \"7b533bd8-7770-4a73-9c49-a768f1273b2b\" (UID: \"7b533bd8-7770-4a73-9c49-a768f1273b2b\") " Sep 06 12:45:56.796167 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.796034 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b533bd8-7770-4a73-9c49-a768f1273b2b-openshift-service-ca\") pod \"7b533bd8-7770-4a73-9c49-a768f1273b2b\" (UID: \"7b533bd8-7770-4a73-9c49-a768f1273b2b\") " Sep 06 12:45:56.796283 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.796218 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/591382be-edad-43f5-af3b-1ebc658eb823-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "591382be-edad-43f5-af3b-1ebc658eb823" (UID: "591382be-edad-43f5-af3b-1ebc658eb823"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:45:56.796283 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.796268 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee118fe-a2e9-4753-9bfc-6d2d8f0da070-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7ee118fe-a2e9-4753-9bfc-6d2d8f0da070" (UID: "7ee118fe-a2e9-4753-9bfc-6d2d8f0da070"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:45:56.796739 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.796614 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b533bd8-7770-4a73-9c49-a768f1273b2b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7b533bd8-7770-4a73-9c49-a768f1273b2b" (UID: "7b533bd8-7770-4a73-9c49-a768f1273b2b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:45:56.796881 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.796775 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee118fe-a2e9-4753-9bfc-6d2d8f0da070-plugins-config-volume\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:45:56.796881 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.796800 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b533bd8-7770-4a73-9c49-a768f1273b2b-openshift-service-ca\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:45:56.796881 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.796815 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/591382be-edad-43f5-af3b-1ebc658eb823-plugins-config-volume\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:45:56.798229 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.798197 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/591382be-edad-43f5-af3b-1ebc658eb823-kube-api-access-vc9xd" (OuterVolumeSpecName: "kube-api-access-vc9xd") pod "591382be-edad-43f5-af3b-1ebc658eb823" (UID: "591382be-edad-43f5-af3b-1ebc658eb823"). InnerVolumeSpecName "kube-api-access-vc9xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:45:56.798229 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.798218 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b533bd8-7770-4a73-9c49-a768f1273b2b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7b533bd8-7770-4a73-9c49-a768f1273b2b" (UID: "7b533bd8-7770-4a73-9c49-a768f1273b2b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 12:45:56.798360 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.798268 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b533bd8-7770-4a73-9c49-a768f1273b2b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7b533bd8-7770-4a73-9c49-a768f1273b2b" (UID: "7b533bd8-7770-4a73-9c49-a768f1273b2b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 12:45:56.798604 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.798583 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee118fe-a2e9-4753-9bfc-6d2d8f0da070-kube-api-access-rbd8b" (OuterVolumeSpecName: "kube-api-access-rbd8b") pod "7ee118fe-a2e9-4753-9bfc-6d2d8f0da070" (UID: "7ee118fe-a2e9-4753-9bfc-6d2d8f0da070"). InnerVolumeSpecName "kube-api-access-rbd8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:45:56.798745 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.798726 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b533bd8-7770-4a73-9c49-a768f1273b2b-kube-api-access-6hq9j" (OuterVolumeSpecName: "kube-api-access-6hq9j") pod "7b533bd8-7770-4a73-9c49-a768f1273b2b" (UID: "7b533bd8-7770-4a73-9c49-a768f1273b2b"). InnerVolumeSpecName "kube-api-access-6hq9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:45:56.897897 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.897874 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b533bd8-7770-4a73-9c49-a768f1273b2b-metrics-certs\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:45:56.897897 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.897895 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc9xd\" (UniqueName: \"kubernetes.io/projected/591382be-edad-43f5-af3b-1ebc658eb823-kube-api-access-vc9xd\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:45:56.898024 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.897906 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hq9j\" (UniqueName: \"kubernetes.io/projected/7b533bd8-7770-4a73-9c49-a768f1273b2b-kube-api-access-6hq9j\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:45:56.898024 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.897915 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b533bd8-7770-4a73-9c49-a768f1273b2b-maas-api-tls\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:45:56.898024 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:56.897924 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbd8b\" (UniqueName: \"kubernetes.io/projected/7ee118fe-a2e9-4753-9bfc-6d2d8f0da070-kube-api-access-rbd8b\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:45:57.695171 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:57.695096 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f1951-b654889c4-fvct2" Sep 06 12:45:57.695570 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:57.695095 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f1951-b654889c4-fvct2" event={"ID":"7ee118fe-a2e9-4753-9bfc-6d2d8f0da070","Type":"ContainerDied","Data":"dcfc54f6d522b76503197656e36a5b01e9ea3eea54a82fa22bad964bea49dcfd"} Sep 06 12:45:57.695570 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:57.695244 2576 scope.go:117] "RemoveContainer" containerID="dadde2fbe89a3a2de216d62db9329e5d616e38417cb9b92470cf257d89cabe12" Sep 06 12:45:57.696655 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:57.696632 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst" Sep 06 12:45:57.696714 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:57.696648 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4f1951-744f4455f7-p949b" event={"ID":"591382be-edad-43f5-af3b-1ebc658eb823","Type":"ContainerDied","Data":"935508acf32bbd1b6298a51bd21145f1a138fb4c996db8cd020674332abb17e7"} Sep 06 12:45:57.696776 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:57.696632 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4f1951-744f4455f7-p949b" Sep 06 12:45:57.705594 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:57.705488 2576 scope.go:117] "RemoveContainer" containerID="f552988df2821a35d4f7bf8c9ab5179e9485061ac51ed15457b0b44a9211f6da" Sep 06 12:45:57.722596 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:57.722570 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4f1951-b654889c4-fvct2"] Sep 06 12:45:57.728842 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:57.728822 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4f1951-b654889c4-fvct2"] Sep 06 12:45:57.738169 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:57.738144 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4f1951-744f4455f7-p949b"] Sep 06 12:45:57.741418 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:57.741397 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4f1951-744f4455f7-p949b"] Sep 06 12:45:57.751070 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:57.751049 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst"] Sep 06 12:45:57.754807 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:57.754787 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4f1951-8548dc96c-dlqst"] Sep 06 12:45:58.653585 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:58.653551 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="591382be-edad-43f5-af3b-1ebc658eb823" path="/var/lib/kubelet/pods/591382be-edad-43f5-af3b-1ebc658eb823/volumes" Sep 06 12:45:58.653943 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:58.653927 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b533bd8-7770-4a73-9c49-a768f1273b2b" path="/var/lib/kubelet/pods/7b533bd8-7770-4a73-9c49-a768f1273b2b/volumes" Sep 06 12:45:58.654236 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:45:58.654224 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ee118fe-a2e9-4753-9bfc-6d2d8f0da070" path="/var/lib/kubelet/pods/7ee118fe-a2e9-4753-9bfc-6d2d8f0da070/volumes" Sep 06 12:47:13.091978 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:13.091946 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3feb92-6ddb5878b5-c6qsd"] Sep 06 12:47:13.092503 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:13.092166 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3feb92-6ddb5878b5-c6qsd" podUID="014d29cf-4ab3-4600-947b-a6b67306edb2" containerName="payload-processing" containerID="cri-o://b32646351a601b833fd5f28be3dce3b75e3d7c0c46e55967da3382ed86809f4e" gracePeriod=30 Sep 06 12:47:13.102062 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:13.102024 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3feb92-64d8bcfbc8-hx7mq"] Sep 06 12:47:13.102536 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:13.102484 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3feb92-64d8bcfbc8-hx7mq" podUID="eb2b858b-eb53-4268-a0e4-e0dde762ae77" containerName="payload-pre-processing" containerID="cri-o://74a82cb9d3ce080972485e02e43ba87300e3dbe0cd7a8c3dedf63f2c870949cd" gracePeriod=30 Sep 06 12:47:13.370117 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:13.370096 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3feb92-6ddb5878b5-c6qsd" Sep 06 12:47:13.373528 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:13.373505 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3feb92-64d8bcfbc8-hx7mq" Sep 06 12:47:13.466608 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:13.466578 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47dpx\" (UniqueName: \"kubernetes.io/projected/eb2b858b-eb53-4268-a0e4-e0dde762ae77-kube-api-access-47dpx\") pod \"eb2b858b-eb53-4268-a0e4-e0dde762ae77\" (UID: \"eb2b858b-eb53-4268-a0e4-e0dde762ae77\") " Sep 06 12:47:13.466797 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:13.466648 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/014d29cf-4ab3-4600-947b-a6b67306edb2-plugins-config-volume\") pod \"014d29cf-4ab3-4600-947b-a6b67306edb2\" (UID: \"014d29cf-4ab3-4600-947b-a6b67306edb2\") " Sep 06 12:47:13.466797 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:13.466675 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9hx5\" (UniqueName: \"kubernetes.io/projected/014d29cf-4ab3-4600-947b-a6b67306edb2-kube-api-access-t9hx5\") pod \"014d29cf-4ab3-4600-947b-a6b67306edb2\" (UID: \"014d29cf-4ab3-4600-947b-a6b67306edb2\") " Sep 06 12:47:13.466921 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:13.466822 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb2b858b-eb53-4268-a0e4-e0dde762ae77-plugins-config-volume\") pod \"eb2b858b-eb53-4268-a0e4-e0dde762ae77\" (UID: \"eb2b858b-eb53-4268-a0e4-e0dde762ae77\") " Sep 06 12:47:13.467159 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:13.467126 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/014d29cf-4ab3-4600-947b-a6b67306edb2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "014d29cf-4ab3-4600-947b-a6b67306edb2" (UID: "014d29cf-4ab3-4600-947b-a6b67306edb2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:47:13.467281 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:13.467177 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb2b858b-eb53-4268-a0e4-e0dde762ae77-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eb2b858b-eb53-4268-a0e4-e0dde762ae77" (UID: "eb2b858b-eb53-4268-a0e4-e0dde762ae77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:47:13.468827 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:13.468806 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014d29cf-4ab3-4600-947b-a6b67306edb2-kube-api-access-t9hx5" (OuterVolumeSpecName: "kube-api-access-t9hx5") pod "014d29cf-4ab3-4600-947b-a6b67306edb2" (UID: "014d29cf-4ab3-4600-947b-a6b67306edb2"). InnerVolumeSpecName "kube-api-access-t9hx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:47:13.468923 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:13.468845 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb2b858b-eb53-4268-a0e4-e0dde762ae77-kube-api-access-47dpx" (OuterVolumeSpecName: "kube-api-access-47dpx") pod "eb2b858b-eb53-4268-a0e4-e0dde762ae77" (UID: "eb2b858b-eb53-4268-a0e4-e0dde762ae77"). InnerVolumeSpecName "kube-api-access-47dpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:47:13.568455 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:13.568433 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9hx5\" (UniqueName: \"kubernetes.io/projected/014d29cf-4ab3-4600-947b-a6b67306edb2-kube-api-access-t9hx5\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:47:13.568455 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:13.568455 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb2b858b-eb53-4268-a0e4-e0dde762ae77-plugins-config-volume\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:47:13.568585 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:13.568465 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47dpx\" (UniqueName: \"kubernetes.io/projected/eb2b858b-eb53-4268-a0e4-e0dde762ae77-kube-api-access-47dpx\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:47:13.568585 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:13.568475 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/014d29cf-4ab3-4600-947b-a6b67306edb2-plugins-config-volume\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:47:14.119187 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:14.119149 2576 generic.go:356] "Generic (PLEG): container finished" podID="014d29cf-4ab3-4600-947b-a6b67306edb2" containerID="b32646351a601b833fd5f28be3dce3b75e3d7c0c46e55967da3382ed86809f4e" exitCode=0 Sep 06 12:47:14.119585 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:14.119200 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3feb92-6ddb5878b5-c6qsd" Sep 06 12:47:14.119585 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:14.119236 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3feb92-6ddb5878b5-c6qsd" event={"ID":"014d29cf-4ab3-4600-947b-a6b67306edb2","Type":"ContainerDied","Data":"b32646351a601b833fd5f28be3dce3b75e3d7c0c46e55967da3382ed86809f4e"} Sep 06 12:47:14.119585 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:14.119306 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3feb92-6ddb5878b5-c6qsd" event={"ID":"014d29cf-4ab3-4600-947b-a6b67306edb2","Type":"ContainerDied","Data":"a399f6cd586724181ce9b82600211e69069004fbedf7b94da20dc340144f8df7"} Sep 06 12:47:14.119585 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:14.119336 2576 scope.go:117] "RemoveContainer" containerID="b32646351a601b833fd5f28be3dce3b75e3d7c0c46e55967da3382ed86809f4e" Sep 06 12:47:14.121009 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:14.120983 2576 generic.go:356] "Generic (PLEG): container finished" podID="eb2b858b-eb53-4268-a0e4-e0dde762ae77" containerID="74a82cb9d3ce080972485e02e43ba87300e3dbe0cd7a8c3dedf63f2c870949cd" exitCode=0 Sep 06 12:47:14.121103 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:14.121025 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3feb92-64d8bcfbc8-hx7mq" event={"ID":"eb2b858b-eb53-4268-a0e4-e0dde762ae77","Type":"ContainerDied","Data":"74a82cb9d3ce080972485e02e43ba87300e3dbe0cd7a8c3dedf63f2c870949cd"} Sep 06 12:47:14.121103 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:14.121038 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3feb92-64d8bcfbc8-hx7mq" Sep 06 12:47:14.121103 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:14.121069 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3feb92-64d8bcfbc8-hx7mq" event={"ID":"eb2b858b-eb53-4268-a0e4-e0dde762ae77","Type":"ContainerDied","Data":"ad06816183e671d18b2f0e839dcdd67d20d78cda726f389dc1c366a5a6a18f92"} Sep 06 12:47:14.130581 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:14.130561 2576 scope.go:117] "RemoveContainer" containerID="b32646351a601b833fd5f28be3dce3b75e3d7c0c46e55967da3382ed86809f4e" Sep 06 12:47:14.130942 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:47:14.130921 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b32646351a601b833fd5f28be3dce3b75e3d7c0c46e55967da3382ed86809f4e\": container with ID starting with b32646351a601b833fd5f28be3dce3b75e3d7c0c46e55967da3382ed86809f4e not found: ID does not exist" containerID="b32646351a601b833fd5f28be3dce3b75e3d7c0c46e55967da3382ed86809f4e" Sep 06 12:47:14.131016 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:14.130949 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b32646351a601b833fd5f28be3dce3b75e3d7c0c46e55967da3382ed86809f4e"} err="failed to get container status \"b32646351a601b833fd5f28be3dce3b75e3d7c0c46e55967da3382ed86809f4e\": rpc error: code = NotFound desc = could not find container \"b32646351a601b833fd5f28be3dce3b75e3d7c0c46e55967da3382ed86809f4e\": container with ID starting with b32646351a601b833fd5f28be3dce3b75e3d7c0c46e55967da3382ed86809f4e not found: ID does not exist" Sep 06 12:47:14.131016 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:14.130967 2576 scope.go:117] "RemoveContainer" containerID="74a82cb9d3ce080972485e02e43ba87300e3dbe0cd7a8c3dedf63f2c870949cd" Sep 06 12:47:14.139925 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:14.139903 2576 scope.go:117] "RemoveContainer" containerID="74a82cb9d3ce080972485e02e43ba87300e3dbe0cd7a8c3dedf63f2c870949cd" Sep 06 12:47:14.140263 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:47:14.140238 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74a82cb9d3ce080972485e02e43ba87300e3dbe0cd7a8c3dedf63f2c870949cd\": container with ID starting with 74a82cb9d3ce080972485e02e43ba87300e3dbe0cd7a8c3dedf63f2c870949cd not found: ID does not exist" containerID="74a82cb9d3ce080972485e02e43ba87300e3dbe0cd7a8c3dedf63f2c870949cd" Sep 06 12:47:14.140328 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:14.140274 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74a82cb9d3ce080972485e02e43ba87300e3dbe0cd7a8c3dedf63f2c870949cd"} err="failed to get container status \"74a82cb9d3ce080972485e02e43ba87300e3dbe0cd7a8c3dedf63f2c870949cd\": rpc error: code = NotFound desc = could not find container \"74a82cb9d3ce080972485e02e43ba87300e3dbe0cd7a8c3dedf63f2c870949cd\": container with ID starting with 74a82cb9d3ce080972485e02e43ba87300e3dbe0cd7a8c3dedf63f2c870949cd not found: ID does not exist" Sep 06 12:47:14.145873 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:14.145852 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3feb92-6ddb5878b5-c6qsd"] Sep 06 12:47:14.149807 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:14.149779 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3feb92-6ddb5878b5-c6qsd"] Sep 06 12:47:14.159138 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:14.159116 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3feb92-64d8bcfbc8-hx7mq"] Sep 06 12:47:14.161387 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:14.161362 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3feb92-64d8bcfbc8-hx7mq"] Sep 06 12:47:14.653542 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:14.653511 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="014d29cf-4ab3-4600-947b-a6b67306edb2" path="/var/lib/kubelet/pods/014d29cf-4ab3-4600-947b-a6b67306edb2/volumes" Sep 06 12:47:14.653914 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:14.653896 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb2b858b-eb53-4268-a0e4-e0dde762ae77" path="/var/lib/kubelet/pods/eb2b858b-eb53-4268-a0e4-e0dde762ae77/volumes" Sep 06 12:47:30.712459 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:30.712426 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e6515e-29811645-rf8nv"] Sep 06 12:47:30.715180 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:30.715029 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e6515e-29811645-rf8nv"] Sep 06 12:47:30.733156 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:30.733121 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e6515e-557dcd8dcf-xzsdw"] Sep 06 12:47:30.733377 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:30.733351 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e6515e-557dcd8dcf-xzsdw" podUID="84081fd7-e184-44e2-9e76-df6ec0014650" containerName="payload-processing" containerID="cri-o://6fdf0e5d29914c65637f9801dcfc5b279c7886b7c285290c556fd482d91887e8" gracePeriod=30 Sep 06 12:47:30.977643 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:30.977619 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e6515e-557dcd8dcf-xzsdw" Sep 06 12:47:31.013958 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:31.013911 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84081fd7-e184-44e2-9e76-df6ec0014650-plugins-config-volume\") pod \"84081fd7-e184-44e2-9e76-df6ec0014650\" (UID: \"84081fd7-e184-44e2-9e76-df6ec0014650\") " Sep 06 12:47:31.013958 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:31.013949 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7k6c\" (UniqueName: \"kubernetes.io/projected/84081fd7-e184-44e2-9e76-df6ec0014650-kube-api-access-k7k6c\") pod \"84081fd7-e184-44e2-9e76-df6ec0014650\" (UID: \"84081fd7-e184-44e2-9e76-df6ec0014650\") " Sep 06 12:47:31.014661 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:31.014636 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84081fd7-e184-44e2-9e76-df6ec0014650-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "84081fd7-e184-44e2-9e76-df6ec0014650" (UID: "84081fd7-e184-44e2-9e76-df6ec0014650"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:47:31.016558 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:31.016535 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84081fd7-e184-44e2-9e76-df6ec0014650-kube-api-access-k7k6c" (OuterVolumeSpecName: "kube-api-access-k7k6c") pod "84081fd7-e184-44e2-9e76-df6ec0014650" (UID: "84081fd7-e184-44e2-9e76-df6ec0014650"). InnerVolumeSpecName "kube-api-access-k7k6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:47:31.115902 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:31.115878 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84081fd7-e184-44e2-9e76-df6ec0014650-plugins-config-volume\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:47:31.115902 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:31.115902 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7k6c\" (UniqueName: \"kubernetes.io/projected/84081fd7-e184-44e2-9e76-df6ec0014650-kube-api-access-k7k6c\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:47:31.218162 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:31.218077 2576 generic.go:356] "Generic (PLEG): container finished" podID="84081fd7-e184-44e2-9e76-df6ec0014650" containerID="6fdf0e5d29914c65637f9801dcfc5b279c7886b7c285290c556fd482d91887e8" exitCode=0 Sep 06 12:47:31.218162 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:31.218127 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e6515e-557dcd8dcf-xzsdw" Sep 06 12:47:31.218340 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:31.218174 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e6515e-557dcd8dcf-xzsdw" event={"ID":"84081fd7-e184-44e2-9e76-df6ec0014650","Type":"ContainerDied","Data":"6fdf0e5d29914c65637f9801dcfc5b279c7886b7c285290c556fd482d91887e8"} Sep 06 12:47:31.218340 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:31.218232 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e6515e-557dcd8dcf-xzsdw" event={"ID":"84081fd7-e184-44e2-9e76-df6ec0014650","Type":"ContainerDied","Data":"398e0208d7609ccfeeeb858aa6eb5d266b6ade717b54be6f55bc70c31f80856c"} Sep 06 12:47:31.218340 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:31.218263 2576 scope.go:117] "RemoveContainer" containerID="6fdf0e5d29914c65637f9801dcfc5b279c7886b7c285290c556fd482d91887e8" Sep 06 12:47:31.227608 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:31.227585 2576 scope.go:117] "RemoveContainer" containerID="6fdf0e5d29914c65637f9801dcfc5b279c7886b7c285290c556fd482d91887e8" Sep 06 12:47:31.227944 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:47:31.227917 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fdf0e5d29914c65637f9801dcfc5b279c7886b7c285290c556fd482d91887e8\": container with ID starting with 6fdf0e5d29914c65637f9801dcfc5b279c7886b7c285290c556fd482d91887e8 not found: ID does not exist" containerID="6fdf0e5d29914c65637f9801dcfc5b279c7886b7c285290c556fd482d91887e8" Sep 06 12:47:31.228017 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:31.227950 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fdf0e5d29914c65637f9801dcfc5b279c7886b7c285290c556fd482d91887e8"} err="failed to get container status \"6fdf0e5d29914c65637f9801dcfc5b279c7886b7c285290c556fd482d91887e8\": rpc error: code = NotFound desc = could not find container \"6fdf0e5d29914c65637f9801dcfc5b279c7886b7c285290c556fd482d91887e8\": container with ID starting with 6fdf0e5d29914c65637f9801dcfc5b279c7886b7c285290c556fd482d91887e8 not found: ID does not exist" Sep 06 12:47:31.241041 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:31.241018 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e6515e-557dcd8dcf-xzsdw"] Sep 06 12:47:31.244626 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:31.244607 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e6515e-557dcd8dcf-xzsdw"] Sep 06 12:47:32.654306 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:32.654260 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84081fd7-e184-44e2-9e76-df6ec0014650" path="/var/lib/kubelet/pods/84081fd7-e184-44e2-9e76-df6ec0014650/volumes" Sep 06 12:47:32.654705 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:32.654555 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf2508bc-b378-4bd4-8306-36ed5bcdee3d" path="/var/lib/kubelet/pods/cf2508bc-b378-4bd4-8306-36ed5bcdee3d/volumes" Sep 06 12:47:39.892857 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:39.892827 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-679b9dbb-8z9qh"] Sep 06 12:47:39.893358 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:39.893048 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-679b9dbb-8z9qh" podUID="199c3984-3125-4b84-b273-58a686af4adc" containerName="manager" containerID="cri-o://01a0ba9cbd53119175d77ef2e1b914d71b08cbe6a71eeb89418faff18228c8e2" gracePeriod=30 Sep 06 12:47:40.146804 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:40.146729 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-679b9dbb-8z9qh" Sep 06 12:47:40.190015 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:40.189981 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/199c3984-3125-4b84-b273-58a686af4adc-webhook-cert\") pod \"199c3984-3125-4b84-b273-58a686af4adc\" (UID: \"199c3984-3125-4b84-b273-58a686af4adc\") " Sep 06 12:47:40.190167 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:40.190041 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/199c3984-3125-4b84-b273-58a686af4adc-metrics-certs\") pod \"199c3984-3125-4b84-b273-58a686af4adc\" (UID: \"199c3984-3125-4b84-b273-58a686af4adc\") " Sep 06 12:47:40.190234 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:40.190182 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jswb8\" (UniqueName: \"kubernetes.io/projected/199c3984-3125-4b84-b273-58a686af4adc-kube-api-access-jswb8\") pod \"199c3984-3125-4b84-b273-58a686af4adc\" (UID: \"199c3984-3125-4b84-b273-58a686af4adc\") " Sep 06 12:47:40.192239 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:40.192212 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/199c3984-3125-4b84-b273-58a686af4adc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "199c3984-3125-4b84-b273-58a686af4adc" (UID: "199c3984-3125-4b84-b273-58a686af4adc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 12:47:40.192728 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:40.192693 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/199c3984-3125-4b84-b273-58a686af4adc-kube-api-access-jswb8" (OuterVolumeSpecName: "kube-api-access-jswb8") pod "199c3984-3125-4b84-b273-58a686af4adc" (UID: "199c3984-3125-4b84-b273-58a686af4adc"). InnerVolumeSpecName "kube-api-access-jswb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:47:40.192847 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:40.192787 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/199c3984-3125-4b84-b273-58a686af4adc-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "199c3984-3125-4b84-b273-58a686af4adc" (UID: "199c3984-3125-4b84-b273-58a686af4adc"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 12:47:40.269827 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:40.269790 2576 generic.go:356] "Generic (PLEG): container finished" podID="199c3984-3125-4b84-b273-58a686af4adc" containerID="01a0ba9cbd53119175d77ef2e1b914d71b08cbe6a71eeb89418faff18228c8e2" exitCode=0 Sep 06 12:47:40.269945 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:40.269850 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-679b9dbb-8z9qh" Sep 06 12:47:40.269945 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:40.269883 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-679b9dbb-8z9qh" event={"ID":"199c3984-3125-4b84-b273-58a686af4adc","Type":"ContainerDied","Data":"01a0ba9cbd53119175d77ef2e1b914d71b08cbe6a71eeb89418faff18228c8e2"} Sep 06 12:47:40.269945 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:40.269939 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-679b9dbb-8z9qh" event={"ID":"199c3984-3125-4b84-b273-58a686af4adc","Type":"ContainerDied","Data":"6de52cd3d26db508ba07fed5b838ff0ef9dcd927632ee5c632f54782f31a63fe"} Sep 06 12:47:40.270057 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:40.269962 2576 scope.go:117] "RemoveContainer" containerID="01a0ba9cbd53119175d77ef2e1b914d71b08cbe6a71eeb89418faff18228c8e2" Sep 06 12:47:40.279623 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:40.279607 2576 scope.go:117] "RemoveContainer" containerID="01a0ba9cbd53119175d77ef2e1b914d71b08cbe6a71eeb89418faff18228c8e2" Sep 06 12:47:40.279955 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:47:40.279939 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01a0ba9cbd53119175d77ef2e1b914d71b08cbe6a71eeb89418faff18228c8e2\": container with ID starting with 01a0ba9cbd53119175d77ef2e1b914d71b08cbe6a71eeb89418faff18228c8e2 not found: ID does not exist" containerID="01a0ba9cbd53119175d77ef2e1b914d71b08cbe6a71eeb89418faff18228c8e2" Sep 06 12:47:40.280014 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:40.279962 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01a0ba9cbd53119175d77ef2e1b914d71b08cbe6a71eeb89418faff18228c8e2"} err="failed to get container status \"01a0ba9cbd53119175d77ef2e1b914d71b08cbe6a71eeb89418faff18228c8e2\": rpc error: code = NotFound desc = could not find container \"01a0ba9cbd53119175d77ef2e1b914d71b08cbe6a71eeb89418faff18228c8e2\": container with ID starting with 01a0ba9cbd53119175d77ef2e1b914d71b08cbe6a71eeb89418faff18228c8e2 not found: ID does not exist" Sep 06 12:47:40.291617 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:40.291575 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-679b9dbb-8z9qh"] Sep 06 12:47:40.292044 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:40.292022 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jswb8\" (UniqueName: \"kubernetes.io/projected/199c3984-3125-4b84-b273-58a686af4adc-kube-api-access-jswb8\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:47:40.292090 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:40.292054 2576 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/199c3984-3125-4b84-b273-58a686af4adc-webhook-cert\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:47:40.292090 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:40.292069 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/199c3984-3125-4b84-b273-58a686af4adc-metrics-certs\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:47:40.293534 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:40.293515 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-679b9dbb-8z9qh"] Sep 06 12:47:40.660846 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:47:40.656633 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="199c3984-3125-4b84-b273-58a686af4adc" path="/var/lib/kubelet/pods/199c3984-3125-4b84-b273-58a686af4adc/volumes" Sep 06 12:48:04.737561 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:04.737537 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98vz4_4103c09f-dd43-419d-92e5-2374af5b5e01/ovn-acl-logging/0.log" Sep 06 12:48:04.745525 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:04.745505 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98vz4_4103c09f-dd43-419d-92e5-2374af5b5e01/ovn-acl-logging/0.log" Sep 06 12:48:41.060637 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.060590 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-679b9dbb-cjzjx"] Sep 06 12:48:41.061298 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.061178 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f0b970c-996d-457e-b26b-66e6b997b591" containerName="cleanup" Sep 06 12:48:41.061298 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.061199 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f0b970c-996d-457e-b26b-66e6b997b591" containerName="cleanup" Sep 06 12:48:41.061298 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.061225 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="199c3984-3125-4b84-b273-58a686af4adc" containerName="manager" Sep 06 12:48:41.061298 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.061233 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="199c3984-3125-4b84-b273-58a686af4adc" containerName="manager" Sep 06 12:48:41.061298 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.061245 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0e5f0f1-b1f6-4f2a-8ffd-c4afb58b211e" containerName="payload-pre-processing" Sep 06 12:48:41.061298 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.061255 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0e5f0f1-b1f6-4f2a-8ffd-c4afb58b211e" containerName="payload-pre-processing" Sep 06 12:48:41.061298 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.061266 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf2508bc-b378-4bd4-8306-36ed5bcdee3d" containerName="cleanup" Sep 06 12:48:41.061298 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.061274 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf2508bc-b378-4bd4-8306-36ed5bcdee3d" containerName="cleanup" Sep 06 12:48:41.061298 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.061286 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84081fd7-e184-44e2-9e76-df6ec0014650" containerName="payload-processing" Sep 06 12:48:41.061298 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.061294 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="84081fd7-e184-44e2-9e76-df6ec0014650" containerName="payload-processing" Sep 06 12:48:41.061637 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.061307 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="591382be-edad-43f5-af3b-1ebc658eb823" containerName="payload-processing" Sep 06 12:48:41.061637 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.061317 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="591382be-edad-43f5-af3b-1ebc658eb823" containerName="payload-processing" Sep 06 12:48:41.061637 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.061327 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb2b858b-eb53-4268-a0e4-e0dde762ae77" containerName="payload-pre-processing" Sep 06 12:48:41.061637 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.061336 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb2b858b-eb53-4268-a0e4-e0dde762ae77" containerName="payload-pre-processing" Sep 06 12:48:41.061637 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.061344 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b533bd8-7770-4a73-9c49-a768f1273b2b" containerName="maas-api" Sep 06 12:48:41.061637 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.061352 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b533bd8-7770-4a73-9c49-a768f1273b2b" containerName="maas-api" Sep 06 12:48:41.061637 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.061365 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="014d29cf-4ab3-4600-947b-a6b67306edb2" containerName="payload-processing" Sep 06 12:48:41.061637 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.061373 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="014d29cf-4ab3-4600-947b-a6b67306edb2" containerName="payload-processing" Sep 06 12:48:41.061637 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.061390 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1aee8d14-7567-41a1-aed4-6469d5e3782d" containerName="payload-processing" Sep 06 12:48:41.061637 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.061401 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1aee8d14-7567-41a1-aed4-6469d5e3782d" containerName="payload-processing" Sep 06 12:48:41.061637 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.061411 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ee118fe-a2e9-4753-9bfc-6d2d8f0da070" containerName="payload-pre-processing" Sep 06 12:48:41.061637 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.061419 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ee118fe-a2e9-4753-9bfc-6d2d8f0da070" containerName="payload-pre-processing" Sep 06 12:48:41.061637 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.061511 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf2508bc-b378-4bd4-8306-36ed5bcdee3d" containerName="cleanup" Sep 06 12:48:41.061637 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.061520 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ee118fe-a2e9-4753-9bfc-6d2d8f0da070" containerName="payload-pre-processing" Sep 06 12:48:41.061637 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.061528 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb2b858b-eb53-4268-a0e4-e0dde762ae77" containerName="payload-pre-processing" Sep 06 12:48:41.061637 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.061536 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0e5f0f1-b1f6-4f2a-8ffd-c4afb58b211e" containerName="payload-pre-processing" Sep 06 12:48:41.061637 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.061543 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1aee8d14-7567-41a1-aed4-6469d5e3782d" containerName="payload-processing" Sep 06 12:48:41.061637 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.061550 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="84081fd7-e184-44e2-9e76-df6ec0014650" containerName="payload-processing" Sep 06 12:48:41.061637 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.061556 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="199c3984-3125-4b84-b273-58a686af4adc" containerName="manager" Sep 06 12:48:41.061637 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.061562 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f0b970c-996d-457e-b26b-66e6b997b591" containerName="cleanup" Sep 06 12:48:41.061637 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.061567 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="591382be-edad-43f5-af3b-1ebc658eb823" containerName="payload-processing" Sep 06 12:48:41.061637 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.061572 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b533bd8-7770-4a73-9c49-a768f1273b2b" containerName="maas-api" Sep 06 12:48:41.061637 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.061579 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="014d29cf-4ab3-4600-947b-a6b67306edb2" containerName="payload-processing" Sep 06 12:48:41.064692 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.064672 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-679b9dbb-cjzjx" Sep 06 12:48:41.068549 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.068512 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 06 12:48:41.068549 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.068518 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-s9zss\"" Sep 06 12:48:41.068809 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.068554 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 06 12:48:41.068809 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.068636 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 06 12:48:41.080793 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.080716 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-679b9dbb-cjzjx"] Sep 06 12:48:41.089592 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.089560 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/50a10192-8718-49f3-8506-b5cfcc5b4b13-webhook-cert\") pod \"maas-controller-679b9dbb-cjzjx\" (UID: \"50a10192-8718-49f3-8506-b5cfcc5b4b13\") " pod="opendatahub/maas-controller-679b9dbb-cjzjx" Sep 06 12:48:41.089720 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.089647 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50a10192-8718-49f3-8506-b5cfcc5b4b13-metrics-certs\") pod \"maas-controller-679b9dbb-cjzjx\" (UID: \"50a10192-8718-49f3-8506-b5cfcc5b4b13\") " pod="opendatahub/maas-controller-679b9dbb-cjzjx" Sep 06 12:48:41.089720 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.089706 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqhfg\" (UniqueName: \"kubernetes.io/projected/50a10192-8718-49f3-8506-b5cfcc5b4b13-kube-api-access-tqhfg\") pod \"maas-controller-679b9dbb-cjzjx\" (UID: \"50a10192-8718-49f3-8506-b5cfcc5b4b13\") " pod="opendatahub/maas-controller-679b9dbb-cjzjx" Sep 06 12:48:41.190463 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.190435 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqhfg\" (UniqueName: \"kubernetes.io/projected/50a10192-8718-49f3-8506-b5cfcc5b4b13-kube-api-access-tqhfg\") pod \"maas-controller-679b9dbb-cjzjx\" (UID: \"50a10192-8718-49f3-8506-b5cfcc5b4b13\") " pod="opendatahub/maas-controller-679b9dbb-cjzjx" Sep 06 12:48:41.190578 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.190487 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/50a10192-8718-49f3-8506-b5cfcc5b4b13-webhook-cert\") pod \"maas-controller-679b9dbb-cjzjx\" (UID: \"50a10192-8718-49f3-8506-b5cfcc5b4b13\") " pod="opendatahub/maas-controller-679b9dbb-cjzjx" Sep 06 12:48:41.190578 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.190527 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50a10192-8718-49f3-8506-b5cfcc5b4b13-metrics-certs\") pod \"maas-controller-679b9dbb-cjzjx\" (UID: \"50a10192-8718-49f3-8506-b5cfcc5b4b13\") " pod="opendatahub/maas-controller-679b9dbb-cjzjx" Sep 06 12:48:41.193199 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.193172 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50a10192-8718-49f3-8506-b5cfcc5b4b13-metrics-certs\") pod \"maas-controller-679b9dbb-cjzjx\" (UID: \"50a10192-8718-49f3-8506-b5cfcc5b4b13\") " pod="opendatahub/maas-controller-679b9dbb-cjzjx" Sep 06 12:48:41.193386 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.193364 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/50a10192-8718-49f3-8506-b5cfcc5b4b13-webhook-cert\") pod \"maas-controller-679b9dbb-cjzjx\" (UID: \"50a10192-8718-49f3-8506-b5cfcc5b4b13\") " pod="opendatahub/maas-controller-679b9dbb-cjzjx" Sep 06 12:48:41.197920 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.197902 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqhfg\" (UniqueName: \"kubernetes.io/projected/50a10192-8718-49f3-8506-b5cfcc5b4b13-kube-api-access-tqhfg\") pod \"maas-controller-679b9dbb-cjzjx\" (UID: \"50a10192-8718-49f3-8506-b5cfcc5b4b13\") " pod="opendatahub/maas-controller-679b9dbb-cjzjx" Sep 06 12:48:41.378090 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.378021 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-679b9dbb-cjzjx" Sep 06 12:48:41.649824 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.649795 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-679b9dbb-cjzjx"] Sep 06 12:48:41.652826 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:48:41.652797 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50a10192_8718_49f3_8506_b5cfcc5b4b13.slice/crio-00899fd38f8c6c19f3f4684bf31b5ee1bd75212c506bea48d28ba76f063af0ec WatchSource:0}: Error finding container 00899fd38f8c6c19f3f4684bf31b5ee1bd75212c506bea48d28ba76f063af0ec: Status 404 returned error can't find the container with id 00899fd38f8c6c19f3f4684bf31b5ee1bd75212c506bea48d28ba76f063af0ec Sep 06 12:48:41.654120 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:41.654103 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 06 12:48:42.610378 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:42.610345 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-679b9dbb-cjzjx" event={"ID":"50a10192-8718-49f3-8506-b5cfcc5b4b13","Type":"ContainerStarted","Data":"dacf25b758e1c0478e4595e9e89981e317ed2e564bec30e83e3d6ed1b60673cd"} Sep 06 12:48:42.610790 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:42.610394 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-679b9dbb-cjzjx" event={"ID":"50a10192-8718-49f3-8506-b5cfcc5b4b13","Type":"ContainerStarted","Data":"00899fd38f8c6c19f3f4684bf31b5ee1bd75212c506bea48d28ba76f063af0ec"} Sep 06 12:48:42.610790 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:42.610459 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-679b9dbb-cjzjx" Sep 06 12:48:42.626532 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:42.626476 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-679b9dbb-cjzjx" podStartSLOduration=1.148956061 podStartE2EDuration="1.626462114s" totalImagesPullingTime="477.506053ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:48:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:48:42.625519519 +0000 UTC m=+1838.481277426" watchObservedRunningTime="2026-09-06 12:48:42.626462114 +0000 UTC m=+1838.482220013" Sep 06 12:48:53.618522 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:48:53.618489 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-679b9dbb-cjzjx" Sep 06 12:49:02.893996 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:02.893948 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-679b9dbb-cjzjx"] Sep 06 12:49:02.894592 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:02.894194 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-679b9dbb-cjzjx" podUID="50a10192-8718-49f3-8506-b5cfcc5b4b13" containerName="manager" containerID="cri-o://dacf25b758e1c0478e4595e9e89981e317ed2e564bec30e83e3d6ed1b60673cd" gracePeriod=30 Sep 06 12:49:03.139205 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:03.139180 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-679b9dbb-cjzjx" Sep 06 12:49:03.274638 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:03.274556 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/50a10192-8718-49f3-8506-b5cfcc5b4b13-webhook-cert\") pod \"50a10192-8718-49f3-8506-b5cfcc5b4b13\" (UID: \"50a10192-8718-49f3-8506-b5cfcc5b4b13\") " Sep 06 12:49:03.274820 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:03.274662 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqhfg\" (UniqueName: \"kubernetes.io/projected/50a10192-8718-49f3-8506-b5cfcc5b4b13-kube-api-access-tqhfg\") pod \"50a10192-8718-49f3-8506-b5cfcc5b4b13\" (UID: \"50a10192-8718-49f3-8506-b5cfcc5b4b13\") " Sep 06 12:49:03.274820 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:03.274682 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50a10192-8718-49f3-8506-b5cfcc5b4b13-metrics-certs\") pod \"50a10192-8718-49f3-8506-b5cfcc5b4b13\" (UID: \"50a10192-8718-49f3-8506-b5cfcc5b4b13\") " Sep 06 12:49:03.276685 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:03.276657 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50a10192-8718-49f3-8506-b5cfcc5b4b13-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "50a10192-8718-49f3-8506-b5cfcc5b4b13" (UID: "50a10192-8718-49f3-8506-b5cfcc5b4b13"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 12:49:03.276891 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:03.276873 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50a10192-8718-49f3-8506-b5cfcc5b4b13-kube-api-access-tqhfg" (OuterVolumeSpecName: "kube-api-access-tqhfg") pod "50a10192-8718-49f3-8506-b5cfcc5b4b13" (UID: "50a10192-8718-49f3-8506-b5cfcc5b4b13"). InnerVolumeSpecName "kube-api-access-tqhfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:49:03.276992 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:03.276969 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50a10192-8718-49f3-8506-b5cfcc5b4b13-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "50a10192-8718-49f3-8506-b5cfcc5b4b13" (UID: "50a10192-8718-49f3-8506-b5cfcc5b4b13"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 12:49:03.375604 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:03.375581 2576 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/50a10192-8718-49f3-8506-b5cfcc5b4b13-webhook-cert\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:49:03.375604 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:03.375604 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqhfg\" (UniqueName: \"kubernetes.io/projected/50a10192-8718-49f3-8506-b5cfcc5b4b13-kube-api-access-tqhfg\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:49:03.375742 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:03.375613 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50a10192-8718-49f3-8506-b5cfcc5b4b13-metrics-certs\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:49:03.719126 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:03.719086 2576 generic.go:356] "Generic (PLEG): container finished" podID="50a10192-8718-49f3-8506-b5cfcc5b4b13" containerID="dacf25b758e1c0478e4595e9e89981e317ed2e564bec30e83e3d6ed1b60673cd" exitCode=0 Sep 06 12:49:03.719303 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:03.719134 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-679b9dbb-cjzjx" Sep 06 12:49:03.719303 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:03.719166 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-679b9dbb-cjzjx" event={"ID":"50a10192-8718-49f3-8506-b5cfcc5b4b13","Type":"ContainerDied","Data":"dacf25b758e1c0478e4595e9e89981e317ed2e564bec30e83e3d6ed1b60673cd"} Sep 06 12:49:03.719303 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:03.719215 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-679b9dbb-cjzjx" event={"ID":"50a10192-8718-49f3-8506-b5cfcc5b4b13","Type":"ContainerDied","Data":"00899fd38f8c6c19f3f4684bf31b5ee1bd75212c506bea48d28ba76f063af0ec"} Sep 06 12:49:03.719303 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:03.719237 2576 scope.go:117] "RemoveContainer" containerID="dacf25b758e1c0478e4595e9e89981e317ed2e564bec30e83e3d6ed1b60673cd" Sep 06 12:49:03.728500 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:03.728481 2576 scope.go:117] "RemoveContainer" containerID="dacf25b758e1c0478e4595e9e89981e317ed2e564bec30e83e3d6ed1b60673cd" Sep 06 12:49:03.728823 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:49:03.728804 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dacf25b758e1c0478e4595e9e89981e317ed2e564bec30e83e3d6ed1b60673cd\": container with ID starting with dacf25b758e1c0478e4595e9e89981e317ed2e564bec30e83e3d6ed1b60673cd not found: ID does not exist" containerID="dacf25b758e1c0478e4595e9e89981e317ed2e564bec30e83e3d6ed1b60673cd" Sep 06 12:49:03.728873 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:03.728833 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dacf25b758e1c0478e4595e9e89981e317ed2e564bec30e83e3d6ed1b60673cd"} err="failed to get container status \"dacf25b758e1c0478e4595e9e89981e317ed2e564bec30e83e3d6ed1b60673cd\": rpc error: code = NotFound desc = could not find container \"dacf25b758e1c0478e4595e9e89981e317ed2e564bec30e83e3d6ed1b60673cd\": container with ID starting with dacf25b758e1c0478e4595e9e89981e317ed2e564bec30e83e3d6ed1b60673cd not found: ID does not exist" Sep 06 12:49:03.739687 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:03.739664 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-679b9dbb-cjzjx"] Sep 06 12:49:03.741490 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:03.741472 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-679b9dbb-cjzjx"] Sep 06 12:49:04.493238 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:04.493210 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-679b9dbb-sf7xf"] Sep 06 12:49:04.493611 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:04.493598 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50a10192-8718-49f3-8506-b5cfcc5b4b13" containerName="manager" Sep 06 12:49:04.493659 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:04.493612 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="50a10192-8718-49f3-8506-b5cfcc5b4b13" containerName="manager" Sep 06 12:49:04.493695 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:04.493687 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="50a10192-8718-49f3-8506-b5cfcc5b4b13" containerName="manager" Sep 06 12:49:04.498341 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:04.498311 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-679b9dbb-sf7xf" Sep 06 12:49:04.501367 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:04.501342 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 06 12:49:04.501497 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:04.501394 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 06 12:49:04.501497 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:04.501467 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-s9zss\"" Sep 06 12:49:04.501638 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:04.501494 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 06 12:49:04.506991 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:04.506971 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-679b9dbb-sf7xf"] Sep 06 12:49:04.585914 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:04.585892 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb2nr\" (UniqueName: \"kubernetes.io/projected/44dac02b-bb13-42bf-9c70-6c47e3d6a2c4-kube-api-access-jb2nr\") pod \"maas-controller-679b9dbb-sf7xf\" (UID: \"44dac02b-bb13-42bf-9c70-6c47e3d6a2c4\") " pod="opendatahub/maas-controller-679b9dbb-sf7xf" Sep 06 12:49:04.586039 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:04.585937 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44dac02b-bb13-42bf-9c70-6c47e3d6a2c4-webhook-cert\") pod \"maas-controller-679b9dbb-sf7xf\" (UID: \"44dac02b-bb13-42bf-9c70-6c47e3d6a2c4\") " pod="opendatahub/maas-controller-679b9dbb-sf7xf" Sep 06 12:49:04.586039 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:04.585954 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44dac02b-bb13-42bf-9c70-6c47e3d6a2c4-metrics-certs\") pod \"maas-controller-679b9dbb-sf7xf\" (UID: \"44dac02b-bb13-42bf-9c70-6c47e3d6a2c4\") " pod="opendatahub/maas-controller-679b9dbb-sf7xf" Sep 06 12:49:04.653312 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:04.653286 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50a10192-8718-49f3-8506-b5cfcc5b4b13" path="/var/lib/kubelet/pods/50a10192-8718-49f3-8506-b5cfcc5b4b13/volumes" Sep 06 12:49:04.687256 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:04.687235 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44dac02b-bb13-42bf-9c70-6c47e3d6a2c4-webhook-cert\") pod \"maas-controller-679b9dbb-sf7xf\" (UID: \"44dac02b-bb13-42bf-9c70-6c47e3d6a2c4\") " pod="opendatahub/maas-controller-679b9dbb-sf7xf" Sep 06 12:49:04.687345 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:04.687264 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44dac02b-bb13-42bf-9c70-6c47e3d6a2c4-metrics-certs\") pod \"maas-controller-679b9dbb-sf7xf\" (UID: \"44dac02b-bb13-42bf-9c70-6c47e3d6a2c4\") " pod="opendatahub/maas-controller-679b9dbb-sf7xf" Sep 06 12:49:04.687345 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:04.687319 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb2nr\" (UniqueName: \"kubernetes.io/projected/44dac02b-bb13-42bf-9c70-6c47e3d6a2c4-kube-api-access-jb2nr\") pod \"maas-controller-679b9dbb-sf7xf\" (UID: \"44dac02b-bb13-42bf-9c70-6c47e3d6a2c4\") " pod="opendatahub/maas-controller-679b9dbb-sf7xf" Sep 06 12:49:04.689588 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:04.689567 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 06 12:49:04.689637 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:04.689626 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 06 12:49:04.695207 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:04.695185 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb2nr\" (UniqueName: \"kubernetes.io/projected/44dac02b-bb13-42bf-9c70-6c47e3d6a2c4-kube-api-access-jb2nr\") pod \"maas-controller-679b9dbb-sf7xf\" (UID: \"44dac02b-bb13-42bf-9c70-6c47e3d6a2c4\") " pod="opendatahub/maas-controller-679b9dbb-sf7xf" Sep 06 12:49:04.700149 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:04.700120 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44dac02b-bb13-42bf-9c70-6c47e3d6a2c4-metrics-certs\") pod \"maas-controller-679b9dbb-sf7xf\" (UID: \"44dac02b-bb13-42bf-9c70-6c47e3d6a2c4\") " pod="opendatahub/maas-controller-679b9dbb-sf7xf" Sep 06 12:49:04.700225 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:04.700160 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44dac02b-bb13-42bf-9c70-6c47e3d6a2c4-webhook-cert\") pod \"maas-controller-679b9dbb-sf7xf\" (UID: \"44dac02b-bb13-42bf-9c70-6c47e3d6a2c4\") " pod="opendatahub/maas-controller-679b9dbb-sf7xf" Sep 06 12:49:04.812471 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:04.812356 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-s9zss\"" Sep 06 12:49:04.824114 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:04.820294 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-679b9dbb-sf7xf" Sep 06 12:49:05.094083 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:05.094057 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-679b9dbb-sf7xf"] Sep 06 12:49:05.096270 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:49:05.096243 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44dac02b_bb13_42bf_9c70_6c47e3d6a2c4.slice/crio-75789a26749d60ddccfbf56eb883afbc4eb15618a38fb88c9f28fcf6cf6b7653 WatchSource:0}: Error finding container 75789a26749d60ddccfbf56eb883afbc4eb15618a38fb88c9f28fcf6cf6b7653: Status 404 returned error can't find the container with id 75789a26749d60ddccfbf56eb883afbc4eb15618a38fb88c9f28fcf6cf6b7653 Sep 06 12:49:05.589182 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:05.589158 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 06 12:49:05.731795 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:05.731745 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-679b9dbb-sf7xf" event={"ID":"44dac02b-bb13-42bf-9c70-6c47e3d6a2c4","Type":"ContainerStarted","Data":"eddbf8e335a04fb0f9576d35cbb3328f5f1e1d179b332f2ad4e969dae4c994df"} Sep 06 12:49:05.731929 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:05.731855 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-679b9dbb-sf7xf" Sep 06 12:49:05.731929 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:05.731874 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-679b9dbb-sf7xf" event={"ID":"44dac02b-bb13-42bf-9c70-6c47e3d6a2c4","Type":"ContainerStarted","Data":"75789a26749d60ddccfbf56eb883afbc4eb15618a38fb88c9f28fcf6cf6b7653"} Sep 06 12:49:05.748079 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:05.748034 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-679b9dbb-sf7xf" podStartSLOduration=1.258460446 podStartE2EDuration="1.748019327s" totalImagesPullingTime="489.558881ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:49:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:49:05.746315162 +0000 UTC m=+1861.602073200" watchObservedRunningTime="2026-09-06 12:49:05.748019327 +0000 UTC m=+1861.603777213" Sep 06 12:49:16.740217 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:49:16.740187 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-679b9dbb-sf7xf" Sep 06 12:50:04.918295 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:50:04.918262 2576 scope.go:117] "RemoveContainer" containerID="f4318170c4c49b01bdbb4d014f08976a595e3d687017e58f780ab2d9f7cd12d1" Sep 06 12:51:04.938195 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:51:04.938165 2576 scope.go:117] "RemoveContainer" containerID="0b7d7fddbe46e305438b275459d74102bfd7f5b40e69e2fe834d125955c06a31" Sep 06 12:53:03.397708 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:03.397676 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7f8ff55747-dz6j6"] Sep 06 12:53:03.402700 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:03.402682 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f8ff55747-dz6j6" Sep 06 12:53:03.417912 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:03.417879 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f8ff55747-dz6j6"] Sep 06 12:53:03.566271 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:03.566240 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40d3a88a-0500-4397-916d-63c72cc3d98c-metrics-certs\") pod \"maas-api-7f8ff55747-dz6j6\" (UID: \"40d3a88a-0500-4397-916d-63c72cc3d98c\") " pod="odh-ai-gateway-infra/maas-api-7f8ff55747-dz6j6" Sep 06 12:53:03.566423 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:03.566287 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40d3a88a-0500-4397-916d-63c72cc3d98c-openshift-service-ca\") pod \"maas-api-7f8ff55747-dz6j6\" (UID: \"40d3a88a-0500-4397-916d-63c72cc3d98c\") " pod="odh-ai-gateway-infra/maas-api-7f8ff55747-dz6j6" Sep 06 12:53:03.566423 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:03.566306 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxlnh\" (UniqueName: \"kubernetes.io/projected/40d3a88a-0500-4397-916d-63c72cc3d98c-kube-api-access-dxlnh\") pod \"maas-api-7f8ff55747-dz6j6\" (UID: \"40d3a88a-0500-4397-916d-63c72cc3d98c\") " pod="odh-ai-gateway-infra/maas-api-7f8ff55747-dz6j6" Sep 06 12:53:03.566423 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:03.566408 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40d3a88a-0500-4397-916d-63c72cc3d98c-maas-api-tls\") pod \"maas-api-7f8ff55747-dz6j6\" (UID: \"40d3a88a-0500-4397-916d-63c72cc3d98c\") " pod="odh-ai-gateway-infra/maas-api-7f8ff55747-dz6j6" Sep 06 12:53:03.666994 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:03.666924 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40d3a88a-0500-4397-916d-63c72cc3d98c-maas-api-tls\") pod \"maas-api-7f8ff55747-dz6j6\" (UID: \"40d3a88a-0500-4397-916d-63c72cc3d98c\") " pod="odh-ai-gateway-infra/maas-api-7f8ff55747-dz6j6" Sep 06 12:53:03.666994 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:03.666974 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40d3a88a-0500-4397-916d-63c72cc3d98c-metrics-certs\") pod \"maas-api-7f8ff55747-dz6j6\" (UID: \"40d3a88a-0500-4397-916d-63c72cc3d98c\") " pod="odh-ai-gateway-infra/maas-api-7f8ff55747-dz6j6" Sep 06 12:53:03.667191 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:03.667013 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40d3a88a-0500-4397-916d-63c72cc3d98c-openshift-service-ca\") pod \"maas-api-7f8ff55747-dz6j6\" (UID: \"40d3a88a-0500-4397-916d-63c72cc3d98c\") " pod="odh-ai-gateway-infra/maas-api-7f8ff55747-dz6j6" Sep 06 12:53:03.667191 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:03.667030 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxlnh\" (UniqueName: \"kubernetes.io/projected/40d3a88a-0500-4397-916d-63c72cc3d98c-kube-api-access-dxlnh\") pod \"maas-api-7f8ff55747-dz6j6\" (UID: \"40d3a88a-0500-4397-916d-63c72cc3d98c\") " pod="odh-ai-gateway-infra/maas-api-7f8ff55747-dz6j6" Sep 06 12:53:03.667653 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:03.667618 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40d3a88a-0500-4397-916d-63c72cc3d98c-openshift-service-ca\") pod \"maas-api-7f8ff55747-dz6j6\" (UID: \"40d3a88a-0500-4397-916d-63c72cc3d98c\") " pod="odh-ai-gateway-infra/maas-api-7f8ff55747-dz6j6" Sep 06 12:53:03.669529 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:03.669508 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40d3a88a-0500-4397-916d-63c72cc3d98c-maas-api-tls\") pod \"maas-api-7f8ff55747-dz6j6\" (UID: \"40d3a88a-0500-4397-916d-63c72cc3d98c\") " pod="odh-ai-gateway-infra/maas-api-7f8ff55747-dz6j6" Sep 06 12:53:03.669638 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:03.669568 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40d3a88a-0500-4397-916d-63c72cc3d98c-metrics-certs\") pod \"maas-api-7f8ff55747-dz6j6\" (UID: \"40d3a88a-0500-4397-916d-63c72cc3d98c\") " pod="odh-ai-gateway-infra/maas-api-7f8ff55747-dz6j6" Sep 06 12:53:03.675290 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:03.675269 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxlnh\" (UniqueName: \"kubernetes.io/projected/40d3a88a-0500-4397-916d-63c72cc3d98c-kube-api-access-dxlnh\") pod \"maas-api-7f8ff55747-dz6j6\" (UID: \"40d3a88a-0500-4397-916d-63c72cc3d98c\") " pod="odh-ai-gateway-infra/maas-api-7f8ff55747-dz6j6" Sep 06 12:53:03.713713 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:03.713694 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f8ff55747-dz6j6" Sep 06 12:53:03.983320 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:03.983292 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f8ff55747-dz6j6"] Sep 06 12:53:03.985181 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:53:03.985153 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d3a88a_0500_4397_916d_63c72cc3d98c.slice/crio-7c63f2746d57142f5dcc63c5be38cd69f7200ec927423eb36c6631d94a90b9b9 WatchSource:0}: Error finding container 7c63f2746d57142f5dcc63c5be38cd69f7200ec927423eb36c6631d94a90b9b9: Status 404 returned error can't find the container with id 7c63f2746d57142f5dcc63c5be38cd69f7200ec927423eb36c6631d94a90b9b9 Sep 06 12:53:04.771078 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:04.771044 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98vz4_4103c09f-dd43-419d-92e5-2374af5b5e01/ovn-acl-logging/0.log" Sep 06 12:53:04.781281 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:04.781246 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98vz4_4103c09f-dd43-419d-92e5-2374af5b5e01/ovn-acl-logging/0.log" Sep 06 12:53:04.995162 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:04.995133 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f8ff55747-dz6j6" event={"ID":"40d3a88a-0500-4397-916d-63c72cc3d98c","Type":"ContainerStarted","Data":"faf04e1fdf6e6be48789887a383e64395b7305fa4b685248c6ee46b0b9443f41"} Sep 06 12:53:04.995325 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:04.995184 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f8ff55747-dz6j6" event={"ID":"40d3a88a-0500-4397-916d-63c72cc3d98c","Type":"ContainerStarted","Data":"7c63f2746d57142f5dcc63c5be38cd69f7200ec927423eb36c6631d94a90b9b9"} Sep 06 12:53:04.995325 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:04.995306 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7f8ff55747-dz6j6" Sep 06 12:53:05.012698 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:05.012654 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7f8ff55747-dz6j6" podStartSLOduration=1.695138238 podStartE2EDuration="2.012640645s" totalImagesPullingTime="317.502407ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 12:53:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:53:05.011329345 +0000 UTC m=+2100.867087259" watchObservedRunningTime="2026-09-06 12:53:05.012640645 +0000 UTC m=+2100.868398562" Sep 06 12:53:11.006853 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:11.006814 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7f8ff55747-dz6j6" Sep 06 12:53:11.063281 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:11.063238 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b4cdc449f-bdh4x"] Sep 06 12:53:11.063581 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:11.063532 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5b4cdc449f-bdh4x" podUID="86425b02-9d6f-4d6b-b5e4-f5ade4648b78" containerName="maas-api" containerID="cri-o://924665ce853e2e35df52811355c34f5741042da9485041c4f18af17322dd41dc" gracePeriod=30 Sep 06 12:53:12.316400 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:12.316374 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b4cdc449f-bdh4x" Sep 06 12:53:12.440624 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:12.440562 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86425b02-9d6f-4d6b-b5e4-f5ade4648b78-openshift-service-ca\") pod \"86425b02-9d6f-4d6b-b5e4-f5ade4648b78\" (UID: \"86425b02-9d6f-4d6b-b5e4-f5ade4648b78\") " Sep 06 12:53:12.440784 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:12.440641 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j85fr\" (UniqueName: \"kubernetes.io/projected/86425b02-9d6f-4d6b-b5e4-f5ade4648b78-kube-api-access-j85fr\") pod \"86425b02-9d6f-4d6b-b5e4-f5ade4648b78\" (UID: \"86425b02-9d6f-4d6b-b5e4-f5ade4648b78\") " Sep 06 12:53:12.440784 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:12.440672 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86425b02-9d6f-4d6b-b5e4-f5ade4648b78-maas-api-tls\") pod \"86425b02-9d6f-4d6b-b5e4-f5ade4648b78\" (UID: \"86425b02-9d6f-4d6b-b5e4-f5ade4648b78\") " Sep 06 12:53:12.440784 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:12.440695 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86425b02-9d6f-4d6b-b5e4-f5ade4648b78-metrics-certs\") pod \"86425b02-9d6f-4d6b-b5e4-f5ade4648b78\" (UID: \"86425b02-9d6f-4d6b-b5e4-f5ade4648b78\") " Sep 06 12:53:12.441019 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:12.440987 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86425b02-9d6f-4d6b-b5e4-f5ade4648b78-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "86425b02-9d6f-4d6b-b5e4-f5ade4648b78" (UID: "86425b02-9d6f-4d6b-b5e4-f5ade4648b78"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 12:53:12.442914 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:12.442883 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86425b02-9d6f-4d6b-b5e4-f5ade4648b78-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "86425b02-9d6f-4d6b-b5e4-f5ade4648b78" (UID: "86425b02-9d6f-4d6b-b5e4-f5ade4648b78"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 12:53:12.443015 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:12.442991 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86425b02-9d6f-4d6b-b5e4-f5ade4648b78-kube-api-access-j85fr" (OuterVolumeSpecName: "kube-api-access-j85fr") pod "86425b02-9d6f-4d6b-b5e4-f5ade4648b78" (UID: "86425b02-9d6f-4d6b-b5e4-f5ade4648b78"). InnerVolumeSpecName "kube-api-access-j85fr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 12:53:12.443133 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:12.443117 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86425b02-9d6f-4d6b-b5e4-f5ade4648b78-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "86425b02-9d6f-4d6b-b5e4-f5ade4648b78" (UID: "86425b02-9d6f-4d6b-b5e4-f5ade4648b78"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 12:53:12.542079 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:12.542050 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j85fr\" (UniqueName: \"kubernetes.io/projected/86425b02-9d6f-4d6b-b5e4-f5ade4648b78-kube-api-access-j85fr\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:53:12.542079 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:12.542077 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86425b02-9d6f-4d6b-b5e4-f5ade4648b78-maas-api-tls\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:53:12.542258 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:12.542092 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86425b02-9d6f-4d6b-b5e4-f5ade4648b78-metrics-certs\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:53:12.542258 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:12.542104 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86425b02-9d6f-4d6b-b5e4-f5ade4648b78-openshift-service-ca\") on node \"ip-10-0-132-159.ec2.internal\" DevicePath \"\"" Sep 06 12:53:13.039303 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:13.039264 2576 generic.go:356] "Generic (PLEG): container finished" podID="86425b02-9d6f-4d6b-b5e4-f5ade4648b78" containerID="924665ce853e2e35df52811355c34f5741042da9485041c4f18af17322dd41dc" exitCode=0 Sep 06 12:53:13.039461 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:13.039323 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b4cdc449f-bdh4x" Sep 06 12:53:13.039461 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:13.039333 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b4cdc449f-bdh4x" event={"ID":"86425b02-9d6f-4d6b-b5e4-f5ade4648b78","Type":"ContainerDied","Data":"924665ce853e2e35df52811355c34f5741042da9485041c4f18af17322dd41dc"} Sep 06 12:53:13.039461 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:13.039399 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b4cdc449f-bdh4x" event={"ID":"86425b02-9d6f-4d6b-b5e4-f5ade4648b78","Type":"ContainerDied","Data":"4d5de4915e1f1072e11ed5ef2f21d0f11abc852f2971ac2ef4de9c7dda73ee13"} Sep 06 12:53:13.039461 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:13.039415 2576 scope.go:117] "RemoveContainer" containerID="924665ce853e2e35df52811355c34f5741042da9485041c4f18af17322dd41dc" Sep 06 12:53:13.049088 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:13.049066 2576 scope.go:117] "RemoveContainer" containerID="924665ce853e2e35df52811355c34f5741042da9485041c4f18af17322dd41dc" Sep 06 12:53:13.049383 ip-10-0-132-159 kubenswrapper[2576]: E0906 12:53:13.049362 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"924665ce853e2e35df52811355c34f5741042da9485041c4f18af17322dd41dc\": container with ID starting with 924665ce853e2e35df52811355c34f5741042da9485041c4f18af17322dd41dc not found: ID does not exist" containerID="924665ce853e2e35df52811355c34f5741042da9485041c4f18af17322dd41dc" Sep 06 12:53:13.049460 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:13.049392 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"924665ce853e2e35df52811355c34f5741042da9485041c4f18af17322dd41dc"} err="failed to get container status \"924665ce853e2e35df52811355c34f5741042da9485041c4f18af17322dd41dc\": rpc error: code = NotFound desc = could not find container \"924665ce853e2e35df52811355c34f5741042da9485041c4f18af17322dd41dc\": container with ID starting with 924665ce853e2e35df52811355c34f5741042da9485041c4f18af17322dd41dc not found: ID does not exist" Sep 06 12:53:13.057009 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:13.056984 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b4cdc449f-bdh4x"] Sep 06 12:53:13.060624 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:13.060601 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b4cdc449f-bdh4x"] Sep 06 12:53:14.653426 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:14.653391 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86425b02-9d6f-4d6b-b5e4-f5ade4648b78" path="/var/lib/kubelet/pods/86425b02-9d6f-4d6b-b5e4-f5ade4648b78/volumes" Sep 06 12:53:33.624447 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:33.624418 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-679b9dbb-sf7xf_44dac02b-bb13-42bf-9c70-6c47e3d6a2c4/manager/0.log" Sep 06 12:53:33.853880 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:33.853852 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-zw6hf_4a537ef0-6b94-407f-b443-0dc16b4aa810/manager/0.log" Sep 06 12:53:34.210103 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:34.210078 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7bf7c58dfb-wgl45_ce2e1a8c-9c75-42bb-ae98-04c1e57f5206/copy-manifests/0.log" Sep 06 12:53:34.216814 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:34.216791 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7bf7c58dfb-wgl45_ce2e1a8c-9c75-42bb-ae98-04c1e57f5206/manager/0.log" Sep 06 12:53:34.549196 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:34.549105 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7f8ff55747-dz6j6_40d3a88a-0500-4397-916d-63c72cc3d98c/maas-api/0.log" Sep 06 12:53:34.858993 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:34.858931 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29811645-n24zn_2f0b970c-996d-457e-b26b-66e6b997b591/cleanup/0.log" Sep 06 12:53:34.963998 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:34.963975 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-qhlnq_de9a2051-cb39-412b-ae3e-2a135c639cf2/postgres/0.log" Sep 06 12:53:36.841804 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:36.841769 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-849pv_a4f7e77d-d6df-44ea-b9d3-07fb828050f5/discovery/0.log" Sep 06 12:53:37.294890 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:37.294859 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-t9grp_33750f3b-84a4-4935-a768-e2f5bda50eb3/payload-pre-processing/0.log" Sep 06 12:53:37.415549 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:37.415520 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-jlvzm_2a0a23e6-8a48-4825-b2d7-e6bfd31fbe1f/payload-processing/0.log" Sep 06 12:53:37.867495 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:37.867465 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-zbl82_7ce6657f-df41-4452-8428-1e8b143d9a8b/main/0.log" Sep 06 12:53:38.419247 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:38.419199 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-wptjv_b41e00d7-fdb8-4a5e-850f-e9492a62f085/main/0.log" Sep 06 12:53:46.969516 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:46.969469 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-m6lx6_e095072c-bd72-49b4-8cba-b49c94e40bf9/global-pull-secret-syncer/0.log" Sep 06 12:53:47.055778 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:47.055729 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-l2nhh_12708d6c-10bb-472c-b4a3-e8ece6ef008f/konnectivity-agent/0.log" Sep 06 12:53:47.148111 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:47.148091 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-159.ec2.internal_45bd7c4cff2b3f51ea19564c2fce8332/haproxy/0.log" Sep 06 12:53:53.482218 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:53.482192 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_79460c57-2888-4499-937f-cbc5752f4956/alertmanager/0.log" Sep 06 12:53:53.501096 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:53.501074 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_79460c57-2888-4499-937f-cbc5752f4956/config-reloader/0.log" Sep 06 12:53:53.519399 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:53.519377 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_79460c57-2888-4499-937f-cbc5752f4956/kube-rbac-proxy-web/0.log" Sep 06 12:53:53.539402 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:53.539379 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_79460c57-2888-4499-937f-cbc5752f4956/kube-rbac-proxy/0.log" Sep 06 12:53:53.558239 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:53.558215 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_79460c57-2888-4499-937f-cbc5752f4956/kube-rbac-proxy-metric/0.log" Sep 06 12:53:53.576598 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:53.576580 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_79460c57-2888-4499-937f-cbc5752f4956/prom-label-proxy/0.log" Sep 06 12:53:53.594514 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:53.594497 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_79460c57-2888-4499-937f-cbc5752f4956/init-config-reloader/0.log" Sep 06 12:53:53.660278 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:53.660258 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-bf74d_be7870d2-35ee-43d8-9b02-74ad8ee286b1/kube-state-metrics/0.log" Sep 06 12:53:53.683975 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:53.683954 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-bf74d_be7870d2-35ee-43d8-9b02-74ad8ee286b1/kube-rbac-proxy-main/0.log" Sep 06 12:53:53.705534 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:53.705516 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-bf74d_be7870d2-35ee-43d8-9b02-74ad8ee286b1/kube-rbac-proxy-self/0.log" Sep 06 12:53:53.733599 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:53.733552 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-757d878946-7fxrv_aa177905-aa73-46c3-8f2b-7a2a76c69607/metrics-server/0.log" Sep 06 12:53:53.765638 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:53.765621 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-w9hp5_b88e1bb2-b111-43b7-82a1-a528302c18b9/monitoring-plugin/0.log" Sep 06 12:53:53.878373 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:53.878354 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n825g_529c654b-f937-4604-8271-488f66663e84/node-exporter/0.log" Sep 06 12:53:53.896681 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:53.896661 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n825g_529c654b-f937-4604-8271-488f66663e84/kube-rbac-proxy/0.log" Sep 06 12:53:53.916076 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:53.916060 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n825g_529c654b-f937-4604-8271-488f66663e84/init-textfile/0.log" Sep 06 12:53:54.036778 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:54.036708 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-5ddg8_212e2510-1c27-4fdf-ad5f-51b8b334106a/kube-rbac-proxy-main/0.log" Sep 06 12:53:54.056059 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:54.056036 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-5ddg8_212e2510-1c27-4fdf-ad5f-51b8b334106a/kube-rbac-proxy-self/0.log" Sep 06 12:53:54.080707 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:54.080684 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-5ddg8_212e2510-1c27-4fdf-ad5f-51b8b334106a/openshift-state-metrics/0.log" Sep 06 12:53:54.159645 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:54.159629 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7afd0d05-3e0b-493f-bba1-999be7877cf5/prometheus/0.log" Sep 06 12:53:54.222084 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:54.222067 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7afd0d05-3e0b-493f-bba1-999be7877cf5/config-reloader/0.log" Sep 06 12:53:54.243214 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:54.243196 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7afd0d05-3e0b-493f-bba1-999be7877cf5/thanos-sidecar/0.log" Sep 06 12:53:54.277523 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:54.277499 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7afd0d05-3e0b-493f-bba1-999be7877cf5/kube-rbac-proxy-web/0.log" Sep 06 12:53:54.314445 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:54.314385 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7afd0d05-3e0b-493f-bba1-999be7877cf5/kube-rbac-proxy/0.log" Sep 06 12:53:54.348022 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:54.348006 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7afd0d05-3e0b-493f-bba1-999be7877cf5/kube-rbac-proxy-thanos/0.log" Sep 06 12:53:54.387008 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:54.386990 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7afd0d05-3e0b-493f-bba1-999be7877cf5/init-config-reloader/0.log" Sep 06 12:53:54.420700 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:54.420684 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-xdf8m_e34034de-85ae-4d49-8e58-ea01c00a894d/prometheus-operator/0.log" Sep 06 12:53:54.439325 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:54.439306 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-xdf8m_e34034de-85ae-4d49-8e58-ea01c00a894d/kube-rbac-proxy/0.log" Sep 06 12:53:54.477933 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:54.477913 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-zrzx8_8befde00-706b-42ac-9fa7-1bf421fe4783/prometheus-operator-admission-webhook/0.log" Sep 06 12:53:54.583285 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:54.583234 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8675944c9b-bnsxn_09447812-284c-43c7-b928-f9afdb434f1f/thanos-query/0.log" Sep 06 12:53:54.604618 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:54.604598 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8675944c9b-bnsxn_09447812-284c-43c7-b928-f9afdb434f1f/kube-rbac-proxy-web/0.log" Sep 06 12:53:54.624049 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:54.624027 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8675944c9b-bnsxn_09447812-284c-43c7-b928-f9afdb434f1f/kube-rbac-proxy/0.log" Sep 06 12:53:54.643466 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:54.643449 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8675944c9b-bnsxn_09447812-284c-43c7-b928-f9afdb434f1f/prom-label-proxy/0.log" Sep 06 12:53:54.662694 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:54.662678 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8675944c9b-bnsxn_09447812-284c-43c7-b928-f9afdb434f1f/kube-rbac-proxy-rules/0.log" Sep 06 12:53:54.685477 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:54.685460 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8675944c9b-bnsxn_09447812-284c-43c7-b928-f9afdb434f1f/kube-rbac-proxy-metrics/0.log" Sep 06 12:53:57.133465 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:57.133435 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-79f466df84-6mqf4_1b577512-6c05-41fd-a560-e3c966cec496/console/0.log" Sep 06 12:53:57.278791 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:57.278742 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q6h8m/perf-node-gather-daemonset-kdbkt"] Sep 06 12:53:57.279194 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:57.279181 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86425b02-9d6f-4d6b-b5e4-f5ade4648b78" containerName="maas-api" Sep 06 12:53:57.279239 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:57.279196 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="86425b02-9d6f-4d6b-b5e4-f5ade4648b78" containerName="maas-api" Sep 06 12:53:57.279277 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:57.279257 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="86425b02-9d6f-4d6b-b5e4-f5ade4648b78" containerName="maas-api" Sep 06 12:53:57.282569 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:57.282554 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q6h8m/perf-node-gather-daemonset-kdbkt" Sep 06 12:53:57.284705 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:57.284684 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q6h8m\"/\"kube-root-ca.crt\"" Sep 06 12:53:57.284825 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:57.284695 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-q6h8m\"/\"default-dockercfg-lf5lq\"" Sep 06 12:53:57.285461 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:57.285429 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q6h8m\"/\"openshift-service-ca.crt\"" Sep 06 12:53:57.291885 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:57.291864 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q6h8m/perf-node-gather-daemonset-kdbkt"] Sep 06 12:53:57.405872 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:57.405838 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d140c4db-0abf-4986-8cd1-f245bc09f7ac-lib-modules\") pod \"perf-node-gather-daemonset-kdbkt\" (UID: \"d140c4db-0abf-4986-8cd1-f245bc09f7ac\") " pod="openshift-must-gather-q6h8m/perf-node-gather-daemonset-kdbkt" Sep 06 12:53:57.405997 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:57.405891 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d140c4db-0abf-4986-8cd1-f245bc09f7ac-proc\") pod \"perf-node-gather-daemonset-kdbkt\" (UID: \"d140c4db-0abf-4986-8cd1-f245bc09f7ac\") " pod="openshift-must-gather-q6h8m/perf-node-gather-daemonset-kdbkt" Sep 06 12:53:57.405997 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:57.405968 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d140c4db-0abf-4986-8cd1-f245bc09f7ac-sys\") pod \"perf-node-gather-daemonset-kdbkt\" (UID: \"d140c4db-0abf-4986-8cd1-f245bc09f7ac\") " pod="openshift-must-gather-q6h8m/perf-node-gather-daemonset-kdbkt" Sep 06 12:53:57.406092 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:57.406006 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d140c4db-0abf-4986-8cd1-f245bc09f7ac-podres\") pod \"perf-node-gather-daemonset-kdbkt\" (UID: \"d140c4db-0abf-4986-8cd1-f245bc09f7ac\") " pod="openshift-must-gather-q6h8m/perf-node-gather-daemonset-kdbkt" Sep 06 12:53:57.406092 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:57.406036 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw4xf\" (UniqueName: \"kubernetes.io/projected/d140c4db-0abf-4986-8cd1-f245bc09f7ac-kube-api-access-rw4xf\") pod \"perf-node-gather-daemonset-kdbkt\" (UID: \"d140c4db-0abf-4986-8cd1-f245bc09f7ac\") " pod="openshift-must-gather-q6h8m/perf-node-gather-daemonset-kdbkt" Sep 06 12:53:57.506846 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:57.506822 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d140c4db-0abf-4986-8cd1-f245bc09f7ac-podres\") pod \"perf-node-gather-daemonset-kdbkt\" (UID: \"d140c4db-0abf-4986-8cd1-f245bc09f7ac\") " pod="openshift-must-gather-q6h8m/perf-node-gather-daemonset-kdbkt" Sep 06 12:53:57.506951 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:57.506858 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw4xf\" (UniqueName: \"kubernetes.io/projected/d140c4db-0abf-4986-8cd1-f245bc09f7ac-kube-api-access-rw4xf\") pod \"perf-node-gather-daemonset-kdbkt\" (UID: \"d140c4db-0abf-4986-8cd1-f245bc09f7ac\") " pod="openshift-must-gather-q6h8m/perf-node-gather-daemonset-kdbkt" Sep 06 12:53:57.506951 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:57.506885 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d140c4db-0abf-4986-8cd1-f245bc09f7ac-lib-modules\") pod \"perf-node-gather-daemonset-kdbkt\" (UID: \"d140c4db-0abf-4986-8cd1-f245bc09f7ac\") " pod="openshift-must-gather-q6h8m/perf-node-gather-daemonset-kdbkt" Sep 06 12:53:57.506951 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:57.506928 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d140c4db-0abf-4986-8cd1-f245bc09f7ac-proc\") pod \"perf-node-gather-daemonset-kdbkt\" (UID: \"d140c4db-0abf-4986-8cd1-f245bc09f7ac\") " pod="openshift-must-gather-q6h8m/perf-node-gather-daemonset-kdbkt" Sep 06 12:53:57.507057 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:57.506964 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d140c4db-0abf-4986-8cd1-f245bc09f7ac-sys\") pod \"perf-node-gather-daemonset-kdbkt\" (UID: \"d140c4db-0abf-4986-8cd1-f245bc09f7ac\") " pod="openshift-must-gather-q6h8m/perf-node-gather-daemonset-kdbkt" Sep 06 12:53:57.507057 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:57.506982 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d140c4db-0abf-4986-8cd1-f245bc09f7ac-podres\") pod \"perf-node-gather-daemonset-kdbkt\" (UID: \"d140c4db-0abf-4986-8cd1-f245bc09f7ac\") " pod="openshift-must-gather-q6h8m/perf-node-gather-daemonset-kdbkt" Sep 06 12:53:57.507127 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:57.507058 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d140c4db-0abf-4986-8cd1-f245bc09f7ac-sys\") pod \"perf-node-gather-daemonset-kdbkt\" (UID: \"d140c4db-0abf-4986-8cd1-f245bc09f7ac\") " pod="openshift-must-gather-q6h8m/perf-node-gather-daemonset-kdbkt" Sep 06 12:53:57.507127 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:57.507069 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d140c4db-0abf-4986-8cd1-f245bc09f7ac-proc\") pod \"perf-node-gather-daemonset-kdbkt\" (UID: \"d140c4db-0abf-4986-8cd1-f245bc09f7ac\") " pod="openshift-must-gather-q6h8m/perf-node-gather-daemonset-kdbkt" Sep 06 12:53:57.507127 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:57.507095 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d140c4db-0abf-4986-8cd1-f245bc09f7ac-lib-modules\") pod \"perf-node-gather-daemonset-kdbkt\" (UID: \"d140c4db-0abf-4986-8cd1-f245bc09f7ac\") " pod="openshift-must-gather-q6h8m/perf-node-gather-daemonset-kdbkt" Sep 06 12:53:57.514483 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:57.514462 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw4xf\" (UniqueName: \"kubernetes.io/projected/d140c4db-0abf-4986-8cd1-f245bc09f7ac-kube-api-access-rw4xf\") pod \"perf-node-gather-daemonset-kdbkt\" (UID: \"d140c4db-0abf-4986-8cd1-f245bc09f7ac\") " pod="openshift-must-gather-q6h8m/perf-node-gather-daemonset-kdbkt" Sep 06 12:53:57.594141 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:57.594112 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q6h8m/perf-node-gather-daemonset-kdbkt" Sep 06 12:53:57.861981 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:57.861954 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q6h8m/perf-node-gather-daemonset-kdbkt"] Sep 06 12:53:57.864028 ip-10-0-132-159 kubenswrapper[2576]: W0906 12:53:57.863997 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd140c4db_0abf_4986_8cd1_f245bc09f7ac.slice/crio-37c88a9ba38fb61fe685b74c979bcba68a999aee2382ad78b6287982b5644a67 WatchSource:0}: Error finding container 37c88a9ba38fb61fe685b74c979bcba68a999aee2382ad78b6287982b5644a67: Status 404 returned error can't find the container with id 37c88a9ba38fb61fe685b74c979bcba68a999aee2382ad78b6287982b5644a67 Sep 06 12:53:58.274098 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:58.274065 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q6h8m/perf-node-gather-daemonset-kdbkt" event={"ID":"d140c4db-0abf-4986-8cd1-f245bc09f7ac","Type":"ContainerStarted","Data":"5a4f07717f783d917586ad99906cada52f818725f65cd6255d25ac56eb35612a"} Sep 06 12:53:58.274548 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:58.274131 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q6h8m/perf-node-gather-daemonset-kdbkt" event={"ID":"d140c4db-0abf-4986-8cd1-f245bc09f7ac","Type":"ContainerStarted","Data":"37c88a9ba38fb61fe685b74c979bcba68a999aee2382ad78b6287982b5644a67"} Sep 06 12:53:58.274548 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:58.274183 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-q6h8m/perf-node-gather-daemonset-kdbkt" Sep 06 12:53:58.289511 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:58.289465 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-q6h8m/perf-node-gather-daemonset-kdbkt" podStartSLOduration=1.2894512040000001 podStartE2EDuration="1.289451204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 12:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 12:53:58.288197993 +0000 UTC m=+2154.143955880" watchObservedRunningTime="2026-09-06 12:53:58.289451204 +0000 UTC m=+2154.145209091" Sep 06 12:53:59.092483 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:59.092450 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c2mh4_5925c1bc-97af-42ac-a824-f6c78a448578/dns/0.log" Sep 06 12:53:59.112559 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:59.112529 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c2mh4_5925c1bc-97af-42ac-a824-f6c78a448578/kube-rbac-proxy/0.log" Sep 06 12:53:59.219067 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:59.219044 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v6tf7_1c8ae8aa-90e6-4a07-bd2c-8d9f0ab53f78/dns-node-resolver/0.log" Sep 06 12:53:59.821881 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:53:59.821855 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lt7cv_470b9439-8d7e-4a84-ae68-b66b4851c5dc/node-ca/0.log" Sep 06 12:54:01.073738 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:54:01.073707 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-849pv_a4f7e77d-d6df-44ea-b9d3-07fb828050f5/discovery/0.log" Sep 06 12:54:01.194182 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:54:01.194156 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-t9grp_33750f3b-84a4-4935-a768-e2f5bda50eb3/payload-pre-processing/0.log" Sep 06 12:54:01.233597 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:54:01.233564 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-jlvzm_2a0a23e6-8a48-4825-b2d7-e6bfd31fbe1f/payload-processing/0.log" Sep 06 12:54:02.143154 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:54:02.143121 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xdrwd_ccbb27a1-69be-4e30-9eff-95cd7280e7c1/serve-healthcheck-canary/0.log" Sep 06 12:54:02.671413 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:54:02.671387 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l9sr9_ccd2b1bb-e49b-4450-92dd-8c7d97a6ebe2/kube-rbac-proxy/0.log" Sep 06 12:54:02.691076 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:54:02.691050 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l9sr9_ccd2b1bb-e49b-4450-92dd-8c7d97a6ebe2/exporter/0.log" Sep 06 12:54:02.710377 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:54:02.710353 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l9sr9_ccd2b1bb-e49b-4450-92dd-8c7d97a6ebe2/extractor/0.log" Sep 06 12:54:04.289000 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:54:04.288973 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-q6h8m/perf-node-gather-daemonset-kdbkt" Sep 06 12:54:05.188623 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:54:05.188590 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-679b9dbb-sf7xf_44dac02b-bb13-42bf-9c70-6c47e3d6a2c4/manager/0.log" Sep 06 12:54:05.296287 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:54:05.296219 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-zw6hf_4a537ef0-6b94-407f-b443-0dc16b4aa810/manager/0.log" Sep 06 12:54:05.505491 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:54:05.505460 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7bf7c58dfb-wgl45_ce2e1a8c-9c75-42bb-ae98-04c1e57f5206/manager/0.log" Sep 06 12:54:05.524024 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:54:05.524005 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7bf7c58dfb-wgl45_ce2e1a8c-9c75-42bb-ae98-04c1e57f5206/copy-manifests/0.log" Sep 06 12:54:06.930201 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:54:06.930143 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7c968668cd-rkhkl_dd4a9b00-86c1-4b71-ae8b-382b54be0583/manager/0.log" Sep 06 12:54:08.077539 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:54:08.077514 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-849pv_a4f7e77d-d6df-44ea-b9d3-07fb828050f5/discovery/0.log" Sep 06 12:54:08.158132 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:54:08.158104 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-t9grp_33750f3b-84a4-4935-a768-e2f5bda50eb3/payload-pre-processing/0.log" Sep 06 12:54:08.187981 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:54:08.187960 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-jlvzm_2a0a23e6-8a48-4825-b2d7-e6bfd31fbe1f/payload-processing/0.log" Sep 06 12:54:09.929645 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:54:09.929615 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-849pv_a4f7e77d-d6df-44ea-b9d3-07fb828050f5/discovery/0.log" Sep 06 12:54:10.033582 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:54:10.033557 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-t9grp_33750f3b-84a4-4935-a768-e2f5bda50eb3/payload-pre-processing/0.log" Sep 06 12:54:10.057570 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:54:10.057546 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-jlvzm_2a0a23e6-8a48-4825-b2d7-e6bfd31fbe1f/payload-processing/0.log" Sep 06 12:54:13.341621 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:54:13.341593 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-57pbd_8b677696-8a5d-4304-bf45-2603cfc71903/kube-multus/0.log" Sep 06 12:54:13.425519 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:54:13.425492 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gqpgt_db464292-ffbf-4c25-85d3-ff4c4be77552/kube-multus-additional-cni-plugins/0.log" Sep 06 12:54:13.444181 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:54:13.444146 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gqpgt_db464292-ffbf-4c25-85d3-ff4c4be77552/egress-router-binary-copy/0.log" Sep 06 12:54:13.462330 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:54:13.462310 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gqpgt_db464292-ffbf-4c25-85d3-ff4c4be77552/cni-plugins/0.log" Sep 06 12:54:13.480323 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:54:13.480299 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gqpgt_db464292-ffbf-4c25-85d3-ff4c4be77552/bond-cni-plugin/0.log" Sep 06 12:54:13.497429 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:54:13.497410 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gqpgt_db464292-ffbf-4c25-85d3-ff4c4be77552/routeoverride-cni/0.log" Sep 06 12:54:13.514132 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:54:13.514116 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gqpgt_db464292-ffbf-4c25-85d3-ff4c4be77552/whereabouts-cni-bincopy/0.log" Sep 06 12:54:13.530930 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:54:13.530911 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gqpgt_db464292-ffbf-4c25-85d3-ff4c4be77552/whereabouts-cni/0.log" Sep 06 12:54:13.886081 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:54:13.886059 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wlhb5_e8d626d1-3961-441f-829f-2c8a637e63f7/network-metrics-daemon/0.log" Sep 06 12:54:13.901237 ip-10-0-132-159 kubenswrapper[2576]: I0906 12:54:13.901221 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wlhb5_e8d626d1-3961-441f-829f-2c8a637e63f7/kube-rbac-proxy/0.log"